Is there a clustering algorithm which accepts some clusters as input and outputs some more clusters?

You are describing semi-supervised learning where the training dataset is only partially labeled.
One common set of techniques to solve that problem is active learning. In active learning, there is a learning loop where the algorithm makes predictions and a human corrects those predictions.
Pre-clustering is a specific active learning technique to address the problem you describe. The goal is to repetitively select the most representative training examples to add new labels and as well as avoiding repeatedly labeling samples in same cluster. "Active Learning Using Pre-clustering" by Nguyen and Smeulders goes into greater detail.


Why after adding categorical data the Linear Regression fails?

One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:

model = LinearRegression(fit_intercept=False, normalize=True).fit(X_train, y_train)

This will avoid the dummy variable trap:
http://www.algosome.com/articles/dummy-variable-trap-regression.html

You could also use dummy encoding to avoid this problem:
http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html


What is a YAML file and where is it used in a machine learning context?

There are numerous ways to serialise data. For instance, JSON, XML, and CSV are possible approaches for serialisation. Another way is YAML. It is human-readable which means it contains readable text which is written using usual words. This stands against binary serialisation. Whenever you open the file, you will not be able to see what is written, because it is serialised in binary format and not as a human-readable format. As you may find it here, it is used for configuration files mostly. In ML context depending on your task, you may face different file format. For instance, one of the widely used formats for supervised ML tasks is CSV though you may find others use their favorite encodings. You may also want to make your own serialisation too which is possible among in-company operations is rare.


Keras P/R metrics at different thresholds during training

You can see the metrics value for each threshold along the fitting process if you explicitely instantiate the corresponding metric class for each threshold, as follows:
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=1e-2),
loss='categorical_crossentropy',
metrics=[metrics.Recall(thresholds=0.6),
         metrics.Recall(thresholds=0.9)])

model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test))

and as you can see in the image below, for each epoch you can see that the first recall value (with threshold 0.6) is higher than the second one (threshold 0.9) as expected:

And for your case, to build the list of metrics objects programatically, where you can now see 3 recalls per epoch:
thresholds = [0.6, 0.7, 0.9]
metrics_objs_list=[metrics.Recall(thresholds=thr) for thr in thresholds]


In general we would need more info on your data and what you are trying to achieve.

However, with 0.999 AUC, it is very likely you are overfitting (or you have a very simple problem).


Probabilistic gold standard vs Deterministic gold standard

No, deterministic probability is when you know for certain. If a person does not have a diagnosis, then he doesn't have the disease/condition. Doctors are not supposed to give a probability but we as human beings always like to know the likelihood.

For example, person A who is 27 years old who has the coronavirus is highly unlikely to die of the virus but it's not impossible.


How to stop a neural network from regressing to the mean

The regression tasks are not very different from classification and the behavior you faced with is probably due to a bug in the code.
If your training set is small and the network is comparably large, it should overfit on it (correlation with inputs doesn't matter, there was a paper https://arxiv.org/pdf/1611.03530.pdf that shows you can randomly shuffle imagenet labels and train one of the version of resnet on it and get perfect accuracy on that training set). Sample agnostic predictions mean underfitting. You can also try to overfit a small batch and get a zero loss. If you won't be unable to do this, there's a bug, definitely. Possibly your gradients affect the latter layers much more than the first layers, which means after some epochs of training you still have random outputs from the first layer and it's no wonder the net is trying to predict the mean of the dataset. This can happen if your net is very deep and you're not using residual connections. But it's not the case for a net of 10 (for instance) or less layers. Maybe your learning rate is too high and your first layers can't be tuned correctly. To check whether your first layers extract some meaningful information, try to look at the outputs of some of your first layers. If those outputs look like a complete noise, double check the code for optimization. You can find more interesting cases of bugs and the behavior of a buggy code in Andrej Karpathy's blog http://karpathy.github.io/2019/04/25/recipe/


Comparing SMOTE to down sampling the majority class in imbalanced binary classification

It's difficult to say without the actual data.

However, I can tell you that SMOTE creates artificial instances, hence, when used in much expanse can
"deviate" from the actual minority class data. It's difficult to determine the expanse. Many factors take place, firstly the Data, then the neighbouring coefficients.

P.S. You could try boosting using many random under Samples. Hence, instead of Random Forest you could try first Adaboost for instance were each classifier is trained on a different sub sample.


How would I chi-squared test these simple results from A/B experiment?

If what you're trying to answer is if the action taken by a user (watch, interact, nothing) is influenced by the group they are in (A or B) you can use the chi2 test of independence.

For that you can use the scipy.stats.chi2_contingency function:

a = [327445, 271602, 744702]
b = [376455, 140737, 818204]

chi2, pvalue, _, _ = scipy.stats.chi2_contingency([a, b])

In this case it returns a chi2 test statistic of 48376.48 and a p-value of 0.0, so the null hypothesis ("action is independent of group") is rejected.

You can also use the scipy.stats.chisquare function to arrive at the same results, but other than with the chi2_contingency function you'll have to calculate the "expected frequencies" yourself. The data you have recorded are your observed frequencies:

obs = np.array([a, b]).astype(float)

(Note that I converted the numbers to float, as the chisquare function will run into some weird integer overflows otherwise...!?)

The expected frequencies are calculated like that:

exp = np.outer(obs.sum(axis=1), obs.sum(axis=0)) / obs.sum()

Finally, calling

chi2, pvalue = scipy.stats.chisquare(obs.ravel(), exp.ravel(), ddof=sum(obs.shape)-2)

returns the same chi2 test statistic and p-value as before.


How to interpret results of a Clustering Heart Failure Dataset?

I would use all the features and see how the separateness of my clusters behave according to some metric, for example, silhouette score
Additionally, it is very important to scale your data prior to clustering since kmeans is a distance-based algorithm.
heart_data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00519/heart_failure_clinical_records_dataset.csv")

from sklearn.cluster import KMeans
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

Features = heart_data.drop(["DEATH_EVENT"], axis = 1).columns

X = heart_data[Features]

sc = []
for i in range(2, 25):
    kmeans = Pipeline([("scaling",StandardScaler()),("clustering",KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0))]).fit(X)
    score = silhouette_score(X, kmeans["clustering"].labels_)
    sc.append(score)
plt.plot(range(2, 25), sc, marker = "o")
plt.title('Silhouette')
plt.xlabel('Number of clusters')
plt.ylabel('Score')
plt.show()

You could also try different combinations of features so that score is maximum
For visualization purposes you can use a decomposition technique
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

pca = Pipeline([("scaling",StandardScaler()),("decompositioning",PCA(n_components = 2))]).fit(X)

X2D = pca.transform(X)

plt.scatter(X2D[:,0],X2D[:,1], c = kmeans["clustering"].labels_, cmap = "RdYlBu")
plt.colorbar();

Last but not least, I recommend to use a manifold projection such as UMAP in your data, It might help on your task by generating "well-defined" clusters (might but not necessarily, nonetheless it is worthy to try)
Look, by using UMAP the results seems to improve:
code:
# pip install umap-learn

heart_data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00519/heart_failure_clinical_records_dataset.csv")

from sklearn.cluster import KMeans, DBSCAN
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
from umap import UMAP

Features = heart_data.drop(["DEATH_EVENT"], axis = 1).columns

X = heart_data[Features]

sc = []
for i in range(2, 25):
    kmeans = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("clustering",KMeans(n_clusters=i, init='k-means++',random_state=0))]).fit(X)
    score = silhouette_score(X, kmeans["clustering"].labels_)
    sc.append(score)
plt.plot(range(2, 25), sc, marker = "o")
plt.title('Silhouette')
plt.xlabel('Number of clusters')
plt.ylabel('Score')
plt.show()

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

kmeans = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("clustering",KMeans(n_clusters=3, init='k-means++',random_state=0))]).fit(X)

pca = Pipeline([("scaling",StandardScaler()),("umap",UMAP()),("decompositioning",PCA(n_components = 2))]).fit(X)

X2D = pca.transform(X)

plt.scatter(X2D[:,0],X2D[:,1], c = kmeans["clustering"].labels_, cmap = "RdYlBu")
plt.colorbar();

Plot show first and second principal components of umap projection (It is simply a projection of how  all the features would look in 2D space)

Colours are the cluster id. i.e. for every colour we see in which cluster the algorithm (k-means) assigned each observation to.


Is there any way of ordering/sorting vectors?

If I understand your question correctly, you're looking for something that another (beyond KD-trees) standard space partitioning algorithm does. This one is called BSP-trees (for Binary Space Partition) and it uses vectors instead of dimensions for its space subdivision/partitioning. You can see it contrasted to KD-trees in slide 29 (previewed below) and beyond of this presentation.

The criterion that is used in the algorithm is intended to balance the data in the branches of the tree. This guides the definition of an order on vectors for what seems to be your purpose as well.

Otherwise, more generally speaking, one way to define a total order on a vector space is by identifying an order vector. Then your ordering is defined by the dot product with this vector. This represents projecting the entire space on the line defined by the vector and thus using the total order of the one-dimensional space it defines. For a more detailed explanation of the dot product, consider this video.


Connect a dense layer to a LSTM architecture

What you are getting as the output is the internal LSTM state.
In order to get value comparable to your labels, add a dense layer on top of it. Output dimension of dense layer would be the number of labels you want result.

If its 0 and 1, only 1 output neuron can work along with sigmoid
If there are 5 label classes, then output dimension of dense layer should also be 5


The paper cited in the question "FaceNet: A Unified Embedding for Face Recognition and Clustering" is available at https://arxiv.org/pdf/1503.03832.pdf . Page no 5 of this paper lists the reference [16] for the Inception module architecture.

The original paper introducing and describing this Inception architecture is - "Going Deeper With Convolutions", it is accessible at https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf .

An overview of inception modules is given in the diagram on page 4, its included here -

The key idea for devising this architecture is to deploy multiple convolutions with multiple filters and pooling layers simultaneously in parallel within the same layer (inception layer). For example, in the architecture shown in this diagram (figure a) employs convolution with 1x1 filters as well as 3x3 and 5x5 filters and a max pooling layer. Figure b demonstrates how the use of 1x1 convolution filters can achieve dimensionality reduction (since no. of channels is reduced).

The intention is to let the neural network learn the best weights when training the network and automatically select the more useful features. Additionally, it intends to reduce the no. of dimensions so that the no. of units and layers can be increased at later stages. The side-effect of this is to increase the computational cost for training this layer. To address this, a number of solutions have been suggested in the paper such as to deploy parallel computations for this architecture.

As an illustration of these concepts, the table on page 5 shows an incarnation of this architecture termed as "GoogLeNet" which was used for the ILSVRC 2014 competition.


Does NER work on large documents around 1500 - 3000 words or so?

The size of the document on which NER is run shouldn't be a problem at all, a standard NER system scans the document sequentially and just marks any entity it finds.
The size of the entities to find might be more of an issue, because typical NER systems rely on the previous few words to detect the boundaries of an entity. If the entity spans a large sequence of text, it's harder for the system to detect where it ends. In case it actually causes a problem with your data, it might be possible to specify exactly which features to use in the CRF model (this depends on the implementation I guess).


Can we use ReLU activation function as the output layer's non-linearity?

Yes, you can. Basically, for regression tasks, it is customary to use the linear function as the non-linearity due to the fact that it's differentiable and it does not limit the output. This means you can make any output using your inputs. People do not use tanh or sigmoid as the activation function of the last layers for regression tasks due to the fact that they are limited and cannot generate all numbers which are needed. In your task, you can use ReLU as the non-linearity. The concept of non-linearities in hidden layers is to add non-linear boundaries and for the last layer in regression tasks, it should make all possible choices. In your case, ReLU is the best.


Approximation of function with neural networks

Neural networks are function approximators, thus the answer is yes. Neural networks go from a set of inputs and map them to a set of outputs. That is precisely what a function does as well. In theory a sufficiently deep enough network and sufficient data coverage can approximate any function within a finite range.

Let's look at an example

import numpy as np
import matplotlib.pyplot as plt

We will approximate some arbitrary function

$y = x_1^2 + x_2^2 + x_1x_2^2 + \text{tan}(x_1)$

def func(x1, x2):
    return x1**2 + x2**2 + x1*x2**2 + np.tan(x1)

We will bound our inputs from $[-1, 1]$.

n = 10000
x_train =  (np.random.rand(n,2) - 0.5) * 2
y_train = np.zeros((n,))

for i in range(n):
    y_train[i] = func(x_train[i,0], x_train[i,1])

n = 1000
x_test =  (np.random.rand(n,2) - 0.5) * 2
y_test = np.zeros((n,))

for i in range(n):
    y_test[i] = func(x_test[i,0], x_test[i,1])

plt.scatter(x_train[:,0], x_train[:,1], c=y_train)
plt.show()

Let's now build a model.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv1D, MaxPooling1D, Reshape
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras import optimizers

input_shape = (2,)

model = Sequential()
model.add(Dense(32, activation='linear',
                 input_shape=input_shape))
model.add(Dense(64, activation='tanh'))
model.add(Dense(32, activation='tanh'))
model.add(Dense(1, activation='linear'))

sgd = keras.optimizers.RMSprop(lr=0.001)
model.compile(loss=keras.losses.mean_squared_error,
              optimizer=sgd,
              metrics=['mae'])

We will train the model

epochs = 50
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

Let's see how well we did. We will compare the test set that we have to the predicted values.

plt.scatter(x_test[:,0], x_test[:,1], c=y_test)
plt.show()

plt.scatter(x_test[:,0], x_test[:,1],
            c=model.predict(x_test).reshape(x_test.shape[0],))
plt.show()

Other functions

def func(x1, x2):
    if x1 >= 0 and x2 >= 0:
        return x1**2 + x2**2
    else:
        return x1*2 + x2*2

Test set

Prediction

def func(x1, x2):
    return np.sinc(x1 + x2)

Test set

Prediction

Try your own function with this network.


keras validation mean squared error always similar to 1

It could be very close to 1 (or for that matter, changing very slowly) for many reasons:

Maybe your learning rate is very small.
Maybe the network has reached its learning capacity. Since we do not know anything about the network architecture, we can't rule this out.

This does not mean there is nothing to learn from the data. In the worst case, there is nothing to learn from data, with the given architecture,  hyperparameters and the time you are willing to let the network learn.


I'm not an expert on the backpropagation algorithm, however I can explain something. Every neural network can update it's weights. It may do this in different ways, but it can. This is called backpropagation, regardless of the network architecture.

A feed forward network is a regular network, as seen in your picture. A value is received by a neuron, then passed on to the next one.

A recurrent neural network is almost the same as a FFN, the difference being that the RNN has some connections point 'backwards'. E.g. a neuron is connecteded to a neuron that has already done his 'job' during backpropagation. Because of this, the activations of the previous output have an effect on the new output.

On question #2

Interesting question. This has to do with weight initialization. Yes, you're right, each neuron in the hidden layer accepts the same connections. However, during the initalization process, they have received a random weight. Depending on your NN libary, the neurons might also have been initialized with a random bias.

So even though the same rule is applied, each neuron has different outcomes as all it's connections have different weights than the other neurons weights.

On your comment: just because all the neurons happen to have the same backpropagation function, doesn't mean they will end up with the same weights.

As they are initialized with random weights, each neurons error is different. Thus they have a different gradient, and will get new weights.

You also have to keep in mind that for a certain output to be reached, there are multiple solutions (due to non-linearity). So due to initialized random weights, one neuron might be close to a certain solution while another neuron is closer to the other.

Additionally, as was stated in the comments, a network works as a whole. The output neuron is also non-linear, and for most test cases, the output should be non-linear and the output neuron most likely requires that the hidden neurons activate at different input values.


You're also scaling $y$, then of course you are getting lower error. That question was regarding scaling $X$.
The same model will have very different error metrics when units on $y$ are changed: if I multiply all $y$ values by 100, the error will be 100 times larger, if I divide all $y$ values by 100 the error will be divided by 100.


Visualizing Decision Tree of K-Nearest-Neighbours classifier

$k$-NN does not build a decision tree to classify a new instance, it looks at the class of the most similar examples (the nearest neighbours) in the training set. So, in short, no you cannot get a decision tree from $k$-NN. You can build a decision tree for your dataset directly by using scikit-learn's DecisionTreeClassifier instead if you need a decision tree.


In general, every ML model needs a function which it reduces towards a minimum value.

DecisionTree uses Gini Index Or Entropy.
These are not used to Decide to which class the Node belongs to, that is definitely decided by Majority.
At every point - Algorithm has N options(based on data and features) to split. Which one to choose.

The model tries to minimize weighted Entropy Or Gini index for the split compared to the parent. Which indirectly imply the cleanliness of split.
 Entropy is not used because it uses log which has a higher computational cost.

Check this sample code

#Gini
a = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
p(0) = 0.4, p(1) = 0.6
gini = 0.4*(1-0.4) + 0.6*(1-0.6) #0.40

########Split - I

#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0, 1] ; a2 = [1, 1, 1, 1, 1]
gini_a1 = 0.8*(1-0.8) + 0.2*(1-0.2) #0.32
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 #0.16

########Split differently
#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0] ; a2 = [1, 1, 1, 1, 1, 1]
gini_a1 = 1*(1-1) #0 - clean split
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 # 0

Obviously 2nd split is better because Gini reduction is more
Case 1 - 0.40 --> 0.16
Case 2 - 0.40 --> 0

You may think, why not simply use Accuracy to decide the split.
Check this blog


That's correct. Without regularization your model would fit to an irrelevant noise present in your dataset. It means that training set will fit better but the overall predictive power will decrease. This is a good article about overfitting and regularization.

Also you can get some intuition by watching these images:
Without regularization
With regularization
You can clearly see that the shape at the 1st image has problems with prediction at the upper left corner but it perfectly fits to the training points.


How to prepare data for LSTM time series prediction

I hope that dataset also consist of meta data, which means you also need to have a one to one mapping of those tuples, eg. dog > good, cat > bad, kittens > bad, puppies > good, etc.

Separate the data into X:training_data, Y:label. Then use a vectorizer and train using X, Y. If you're able to do above steps then use methods like test_train set  , cross_folds etc.

Friendly suggestion: Try seq2seq layers before LSTM (they require more resources).


Balanced vs total dataset rows, which one is better?

You could work with your database as it is, because 20% positives is not really that unbalanced. Most of the algorithms work perfectly on this kind of dataset.

But if you want to work it differently, I suggest you use Synthetic Minority Over Sampling (SMOTE), which is a technique which repeats the minority dataset until is balanced.


There are many good libraries for identifying number-like values, but identifying corresponding fieldnames is trickier and likely very problem-specific.

A purely data driven approach might look for co-occurrences with numbers, for example:

if [*number-like* capacity] or [capacity: *number-like*] occurs in > x% of

the instances where "capacity" occurs,
    then "capacity" can be guessed to be a label of number-like things.

Even the nature of these relationships (eg, whether it should be before, after, capitalised) could be learnt from existing known labels.

This can be done with Google BigQuery using the public tri-gram sample data.  But I assume that's out of scope and impractical here.

A practical approach, might aim to classify these strings as numerical_units, categorical_units or unsure or even more classes, and then do human review of the latter.  (There are some very tricky cases, for example, "capacity" is numerical, but "capacity type" would not be.)

As a starting list you can use:
wiktionary:Category:en:Units_of_measure
wiktionary:Category:Symbols_for_SI_units
wiktionary:Category:en:Mathematics
Note that they are in the singular.  For your domain considering finding other such lists.

If a string is a full match with one of those labels, you can consider it a numerical_unit.
Your next concern is tolerant, fuzzy matching.  You could treat '.' as a wildcard (so "in*" matches "inch"), or find the actual abbreviations of units like "inch".  These you can label as unsure and then review.  Likewise if the word is simply contained in the string, eg "arch length" contains "length".

I think once you have done this you can make some refinements and add some special cases.  Without seeing even a sample of your data, it is difficult to say more.  If most of your strings are numerical units, then it may be easier to identify categorical units instead.


PyTorchs ConvTranspose2d padding parameter

As you quoted The padding argument effectively adds dilation * (kernel_size - 1) - padding, so you see the padding value is subtracted, the resulting shape becomes lower. It's a reverse (in some sense) operation to Conv2d, which means the arguments work the opposite way here. And I think this behavior is introduced to make it easier to design neural nets with symmetric architecture (like autoencoders) -- you just copy the parameters of kernel size, stride and padding from the corresponding Conv2d layer and get an operation which preserves the input shape of an image.


Variance of product of a matrix and vector using Python

Your are mixing some concepts. It's hard to explain but i will try.
statistics.variance(b) gives you the variance of n samples in variable b in 1 dimension. The variance matrix for b is a 1x1 matrix.
Vb is the variance matrix for 3 dimensional sample with several samples and zero covarinace between the dimensions. But it is not the variance matrix for b.
The definition of A for b is wrong. It works because np.dot uses automatically b.T, but conceptually is wrong.
I hope this help to understand it.


On Python is the `i` variable defined inherently?

No, i is not predefined. If you use enumarate(), i (or whatever you may call it) is a count: see the docs.

squares=['red', 'yellow', 'green', 'purple', 'blue']

for i, square in enumerate(squares):
    print(i, square)

Gives you:

0 red
1 yellow
2 green
3 purple
4 blue

You can also set a start value for the count (Python by default starts counting by 0):

for i, square in enumerate(squares,10):
    print(i, square)

Gives you:

10 red
11 yellow
12 green
13 purple
14 blue

In case you just want to loop over a list, you can write:

for i in squares:
    print(i)

Which gives you:

red
yellow
green
purple
blue


Pass 2 different kinds of X training data to ML model simultaneously

You could just apply two independent vectorization steps on your input X (one vectorizer for description and another for summary) and then concatenate the obtained feature matrices into a single feature matrix.

Doing it this way you will have features such as description_"such", description_"long", ..., summary_"bombay", summary_"1971", ..., so any model you apply will be able to:

Use all the features from the description and summary altogether
Give a different weight to desctiption tokens and summary tokens


How do I show plots when using Graphlab?

I have found some solution and will post it here, because somebody, who works with graphlab, can have the same question.

We can look at the example here: Six degrees of Kevin Bacon

At te beginning of the program execution you need to run next command:

graphlab.canvas.set_target('ipynb')

Exactly this is a key of the whole problem (at least by me:-)

At the beginning it is important to know, which parameter of set_target() command you want to execute. You can use two options for argument of this command:  'ipynb' (which is executed direct in iPython Notebook, like in example) or 'browser' (which open the new window with the plots)
On my machine 64-bit, Ubuntu, I can not use the command 'ipynb'. Only the 'browser'-command get me the plot back. I don't think, it is necessary to change https to http, but you can do it anyway. We have also the other machine by us (32-bit, Ubuntu) and it executes the other command 'ipynb', but not 'browser' (without to change https to http)


Is there a way (like planning algorithms) to draw a successive polyline to fill a specific shape (like triangle)?

Would not it be easier to simply apply geometric computation to your triangle to get smaller triangles whose vertices can be used for the polyline.?

With Wolfram Language

You have a triangle.

shp = Triangle[];
Graphics[{LightGray, shp}]

A ScalingTransform about the RegionCentroid can be performed with TransformedRegion to get smaller inner triangles. Below, shp is scaled by factors 0.8, 0.6, and 0.4 for the inner triangles.

scaledShps =
  TransformedRegion[shp,
     ScalingTransform[{#, #},
      RegionCentroid@shp]] & /@ Range[.8, .4, -.2];
SeedRandom[456]
Graphics[{LightGray, shp, Riffle[Hue /@ RandomReal[1, Length@scaledShps], scaledShps]}]

Each of scaledShps has the vertices as the first argument. Map (/@) First to collect these and Flatten them into a single Line.

Graphics[{
  LightGray, shp,
  Orange, Thick, Line@Flatten[First /@ scaledShps, 1]
  }]

Get more layers by increasing the number of scaling factors. For example, from .95 to .05 in steps of -.05

scaledShps =
  TransformedRegion[shp,
     ScalingTransform[{#, #},
      RegionCentroid@shp]] & /@ Range[.95, .05, -.05];
Graphics[{
  LightGray, shp,
  Orange, Thick, Line@Flatten[First /@ scaledShps, 1]
  }]

Wolfram recently released a free Wolfram Engine that can be called from many languages (Python, C, ...) so you can use the above code directly in your project.

Hope this helps.


Create continous plot using matplotlib/seaborn with specific hue

You mean something like this plot?

The code for doing it is:

import matplotlib.pyplot as plt

day=[0,1,2,3,4,5]
SBP=[146.0, 149.0, 146.0, 148.0, 149.0, 149.0]
exercise=[False, False, True, False, False, False]

plt.scatter(day, SBP, c=exercise, s=50)
plt.xlabel("Day")
plt.ylabel("SBP")
plt.show()

The color of each point is already specified in the array "exercise", so you have only to let matplotlib point to it.


When you create a Numpy array like this:

x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]])

The internal Numpy dtype is "object":

array([[1, 2], [4, 5, 6], [1, 2, 3, 4, 5, 6]], dtype=object)

and this cannot be used as a Tensor in TensorFlow. In any case, Tensors must have same size in each dimension, they cannot be "ragged" and must have a shape defined by a single number in each dimension. TensorFlow basically assumes this about all its data types. Although the designers of TensorFlow could write it in theory make it accept ragged arrays and include a conversion function, that kind of auto-casting is not always a good idea, because it might hide a problem in the input code.

So you need to pad the input data to make it a usable shape. On a quick search, I found this approach in Stack Overflow, replicated as a change to your code:

import tensorflow as tf
import numpy as np

x = tf.placeholder( tf.int32, [3,None] )
y = x * 2

with tf.Session() as session:
    x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]] )

    # Get lengths of each row of data
    lens = np.array([len(x_data[i]) for i in range(len(x_data))])

    # Mask of valid places in each row
    mask = np.arange(lens.max()) < lens[:,None]

    # Setup output array and put elements from data into masked positions
    padded = np.zeros(mask.shape)
    padded[mask] = np.hstack((x_data[:]))

    # Call TensorFlow
    result = session.run(y, feed_dict={x:padded})

    # Remove the padding - the list function ensures we
    # create same datatype as input. It is not necessary in the case
    # where you are happy with a list of Numpy arrays instead
    result_without_padding = np.array(
       [list(result[i,0:lens[i]]) for i in range(lens.size)]
    )
    print( result_without_padding )

Output is:

[[2, 4] [8, 10, 12] [2, 4, 6, 8, 10, 12]]

You don't have to remove the padding at the end - only do this if you require to show your output in the same ragged array format. Also note that when you feed the resulting padded data to more complex routines, the zeros - or other padding data if you change it - may get used by whatever algorithm you have implemented.

If you have many short arrays and just one or two very long ones, then you might want to consider using a sparse tensor representation to save memory and speed up calculations.


I took a look at that link.  It's pretty informative.  As you already know, The cut function is used to specifically define the bin edges.  There is no guarantee about the distribution of items in each bin.  In fact, you can define bins in such a way that no items are included in a bin or nearly all items are in a single bin.  The qcut function is slightly different.  The qcut function tries to divide up the underlying data into equal sized bins.  The function defines the bins using percentiles based on the distribution of the data, not the actual numeric edges of the bins.  In conclusion, if you want equal distribution of the items in your bins, use qcut . If you want to define your own numeric bin ranges, then use cut.

See the link below for more details.

https://pbpython.com/pandas-qcut-cut.html


How does the construction of a decision tree differ for different optimization metrics?

Splitting is the same your are optimising for the parameters.

In other words, gini, entropy etc logic will stay the same but the breadth, width, number of features etc will be different when you optimise it with different loss functions.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions. They are NOT (but there are modification) for them to be a loss function.
For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates)


How exactly does matrix factorization help with collaborative filtering

Answers to your question

You factorize the matrix in order to approximate original one as closely as possible. This is generally done by starting with randomized values, and updating based on error (between product of factors and original matrix). In other words, for a given matrix A, you are trying to find matrices C & D such that Error(A - (C x D)) is lowest. The algorithm is designed to find an approximation , which might result in original missing entries being replaced by new values (recommendations or ratings).
Do you just multiply? Yes. That it is the essence of calculation. For every user and product, multiplication gives you ratings or score. Sorting by score, and picking the index you get recommendation for each user. It also allows you to now store much smaller matrices than the original one.
Choice of factorization will be also dictated by application. If your application has only positive ratings, then it is better to use non-negative matrix factorization. You may start using matrix factorization methods without knowing the implementation to start with, as long as you are aware of the overall idea (and the pitfalls in using it).

Further Comments

It is a little perplexing that you start with a matrix with many missing entries (unseen items), approximate the matrix via factorization and expect to get non-missing entries (which help in doing predictions). If the task is to approximate original entries, then recommendations won't be good since you can have missing/zero entries in the approximation and still get lowest error on approximation task. The idea: regularization imposed in the algorithm (point 1 above) ensures that noise in the original data is filtered and only patterns are detected. But this idea has its fair share of critics. The introduction section of this paper by Stefen Rendle gives a readable introduction to what is happening in simple matrix factorization methods and what can go wrong. The paper also re-formulates the task with better optimization criteria.
You can also read this post by Simon Funk which explains the mechanics in a readable language (and code), for matrix factorization applied to recommendations.


What is the best deep learning library for scala?

Hi (I created deeplearning4j: warning biased source),

It's fairly new but we are open to feedback. We ported keras to scala:
http://github.com/deeplearning4j/ScalNet/

This uses deeplearning4j underneath. We would love contributions or issues.


Python: validating the existence of NLTK data with database search

NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) NLTK book (look for section "5 Named Entity Recognition").

However, if your input text has organizations in a very specific context that wasn't seen by NLTK NER model, performance might be quite low. In that case you should be looking into training your own NER model, what would extract company names. For that you would require to manually markup a small amount of your dataset.


If the categorical variable is binary(like e.g sex) you try
Point biserial correlation coefficient.
Or recode levels of var(woman->1, man->0) and use pearson correlation. Recoding it's a risky way because of you indicate order(woman>man). You should be aware of that.

Also $\chi^{2}$ test is used to determine whether an association (or relationship) between 2 categorical variables.

To find association between categorical and numerical variable
try perform One-way ANOVA test.


Model selection by Interpreting p-value of anova function

To understand why, we should try to understand what we're doing here.

Let's start with the first p-value: 2e-16, what does that mean? This is the p-value under the null hypothesis that the linear model and the second polynomial model are statistically identical. We say that they are the same if the extra coefficient in the polynomial is statistically zero. This is exactly what the p-value is telling you. It reads like this: There is 2e-16 probability that the null hypothesis of the linear model and the polynomial are identical. This is a very small probability, so you can conclude that they are not identical. This means, the second-order polynomial is a better model than the linear model with a significance level of 5%.

(PS: a more correct statistical interpretation of the p-value relates to false-positive rejection, but let's not go into that deep)

Now, using the same logic, you can conclude the third-order polynomial fits better than the second-order polynomial.

Next, the p-value comparing the third and fourth order polynomial is about 0.05. You can reject it or you don't want to reject it, this is up to you. But If was you, I'd simply fail to reject it because it's larger than 0.05.

Finally, the final p-value is about 0.37 which is too high. This means although the fifth order polynomial fits better than the fourth order, the loss in your RSS is insufficient to justify the loss of the degree of freedom. Therefore, we say the fifth order polynomial is statistically no better than the fourth-order.

Conclusion: It's "bad" to have a large p-value because you really want to reject a null hypothesis. Statistically, we do it to control the false positive rate.

PS: In your example, R uses the F-test to compare the two models.


The loss function handles the conversion. TensorFlow has a SparseCategoricalCrossentropy and a CategoricalCrossentropy loss function. The first expect your labels to be provided as integers, where the latter expects one-hot encodings. In the given example, they use the SparseCategoricalCrossentropy loss function, therefore it is ok to supply your labels as integers.


identifying input sequence using neural network

A layer in a neural network does not process logic of the kind you present. It cannot "check the input pattern with a simple if else pattern". You could make a model that processed your example inputs to example outputs as you show, but it would not be a neural network.

In machine-learning, neural networks have a formal, numeric structure. There are a few variations, but essentially in your diagram:

Each circle is a "neuron" which may have a numeric output value, also called its activation.
Each arrow is a connection between neurons that shows where values are passed between neurons. Each connection has a strength or weight associated with it.
When you have many arrows going into a neuron, you can calculate its new activation starting with the sum of weights times the activation values of neurons that they are connected from. Call this sum $z$. Then the activation of the neuron you are looking at is $a = f(z)$ where f is an "activation function" (also called "transfer function").

The above very roughly covers how many neural networks are constructed. As you can see though, there is no room in that for having your if/else statement.

Instead, neural networks like the one you show (technically a feed forward network also called a multi-layer perceptron or MLP) are most often trained by example. This involves repeatedly giving the network inputs where you know the output you want. To start with the network will give completely incorrect outputs, and you then adjust the weights to try and make the result closer. This is repeated very many times until the network becomes the best model it can for your purpose. A very common algorithm used to train a neural network in this way is backpropagation and gradient descent.

For your problem as described, a neural network is not a good model, and would not be used in the real world. That is because you already know the full logic of a simpler model that covers all possible inputs. At best with a neural network you will get an approximation to that already-perfect model.

However, if this is for the purposes of understanding neural networks, your approach would be roughly this:

Generate some training data with example inputs and outputs. Most example outputs will be the 23,23 default values, but when inputs are the special values 6,7,8 then the training outputs should be 3 and 4.
Build an initial network. Normally you would use an appropriate library - it is important you learn how the library works and understand any limitations. E.g. many activation functions have a maximum output of 1, so cannot learn the values 3,4, or 23.
Train the network to fit the example data. There are very many options there, depending on the library you are using. Usually it is just a matter of selecting parameters for the learning that help it progress smoothly. You may have to run training for a large number of repeats over the whole dataset (called epochs)
Test that your network works as you expect. Even if you have trained it well, the response of the network is never as precise as your problem statement. If you gave it inputs of 5.8,7.2,8.1 (i.e. close to 6,7,8) then it would likely output something like 7.2,8.5 and not 23,23 .  . . it will interpolate between values.


Creating queries on the fly and general manipulation for dataset of half a million data records

Checkout the R programming language, specifically the dplyr and ggplot2 packages.


Normally, you would shuffle up all of the examples and then portion them off into batches of some chosen size. Then, you would do a parameter update based on the gradient of the loss function with respect to each batch. This whole process is one "epoch" of training. Typically, deep neural nets are then trained over many epochs, often with a learning rate that varies as training proceeds.

An important aspect of this process is that when the data is shuffled up at the beginning of an epoch, examples are put into batches with different examples than they were matched with in the previous epoch. This gives us a more complete sampling of batch gradients and improves our collective stochastic estimation of the optimal gradient (the derivative of the cost function with respect to the model parameters and data).

Short answer: your model performance will almost certainly be worse if you choose static batches and shuffle those batches around instead of shuffling the data and then dividing them into batches.

Also, be careful of your shapes. With a dataset shape of (60, 784, 1000), it's highly likely that you're working on MNIST or one of its cousins - MNIST is 60,000 examples of length (784,), if the images have been flattened down from 28 x 28 pixels. The 784 being axis 1 in your shape tuple makes me concerned that you've reshaped your data incorrectly. Make sure that the array entries that represent a single image are where you think they are.


Probably.  It depends on what you are doing exactly, and in particular which summaryFunction you are using.  If it's twoClassSummary for common binary classification then you should be able to specify metric='Sensitivity'.  If that's not the case you also have the option of overriding the default performance summaryFunction with your own implementation that calculates the metrics you desire.

I highly recommend reading through the documentation at http://topepo.github.io/caret/training.html#control


Batch-mode active learning: How to select the batch that will bring the largest model improvement

Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.


is it possible (and/or logical) to set feature importance for xgboost?

You may duplicate some features in your dataframe. I certainly was looking for two things as you mention,one of them is putting some features closer to the root node, and also want some features to appear more on branches. I understand your concern is to make your trees more aware to some features. I may suggest something there. XGBoost samples each feature uniformly, which it would be nicer if we can say that some features are more important and should be used more. Short hack would be duplicating the columns while decreasing the colsample_bytree ratio. XGBoost for now doesn't support weighted features since it draws features uniformly. I've seen it on another place, there's no specific sampling technique for features(columns in XGBoost) in the documentation. Check colsample_bytree : https://xgboost.readthedocs.io/en/latest/parameter.html (Credits: Heard that technique from a webinar of https://www.kaggle.com/aerdem4 )
Generally, newbies like me starts to create a lot features via engineering, however also may discard that important features becomes minority. Although XGBoost parameters can deal with it while hyperparameter search, it seems really not beneficial creating trees with no rain data because of the domination of high number of engineered wind data when problem is predicting drought .
If your concern is specifying domain knowledge, you may also define a domain knowledge via "feature interaction constrainst" in XGBoost, documentation. I hope I'll edit the answer when I become %100 sure what I'm doing with feature interactions: https://xgboost.readthedocs.io/en/latest/tutorials/feature_interaction_constraint.html#:~:text=Feature%20interaction%20constraints%20are%20expressed,but%20with%20no%20other%20variable.
I'm aware these doesn't answer your primary question, it seems not possible for now. Also, I'm not sure that features put on the top are more important, this doesn't seem always true.
Edit: XGBoost has added feature_weights to the DMatrix, in 1.3.0! However, 1.3.0 is not stable yet. I'll edit here when I find/apply feature weights. https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit


How multi-scale CNN selects final output map

Think of this as varied filter size and varied filter values. It will extract different representation (or say capture different part of the image), and then stack them to get a bigger feature vector. Then, you do the featurisation .Also, check for dilated CNNs used for NLP. They are based on somewhat similar concepts.


convert list of tuple of tuple to list of tuple in pySpark

To get the unique elements you can convert the tuples to a set with a couple of comprehensions like:

Code:

[tuple({t for y in x for t in y}) for x in data]

How:

Inside of a list comprehension, this code creates a set via a set comprehension {}. This will gather up the unique tuples. Two loops are needed inside of the set comprehension:

for y in x for t in y

because the tuples of interest are themselves inside of a tuple.

Test Code:

data = [
    [
        ((4, 2), (2, 1)),
        ((4, 2), (6, 3)),
        ((2, 1), (4, 2)),
        ((2, 1), (6, 3)),
        ((6, 3), (4, 2)),
        ((6, 3), (2, 1))
    ], [
        ((4, 2), (-3, 4)),
        ((-3, 4), (4, 2))
    ], [
        ((2, 1), (-3, 4)),
        ((-3, 4), (2, 1))
    ], [
        ((-3, 4), (6, 3)),
        ((6, 3), (-3, 4))
    ]
]

expected = [
    ((6, 3), (4, 2), (2, 1)),
    ((4, 2), (-3, 4)),
    ((-3, 4), (2, 1)),
    ((6, 3), (-3, 4))
 ]

assert expected == [tuple({t for y in x for t in y}) for x in data]


About confidence/prediction intervals: parametric methods VS non-parametric (via bootstrap) methods

The simplest type of bootstrapped confidence intervals, percentile intervals (described in simple terms here) are by no means the only way of creating bootstrapped CIs. You should think of bootstrapped confidence intervals as a family of techniques - there are some examples listed here along with some good information around the assumptions underlying the use of bootstrapped statistics.

How many original samples you need in order for bootstrapped confidence intervals to converge varies. Like the parametric approach, this convergence will arrive at the wrong value to the extent that your sample population is not representative of the global population you are estimating. A rule of thumb is to use parametric approaches if the sample population is n < 30, but in principle bootstrap could sometimes work with smaller samples (see this thread). If you have 30 samples or more, standard practice is to draw >1000 bootstrap samples, but the important thing is that you see the estimates converging.

If you want to use bootstrap CIs you should be really explicit about the approach you have used whenever you present your analysis, because different ways of producing them can lead to different intervals.


Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution

Depending on the size of the test and training set, and how they were sampled, their distributions may not be the same. if the sets aren't very big or weren't randomly sampled then their distributions won't necessarily be the same, and may not correspond to the distribution of the population
One way to test this is to compare the distributions of the other variables in the test and training sets.
In terms of selecting individual models or the ensemble for prediction, I would recommend trying the individual models and the ensembled combination in k-fold validation, and selecting whichever approach results in the best performance. This way you are using the entire training set for training and validation, and the assessment of this performance should be the best approximation of test set performance.


Training Machine Learning Model - Neural Network - Islands Problem

For a start I would begin with simpler tests cases because they look rather complex (see an exemple of what you seems to simulate below). May be start with a 10x10 grid and generate 1 to 5 islands of different size.


Why exactly using a test set for model evaluation is a bad idea?

In the linked video, a polynomial is fitted to some data. The degree of the polynomial has to be chosen, but it cannot be trained. It is therefore a hyperparameter. In order to choose the degree, several instances of the algorithm, each one trained with a polynomial of different degree, have to be evaluated on a data set that was not used for training. However, if we used the test set for assessing the values of hyperparameters, we would also be coupling the resulting model to the data in the test set, as the chosen hyperparameter (and therefore the whole model) would depend on the test set data.

The test set should only be used for the final evaluation in order to know how your algorithm is going to behave on data that has never been seen before by it. This is why there is normally a validation set used to tune the hyperparameters.


To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect.

Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the  increased complexity overweight the added value from extra features.

I understand you‘ve tested quite wide range of hyper-parameters and enough combinations. If you apply regularisation via colsample_bytree and/or colsample_bylevel it might happen that at some stage the randomly chosen columns (features) are less informative than your original features and the algorithm is forced to use these features for further splits. Does it make sense to you?

The number of added features might be crucial, if it is too high relative to original 20 the new features become just too dominant. For example, this might happen if one adds some nominal features with high cardinality, which are then dummy-encoded.

In order to improve the results with wider data you might want to play with the parameters controlling the early stopping and stop the fitting even earlier.

Edit

The rationale for my last suggestion:
I assumed that worsened performance on wider dataset is due to overfitting, i.e. significantly worse performance on test dataset than on training data. Early stopping should prevent / control the overfitting, but it seems it didn’t work so good in your case and thus should be improved.

You can and should test different combination of new features, but trustworthy quality metric is crucial for the model choice. If you significantly overfit, the performance on training data (or even cross-validation) will not help you to choose the model (or feature combination) with good performance on test data set.


Timeseries of odds in race - how to pick a model

Odds are directly connected to prediction accuracy. 4 means that you loose your money in 4 cases and win in one case. This is 20% probability $\left(\frac1{4+1}\right)$. If the bet goes down to 0.25 then the probability is 80% $\left(\frac1{0.25+1}\right)$. This logic is true if the spread between betting for the horse and against the horse is low. Betting is a financial market where people with the best betting algorithms bet the most and actually decide what the bet should be. If there is a situation on the market where someone could earn money it will be closed in milliseconds by trading robots. This is called an Efficient Market Hypothesis. In your data you can actually check if the hypothesis is true for horse betting.

If you would like to use ML algorithms to try to predict the outcomes better:

There are several ways to interpret the data.

You predict the probability that each horse is a winner.
You predict a rank of each horse.
You predict time difference between horses and the winner
You predict time

Some algorithms could be better at predicting one problem or the other.

Possible algorithms:

Linear regression. I always start with this because it is easy to calculate and easy to interpret. You build separate models for each horse number. Can be used for all problem definition.
Logistic regression. Similar to regression, but can only be used for the first problem definition.
LGBoost and XGBoost. Most likely the best models for such data, because the data is structured (not an image, sound). Can be used for all problem definitions.
Neural networks. For example MLP (multilayer perceptron).

You can also calculate some additional features. For example, multiply some odds, or calculate inverse normal distribution from the odds.

I would also recommend to collect more data. For example weather data. Then it is good to know the horse. Some horses could be more popular than others and receive better odds. Also time of the year because the horses could have a training plan according to the time of the year.


Methods to analyze time series data when only small amount of it is labeled

As mentioned in the comments semi-supervised methods are worth having a read. If the time-series are labelled per time-step you can make use of dynamical systems and Gaussian processes. Some methods that could be useful using NN's are:

Pretraining a neural network using an auto encoder. You can use an RNN to encode the temporal aspect of the series (https://papers.nips.cc/paper/5271-pre-training-of-recurrent-neural-networks-via-linear-autoencoders.pdf).
GAN's (Generative adversarial networks) currently very popular way of making use of your copious amounts of unlabeled data by jointly training two networks (supervised vs unsupervised) to play a zero sum game vs each other. https://arxiv.org/pdf/1611.09904.pdf (some work on RNN-GAN's again if you want to account on the temporal structure of your data). Note these are very hard to train.
Ladder Network https://arxiv.org/pdf/1507.02672.pdf . A very interesting semi-supervised method for regular fully connected networks. Which yielded very interesting results on the MNIST data set (only 300 labelled samples were used to reach state of the art results).
CNN's are known to be effective with time series too. Since they implement adaptive FIR filters on the data. It can be worth trialing in conjunction to RNN's.

All the methods above still require some reasonable amount of labelled data.


Large Graphs: NetworkX distributed alternative

Good , old and unsolved question! Distributed processing of large graphs as far as I know (speaking as a graph guy) has 2 different approaches, with the knowledge of Big Data frameworks or without it.

SNAP library from Jure Leskovec group at  Stanford which is originally in C++ but also has a Python API (please check if you need to use C++ API or Python does the job you want to do). Using snap you can do many things on massive networks without any special knowledge of Big Data technologies. So I would say the easiest one.

Using Apache Graphx is wonderful only if you have experience in Scala because there is no Python thing for that. It comes with a large stack of built in algorithms including centrality measures. So the second easiest in case you know Scala.

Long time ago when I looked at GraphLab it was commercial. Now I see it goes open source so maybe you know better than me but from my out-dated knowledge I remember that it does not support a wide range of algorithms and if you need an algorithm which is not there it might get complicated to implement. On the other hand it uses Python which is cool. After all please check it again as my knowledge is for 3 years ago.

If you are familiar with Big Data frameworks and working with them, Giraph and Gradoop are 2 great options. Both do fantastic jobs but you need to know some Big Data architecture e.g. working with a hadoop platform.

PS

1) I have used simple NetworkX and multiprocessing to distributedly process DBLP network with 400,000 nodes and it worked well, so you need to know HOW BIG your graph is.

2) After all, I think SNAP library is a handy thing.


Neural network for Multiple integer output

welcome to the site!

I think the key word you need to know that defines your task is: multi-target classification or regression.

You can find an explanation and some possible techniques at this link.

For neural networks:

The key is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (135,) and the output (132,).

The usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras:

model = Sequential()
model.add(Dense(200, input_dim = (135,)))
model.add(Activation('relu'))
model.add(Dense(200))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(132))
model.compile(loss='mean_absolute_error', optimizer='Adam')


You can think of a multimodal distribution as a union of multiple unimodal distributions. In the case of GANs for image processing, each mode could be a category of images. To significantly simplify what's going on to motivate the idea, if you have a dataset of cat pictures and dog pictures, you can think of that as bimodal. Just cat pictures, that's unimodal. Training a GAN on a multimodal dataset of images means that it should be able to generate members of any image category in your dataset, and it should generate different categories with the same frequency with which they appeared in the training data.

GANs are trained by taking a random vector as input and attempt to construct a feasible member of the data distribution as output. In effect, the GAN learns a (surjective) mapping from the random space onto the multimodal distribution, such that random inputs will generate samples from the multimodal data distribution as outputs.


Why are RNNs used in some computer vision problems?

When I think about RNNs applied to Computer Vision, two main research areas of Deep Learning come up to my mind:

Image Captioning: Neural Networks trained to produce descriptions of images. In that case, you have a Conv ecoder that processed pixel data, and an RNN decoder that produces a description.

Video processing (I don't have a better term). Anything regarding video data requires a mixture of CNNs and RNNs. In general: you process each frame with Conv layers, and the sequence of processed image frames is then fed into some RNN layer that performs the final task, that could be classification, segmentation, self driving cars, ... you name it.


Partial derviative of prediction (sigmoid applied) with respect to weight

Recall that for chain rule, we have $$\frac{d}{dw}h(g(w))=h'(g(w))g'(w)$$

For the context of your question, $h(t)=\sigma(t)$ and $g(w)=Wx+b$,

hence that is why we have one more term.


The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() using panda python

while time <= 30:

You are trying to evaluate the vector time, but in python, boolean evaluations with that expression can only be done over scalars. The thing is, as time has multiple values, the Python interpreter can't figure out if the condition must hold over all of them, just one, etc.

If you want to evaluate if any of the values of time, or all the values in time you can use methods like time.any() or time.all()

You can always walk through all the items in time and perform the evaluation on them:

for t in time:
    if t <= 30:
        ...


Are there pitfalls in using the output of machine learning model, such as a neural net as the input to a traditional GLM or similar?

Theoretically there is no problem. I've seen tree models put as predictors in logistic models. NN as input into a GLM moodel makes sense. The ultimate decition should be made based on the predictability of the NN.

You have to mind a few issues:

model mantenance and deployment. The NN model would probably have more parameters than your vanilla GLM deployment might be more involved. You could live it frozen and never update it, and let the GLM model use the NN score as long as it adds to the GLM prediction.
interpretability. There might be managers whose acountability is to review risk, and they may not be comfortable with a NN. In that case feeding the NN results into a GLM might make it more acceptable, much as credit scores are used in some risk models.


What do you use to generate a dashboard in R?

I think that Shiny is an overkill in this situation and doesn't match your requirement of dashboard reports to be static. I guess, that your use of the term "dashboard" is a bit confusing, as some people might consider that it has more emphasis of interactivity (real-time dashboards), rather than information layout, as is my understanding (confirmed by the "static" requirement).

My recommendation to you is to use R Markdown and knitr, especially since these packages have much lower learning curve than Shiny. Moreover, I have recently run across an R package, which, in my view, ideally suits your requirement of embedding small charts/plots in a report, as presented on your picture above. This package generates static or dynamic graphical tables and is called sparkTable (http://cran.r-project.org/web/packages/sparkTable). Its vignette is available here (there is no link to it on the package's home page): http://publik.tuwien.ac.at/files/PubDat_228663.pdf. Should you ever need some interactivity, sparkTable provides some via its simple interface to Shiny.


Is it possible to automate generating reproducibility documentation?

To be reproducible without being just a replication, you would need to redo the experiment with new data, following the same technique as before.  The work flow is not as important as the techniques used.  Sample data in the same way, use the same type of models.  It doesn't matter if you switch from one language to another, so long as the models and the data manipulations are the same.

This type of replication will show that the results you got in the first experiment are less likely to be a fluke than they were earlier.


What are interleaved layers of convolutions?

The Oxford dictionary explains "interleave" as

  Place something between the layers of (something)

or in the context of telecommunication, as

  Mix (digital signals) by alternating between them.

In the FaceNet paper, Schroff et. al describe their first architecture as

  multiple interleaved layers of convolutions, non-linear activations, local response normalizations, and max pooling layers

which means they create a CNN by "altering between" these layer types (e.g. Conv -> Pool -> Conv -> Pool -> ..). Or with the wording of the first definition, you "place one layer" of each type "between" the other existing layers (i.e. start with a couple of Conv layers, then place Pool layer between two Conv layers, and so on).
The final architecture would look somewhat like this:

Conv -> ReLu -> LRN -> Pool -> Conv -> ReLu -> LRN -> Pool -> Conv -> ...

So, this "interleaved" CNN is a network where the different layer types are applied in series, layer-by-layer.
As a comparison, in "inception" type CNN, you apply the different layer types in parallel (the graphic below is 1 layer):

              +---> Conv 3x3 ---+
              |                 |
prev layer ---+---> Conv 5x5 ---+---> next layer
              |                 |
              +---> Pool 3x3 ---+


You need to use loaded_model.predict(TestValue), not loaded_model.score(TestValue). The latter is for evaluating the models accuracy, and you would also need to pass the true height of the son, which is the y value it's asking for.


Where can I find a good machine learning research team /university Lab for a PhD in CS?

University websites might be a  good way to search a suitable laboratory for you since you have a topic in mind. You may go through the current researches and research interest of the laboratories and professors. Sometimes, the university may even publish a list of available projects and/or supervisors who are looking for Ph.D. students. Try contacting the supervisors through email to discuss your ideas and the opportunities.

Another way is to search for Ph.D. opportunity through your undergraduate and master supervisor. They might have a lot of connections with other universities as well as recent researches. They may guide you based on your knowledge and interest.


Techniques for Cluster Analysis of a Very Large (n=140000) Binary Dataset in Python?

Since you have binary data, you can use hamming distance as your similarity metric for clustering.
Then you have to handle the size of your data. Here are couple of options:

Rent a bigger computer or a cluster of computers from cloud computing service.

Treat the problem as a streaming data problem. Use "online" techniques to load and fit only a portion of the data at a time.


How to optimize cohort sizes to reduce pair-wise comparisons?

There is a better way if I understand your question correctly. Here is the algorithm I propose:

Initialize a 'window' list and a 'pairs' list
Sort your data on birthday from old to young (or the other way around)
Loop over your rows and keep track of all the rows that are still in the last three years since your current row. When you get to a new row, throw out rows that are now more than three years apart, add the current row together with the rows in your 'window' to your pairs set and add current row to your 'window'.

This means you only iterate over your main list once, and if you implement your 'window' list properly (like a linked list for example) you don't need to do a lot of looping in that regard either. You also get all the pairs only once as a by product. Plus you actually get all the pairs, while with your binning approach you get missing pairs around the bin edges (if you don't overlap).

If you use the overlapping binning approach I think you should have bins of 6 years width and the borders shift 3 years each time, that way all true pairs are in at least 1 bin together, and there is the least amount of unwanted pairs.


Does Orange scale the data automatically for the linear regression with Ridge regularization

Not by default, no, as shown by the normalise=False here:

  class Orange.regression.linear.RidgeRegressionLearner(alpha=1.0,
  fit_intercept=True, normalize=False, copy_X=True, max_iter=None,
  tol=0.001, solver='auto', preprocessors=None)


How does epochs related with converging the model?

I don't know if I understood correctly the question, but basically, epochs are the unit of measure of the time spent on training the machine learning models.
An epoch is conventionally considered as a full pass on the dataset, meaning that for each epoch, you feed the data to the model from the beginning to the end of the dataset. Once an epoch is complete, you can start again another epoch training your model from the beginning of the dataset to the end, and so on and so forth.
The convergence of the performance of the models is of course dependant on the epochs, because usually what happens is that the more epochs you train your model, the less noisy (more stable) will be the predictions.
Hope I answered your question.


Try to capture output, and modify it in a desired way:

rev.stem <- function(data){
  out <- capture.output(stem(data))
  out <- c(out[1:3], rev(out[4:length(out)]))
  cat(out, sep="\n")
}

rev.stem(cars$dist)


Is it possible to create feature groups in Orange?

This is not Orange-specific, but IIUC, you could preprocess your data (e.g. in Python or Excel) to have each of the 10 coughs pertaining to a patient on the same patients line. Thus you would have: 100 rows of patients with each row (10*170 + other patient data) attributes wide.


Neural network model for sparse multi-class classifier on Tensorflow

The model seems fine to me. Did you try to optimize the hyperparameters? You could use trial and error methods for optimiring the parameters such as,

Learning rate
Number of epochs
Batch size etc,.

And instead of vanilla gradient descent optimizer, you could try other optimizers such as Adagrad, Adam or RMSProp etc,.

Best,
Sangathamilan Ravichandran.


Dealing with input images of different shapes in PyTorch

Try this function that uses cv2 resize function:

def leaf_image(image_id,target_length=160):
    """
    `image_id` should be the index of the image in the images/ folder

    Reture the image of a given id(1~1584) with the target size (target_length x target_length)

    """

    image_name = str(image_id) + '.jpg'
    leaf_img = plt.imread('images/'+image_name)  # Reading in the image
    leaf_img_width = leaf_img.shape[1]
    leaf_img_height = leaf_img.shape[0]
    #target_length = 160
    img_target = np.zeros((target_length, target_length), np.uint8)
    if leaf_img_width >= leaf_img_height:
        scale_img_width = target_length
        scale_img_height = int( (float(scale_img_width)/leaf_img_width)*leaf_img_height )
        img_scaled = cv2.resize(leaf_img, (scale_img_width, scale_img_height), interpolation = cv2.INTER_AREA)
        copy_location = (target_length-scale_img_height)/2
        img_target[copy_location:copy_location+scale_img_height,:] = img_scaled
    else:
        # leaf_img_width < leaf_img_height:
        scale_img_height = target_length
        scale_img_width = int( (float(scale_img_height)/leaf_img_height)*leaf_img_width )
        img_scaled = cv2.resize(leaf_img, (scale_img_width, scale_img_height), interpolation = cv2.INTER_AREA)
        copy_location = (target_length-scale_img_width)/2
        img_target[:, copy_location:copy_location+scale_img_width] = img_scaled

    return img_target

# Test the leaf_image function
leaf_id = 343
leaf_img = leaf_image(leaf_id, target_length=160);

plt.imshow(leaf_img, cmap='gray'); plt.title('Leaf # '+str(leaf_id)); plt.axis('off'); plt.show()

Source: https://github.com/WenjinTao/Leaf-Classification--Kaggle/blob/master/Leaf_Classification_using_Machine_Learning.ipynb


Reason for having both low loss and same predicted class?

From your description, it might be an issue of too high learning rate. When this happens, the weights cannot reach closer to the minimum and the loss does not go down.

Accordingly, I would suggest reducing the learning rate to 0.001 and using Adam.


Potentially your looking for a description of the “vanishing or exploding gradient problem” in neural networks.

This post does a deep dive into this problem.  The author states the issue briefly as:

  an important observation: in at least some deep neural networks, the
  gradient tends to get smaller as we move backward through the hidden
  layers. This means that neurons in the earlier layers learn much more
  slowly than neurons in later layers.”

On a high level, we can look at the output of the neural network as a composition of functions $\hat{y} = f(g(h(x)))$.

After obtaining the loss, $y - \hat{y}$, we use gradient descent to begin to updating the weights.  The weights in the deepest layer depend only on the derivative of the deepest function in the earlier composition, while shallow layers’ weights depend on the outer functions in the composition.

The authors explain that this results in a derivative for the shallow layers that includes the multiplication of the deeper weights, which are usually < 1 (in the vanishing gradient problem).  Multiplying many numbers < 1 results in a very small number and a vanishing update to the weights in the shallow layers of the network.


Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

Neural networks are different from other Machine Learning techniques in that they have to learn by repetition.  In Neural Network parlance, the repetition is called an epoch. During an epoch, each instance, in your case each word, is evaluated and the error is applied backwards through the layers and the weights are adjusted.  The changes to the weights can happen more or less aggressively based on the hyperparameters, but they tend to only move a little at a time.

You have trained your word2vec model for two epochs.  I didn't see how you initialized the weights for your model, but typically they are set to random values or set to a zeros. Given your results look a little random, I am sure it is because the model has barely begun to learn.

In order to get a well trained word embedding model, or any other Neural Network, you may need to train for hundreds, thousands, or even millions of epochs.  This is why the pre-trained word embedding models are so valuable even though they are freely available.  If you just want a well trained embedding model, download it. If you need to incorporate some domain specific usage, you can further train the downloaded embeddings.

HTH


How you pad it (and even whether you do so) would depend on what you expect of the data. This imposes boundary conditions on the data which will induce artifacts in any transform you make. How bad this effect depends on how well geared your data is to accepting a particular padding method.

Padding methods include zero padding or a periodic bound.

Padding doesn't have to be done in the time domain. Eg interpolating in the frequency domain and back transforming allows you to extrapolate.

If your analytics has a finite history (eg FIR filters) then you can isolate time regions where padding is unnecessary and draw comparisons therefrom.


What to do about non-responses to demographic survey questions?

It depends on what you want to do with the data. One common approach is to treat non-response as „missing“, meaning, you drop this observation. This will possibly lead to a bias in the data, e.g. if a certain group refuses to answer a question more often than others. As a cure to this problem, some use ex-post weighting of data. You can do this if you know how often some group will occur in the population as a whole. So if you see that group A is underrepresented in your data, you assign a higher weight to each observation in group A. You can use this weights to adjust descriptive statistics or regression results. The technique is called probability weighting: https://en.m.wikipedia.org/wiki/Inverse_probability_weighting.

Note that there are limitations of this approach. You can reasonably claim that ex post weighting is okay for things such as income if you have sufficient data, assuming that (important!) say a low income bracket in the available data is representative for the whole population of low income households/people. This will not necessarily hold in other cases, e.g. if you study certain lifestyles or so.

Non-response may also occur if people are ashamed to answer some question. In this case, there is a systematic bias in the data with unknown data generating process, so that it is more or less impossible to cure this problem/bias.

Yet another option would be imputation, meaning you estimate what someone would answer (often applied to income for instance). However, in this case you can never be sure that you get things right. So imputation often is very problematic.


How many dimensions to reduce to when doing PCA?

After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.


What are some appropriate models to use for inventory forecast based on consumption history or trends?

The model you are looking for is called ARIMA, which are time series models where you can obtain trends, cycles, etc.

The ARIMA models lets you model your univariate data by detecting monthly/annual cycles, natural growth (trends) for a variable depending on the history of the same variable.


Output of evaluation metric for XGBoost - is it cumulative?

That reflects the performance of the boosting of the 10 trees together.
I do reccomend you this tutorial https://www.youtube.com/watch?v=sRktKszFmSk&t=4s
Gradient boosting is well explained there.
Using as a predictor only the 10th tree will give you weird results, since it is fitted on the residual errors of the previous 9


What makes a graph algorithm a good candidate for concurrency?

Two words: associative and commutative

In other words, the operations that the algorithm does need to be independent of how you order or group your data...this minimizes the need for cross-talk in the algorithm and leads to more efficiency.


You can run a statistical test like Augmented Dickey Fuller to check if the variable follows a random walk.
Use the adfuller function from statsmodels:

from statsmodels.tsa.stattools import adfuller

results= adfuller(df['<your_col>'])
print('p value:', results[1])
print(results)


If you know what the state history is, you don't need a 'hidden' Markov model, you just need a Markov model (or some other mechanism).  The 'hidden' part implies a distinction between some sequence of unobservable states, and some observations that are related to them.  In your case, you say you have observed the past states for each customer, so you don't necessarily need to infer anything 'hidden'.

The simplest way to proceed in your case would be to calculate a transition matrix, i.e. probability of state given previous state.  That's a very simple model but it might do what you want.  To do this, just look at all state pairs, and count to get p(s2 | s1) = p(s1 & s2)/p(s1).  This is equivalent to a 1-gram model that you've probably read about. Each state is akin to a word.

You could also make a more complex model, like a 2-gram model or even an RNN.  Honestly, since you have a fixed amount of history, you can just throw your data into an scikit-learn model or xgboost or something, where each customer's history is the vector of predictors and the next state is the outcome.  It won't know the sequential dependencies, but you are essentially indexing the past states by time, so it may work pretty well.

If you need more clarification about part of this, just ask.


Because a NN doesn't see a box. The only job it does is to minimize the loss using Feedforward and Backpropagation.

Let's say the image has 1 object -
=> 4 o/p values => 4 o/p neurons => 4 y_train values => 4 losses into the NN
=>To have the loss minimum it will adjust itself to have all the 4 o/p neurons as close to the truth as posible.

Let's say the image has 2 object -
=> 8 o/p values => 8 o/p neurons => 8 y_train values => 8 losses into the NN
=>To have the loss minimum it will adjust itself to have all the 8 o/p neurons as close to the truth as posible.

The difference - Obviously, later will be a bigger network and need more epochs to get the golden values of weights

If you have just started Deep Learning -
Michael Nielsen


Optimal proportion between the amount of Class = 1 and the amount of Class = 0?

The balance between two classes in a classification is very important, as you do not want your model to overfit for a particular class.

This is where you use metrics apart from accuracy to really evaluate how good your model truly is. In case you are not able to balance the dataset, there are multiple ways to work with imbalanced data. They are as follows:

You could use certain techniques like SMOTE to generate more samples of the undersampled class
You must split your dataset in test and train with stratification so that you have a balance in your evaluation.
You can sub-sample the large class and balance the two classes, and this multiple times by taking random sub samples of the larger class.

Thorough analysis of the result is very important to understand how to proceed towards the solution to the problem. Please look at f1 score, precision and recall apart from accuracy. Also read about micro/macro averaging of these metrics.

There are a lot of conversations on datascience stackexchange and stackoverflow on how to work with imbalanced data for classification. Here is a link : https://stackoverflow.com/questions/40568254/machine-learning-classification-on-imbalanced-data

Have fun with machine learning :)


Splitting train and test in multilabel classification to avoid missing data in the train set

Accuracy is an awkward measure to use to assess a model predicting classification into multiple classes, and rare events are hard for models to predict well. Outcome categories that exist in your test set but not your training set will of course lead to lower accuracy, but it's far from the only thing that can go wrong.

And, if these outcomes are rare relative to the number of observations you have, it's unlikely that your model is highly predictive but reduced to 20% accuracy based solely on the failure to classify a handful of rare events. It's more likely that the model has fairly poor performance overall. Importantly, this is not necessarily the result of you doing something wrong. Some things are difficult to model, and the best-performing model possible may still have a poor success rate (however you want to measure that).

Without knowing more about your modelling procedure it's hard to make specific suggestions, but a few things come to mind (this is nowhere near an exhaustive list):

Summarizing the documents may not be a very reliable process.

If you have strong protocols for summarizing each, reviewing those might help to find odd edge cases where similar observations get summarized differently in some systematic way, or differing observations being summarized together improperly. If you are manually summarizing each document, it may be worth having multiple people perform the summarization and then evaluation the inter-rater reliability of the summaries.

The documents from which the summaries are derived may differ from one another.

If we're describing highly formulaic documents this might not be an issue. But if the documents are written in prose by different people, it's very possible that issues of personal style and preference on the part of the writers is producing systematic differences from other writers, making it hard to classify a document from an imaginary, generic author.

There could be a mismatch between the number of predictors and outcome classes and the number of observations available to train the model on.

600 observations may be enough for a model classifying a document into one of three categories based on the presence or absence of 15 predictor variables. It is not enough for a model classifying a document into one of 400 categories based on 600 predictor keywords.


It is hard to tell if it is the model or the signal/noise ratio in data. One sanity check could be to see how well a human would perform with the most important features.

I don't think you want to switch to a sequence model, I have a hard time seeing it do better for this data. Also, don't switch to regression.

Some suggestions:

Try a gradient boosting model, like LightGBM or XGBoost. It is really good on tabular data.
Do feature engineering and try to come up with features that might be good but does not exist in the data. You could aggregate to get the sequential information in there, but you already seem to be doing that to some extent.


Minimize correlation between input and output of black box system

Please allow me to paraphrase your question to make sure I get your question right.
Suppose you have 1 million data entries. Each entry consists of three inputs $X_1$, $X_2$, $X_3$ and one output $Y$. One of the three inputs, $X_1$, is a noise. You want to find the relation between $X_1$ and $Y$. So you can remove the impact of $X_1$ on $Y$.
Let's use a simplified example, $Y = 2X_1 + 3X_2 + 4X_3$
If you update each of the 1 million entries with:
$Y_{new} = Y - 2X_1$
Now you can examine the updated 1 million entries to find the relation between input $X_2$, $X_3$ and $Y_{new}$:
$Y_{new} = 3X_2 + 4X_3$
The relation between $X_1$ and $Y$ can be computed with different algorithms, such as regression, decision tree. How to measure which algorithm generates the best result, i.e. truly reflects the relation between $X_1$ and $Y$ hence removes the impact of $X_1$ on $Y$ as much as possible?
The problem you're solving is an interesting one. I would use a free tool such as Knime to find out which algorithm generates the best result in modeling the relation between $X_1$ and $Y$. It's faster than coding in Python or Matlab.
Details: to put the problem intuitively, suppose the price of a house is determined by location, size of the house, condition of the house. Assume these three factors are independent from each other. You want to remove the impact of "condition of the house" on "price" as much as possible. In essence, you want to find the relation between "condition of the house" and "price" which may not be linear.
You can let Knime read the 1 million entries of "condition of the house" and "price", try Decision Tree, SVM, or even Ensemble Learning, and see which algorithm generates the best model of $X_1$ and $Y$. In other words, try and error :)


How match output (pred value) to input value

The outputs of your model are in the same order as your inputs, so the first row in your output array corresponds to the first row in the testX array. If you want to have both the inputs and the model prediction in one table you can just concatenate them along the column axis.


How to annotate text documents with meta-data?

Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving beyond this level of metadata. I would especially pick a format that can be used regardless of development environment and one that can keep some basic structure if that becomes relevant (like tokenization)

It might seem strange, but I would honestly suggest JSON. It's extremely well supported, supports a lot of structure, and is flexible enough that you shouldn't have to move from it for not being powerful enough. For your example, something like this:

{'text': 'I saw the company's manager last day.", {'Person': [{'name': 'John'}, {'indices': [0:1]}, etc...]}

The one big advantage you've got over any NLP-specific formats here is that JSON can be parsed in any environment, and since you'll probably have to edit your format anyway, JSON lends itself to very simple edits that give you a short distance to other formats.

You can also implicitly store tokenization information if you want:

{"text": ["I", "saw", "the", "company's", "manager", "last", "day."]}

EDIT: To clarify the mapping of metadata is pretty open, but here's an example:

{'body': '<some_text>',
 'metadata':
  {'<entity>':
    {'<attribute>': '<value>',
     'location': [<start_index>, <end_index>]
    }
  }
}

Hope that helps, let me know if you've got any more questions.


Looking into so many plots may miss the bigger picture.
We should keep the initial plot in "One Consolidated plot"
e.g.
Can add few other consolidated views - Pie on aggregated data on region and Vehicle


Why is the difference of datetime = zero in the following dataframe?

It is hard to see where the problem is with your code; given your comments, my guess is a problem in the timezone conversion.

I cannot see a problem exactly, buy my suggestion to help debug your situation would be to convert all time to timestamps - by default that will be seconds since the epoch (January 1st 1970). These are then simply normal float64 values (I convert to integer microseconds below). If your differences are still returning zero, subtracting simple number, then the problem cannot be in the differences.

Here is a minimal working example. A simple dateframe with timestamps with millisecond frequency:

import pandas as pd
from datetime import datetime

In [1]: df = pd.DataFrame(pd.date_range(start=datetime(2016,1,1,0,0,1),
    ...:     end=datetime(2016,1,1,0,0,2), freq='ms'), columns=["dates"]).head(10)    # just take first 10 rows for simplicity

Make a new column, converting the dates into microseconds since the epoch, as an integer:

In [2]: df["timestamps"] = df.dates.apply(lambda x: int(datetime.timestamp(x) * 1e6))

In [3]: df
Out[3]:
                    dates        timestamps
0 2016-01-01 00:00:01.000  1451602801000000
1 2016-01-01 00:00:01.001  1451602801001000
2 2016-01-01 00:00:01.002  1451602801002000
3 2016-01-01 00:00:01.003  1451602801003000
4 2016-01-01 00:00:01.004  1451602801004000
5 2016-01-01 00:00:01.005  1451602801005000
6 2016-01-01 00:00:01.006  1451602801006000
7 2016-01-01 00:00:01.007  1451602801007000
8 2016-01-01 00:00:01.008  1451602801008000
9 2016-01-01 00:00:01.009  1451602801009000

You could at this point check that there are no duplicates in any of your columns, using:

In [4]: df.T.duplicated()
Out[4]:
dates         False
timestamps    False
dtype: bool

If there are duplicates, that could be the cause of differences equal to zero.

Now compute the differences, in my case all 1-millisecond differences (1000 microseconds):

In [5]: df[["date_diffs", "timestamp_diffs"]] = df.diff(1)

In [6]: df
Out[6]:
                    dates        timestamps      date_diffs  timestamp_diffs
0 2016-01-01 00:00:01.000  1451602801000000             NaT              NaN
1 2016-01-01 00:00:01.001  1451602801001000 00:00:00.001000           1000.0
2 2016-01-01 00:00:01.002  1451602801002000 00:00:00.001000           1000.0
3 2016-01-01 00:00:01.003  1451602801003000 00:00:00.001000           1000.0
4 2016-01-01 00:00:01.004  1451602801004000 00:00:00.001000           1000.0
5 2016-01-01 00:00:01.005  1451602801005000 00:00:00.001000           1000.0
6 2016-01-01 00:00:01.006  1451602801006000 00:00:00.001000           1000.0
7 2016-01-01 00:00:01.007  1451602801007000 00:00:00.001000           1000.0
8 2016-01-01 00:00:01.008  1451602801008000 00:00:00.001000           1000.0
9 2016-01-01 00:00:01.009  1451602801009000 00:00:00.001000           1000.0

Add a sample zero difference and retrieve the index of zero differences:

In [7]: df.iloc[3, 3] = 0.0

In [8]: np.where(df == 0)
Out[8]: (array([3]), array([3]))

Hopefully that will be enough to find where there could actually be zero difference. If all of that works out without zero differences, I would either look into your timezone conversion (maybe they do some rounding there?) or report a bug to the pandas issues

EDIT

After your update trying my debugging method, and seeing this:

I believe there must be either a bug or an inherent limitation to your system.

Pandas Bug

A bug might be in your specific version of Pandas and its df.diff() method. Check your version of Pandas with pd.__version__ and look on the issues page I linked above for any clues ... maybe just try the latest stable version anyway.

32-bit system

Another possible solution would be that you are running on a 32-bit system and so could actually lose the precision required for my example above.
A 32-bit integer can only retain precision for 10 digits. My timestamps above require 12 digits. You can find out on like this, on Linux or Windows or Mac.

Additionally, you could retry my example, but only look at seconds instead of microseconds, just to be sure.


What is the meaning of term Variance in Machine Learning Model?

It is pretty much what you said. Formally you can say:

Variance, in the context of Machine Learning, is a type of error that occurs due to a model's sensitivity to small fluctuations in the training set.

High variance would cause an algorithm to model the noise in the training set. This is most commonly referred to as overfitting.
When discussing variance in Machine Learning, we also refer to bias.

Bias, in the context of Machine Learning, is a type of error that occurs due to erroneous assumptions in the learning algorithm.

High bias would cause an algorithm to miss relevant relations between the input features and the target outputs. This is sometimes referred to as underfitting.
These terms can be decomposed from the expected error of the trained model, given different samples drawn from a training distribution. See here for a brief mathematical explanation of where the terms come from, and how to formally measure variance in the model.
Relationship between bias and variance:
In most cases, attempting to minimize one of these two errors, would lead to increasing the other. Thus the two are usually seen as a trade-off.

Cause of high bias/variance in ML:
The most common factor that determines the bias/variance of a model is its capacity (think of this as how complex the model is).

Low capacity models (e.g. linear regression), might miss relevant relations between the features and targets, causing them to have high bias. This is evident in the left figure above.

On the other hand, high capacity models (e.g. high-degree polynomial regression, neural networks with many parameters) might model some of the noise, along with any relevant relations in the training set, causing them to have high variance, as seen in the right figure above.

How to reduce the variance in a model?
The easiest and most common way of reducing the variance in a ML model is by applying techniques that limit its effective capacity, i.e. regularization.
The most common forms of regularization are parameter norm penalties, which limit the parameter updates during the training phase; early stopping, which cuts the training short; pruning for tree-based algorithms; dropout for neural networks, etc.
Can a model have both low bias and low variance?
Yes. Likewise a model can have both high bias and high variance, as is illustrated in the figure below.

How can we achieve both low bias and low variance?
In practice the most methodology is:

Select an algorithm with a high enough capacity to sufficiently model the problem. In this stage we want to minimize the bias, so we aren't concerned about the variance yet.
Regularize the model above, to minimize its variance.