What techniques should I use to compare the similarity between a bunch of texts?

As of now, I can think of two ways to formulate this problem:
1. Search problem
Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build capabilities like Semantic search using Word2Vec models, etc.
Now write a query engine which takes a resume and queries this Search engine. It will be blazing fast since job listing will be all indexed.
2. Similarity problem
I would have created hybrid similarity function. For example:
a) How many top key words matched between resume and job listing
b) Similarity of resume and job listing using Doc2Vec (Pre compute vector for job listings)
c) Using algorithms like Locality Sensitive Hashing to reduce the lookup space
This approach will be slow but might yield a good result.


doc2vec - How does the inference step work in PV-DBOW

The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model.

Training Phase

In this model, we forget word ordering information and setup a very simple neural network.
Represent all input document tags in a vocabulary to get unique ID for each document tag. Do the same for words.
For a given document ID, the input is one-hot encoded representation of document tag IDs. Output is one hot encoded representation of a randomly selected word.
We want to setup training such that for a given tag ID, a randomly selected word from that document will be predicted with high probability.
So the neural network getting trained transforms one hot encoded representation to document/tag vector. The tag vector is passed through another layer with softmax at the output. Both set of weights are adjusted during training phase.
What training achieves: It represents all document tags in a new space such that probabilities for randomly selected words in each document are maximized starting from that vector space representation to softmax output.
Important to note that there are two set of weights. Input to hidden layer and hidden to softmax output.

Inference Stage

Words are unique across document. So a word 'India' in training and inference stage gets mapped to same ID in vocabulary.
But there's not such concept for document ID since documents are assumed to be unique and there's no shared ID between training and inference.
Inference stage runs a (sort of) reverse calculation. What vector space representation is most appropriate for this document, if I use the same set of weights from hidden space to output layer? The weights from hidden layer to softmax output are kept constant!
Select a word from new document at random. Start with a random representation for the document vector (input to hidden layer). Pass it through neural network from hidden to softmax output with constant weights (learnt during training). Adjust the randomly initialized weights such that softmax probability is maximized for the selected word. Repeat this process many times.
This is the reason why you need stochastic gradient descent and need to specify number of steps.

Summary

Document IDs are not common from training and inference stage (like word IDs in w2v).
There are two set of weights, keep one constants during inference stage based on learning from training phase. Adjust the other using SGD.
You mapping new document to document vector space such that documents which had similar words during the training phrase are close in the document vector space.

This is a very good tutorial on the implementation by Richard Berendsen. I have ignored additional complexities like negative sampling, context window from the explanation.


Answer:
The error states that the target needs to be in single row i.e. (1,) where as you are giving (10,)

ValueError: Error when checking target: expected dense_10 to have shape (1,) but got array with shape (10,)

SOLUTION 1:
Use "keras.utils.np_utils.to_categorical" to convert your model data labels to categorical one hot vectors.

from keras.utils import to_categorical
label = array(label_column)
# one hot encode
encoded_label = to_categorical(label)

Then you can have 10 labels in target which your last layer expects.

model.add(keras.layers.Dense(10, activation='softmax'))

SOLUTION 2: If above does not work, then change the last layer of your CNN network to have one output as given below.

 model.add(Dense(1, activation='sigmoid'))


Does resizing images during training affect the bounding box annotations?

My doubt is whether the resize would now change the position of my object according to annotation?

Yes, it will.

  should i annotate on the resized images(resize them myslef before
  training?)

No, you should annotate at the original size.

You solve this by applying the corresponding transformations on your bounding boxas well. So if you resize your image, you rescale your bounding box. This allows you to expand to different image augmentations without redoing annotations for all of them.

I recommend you to chose a library that has to ability to apply transformations on both images and their bounding boxes. I use albumentations but there are others such as imaug.


scipy fit for t distribution seems broken for bi-modal data

The t-test has many assumptions. That dataset violates several of them:

Data should be sufficiently large (>30 independent points)
Data should be approximately normally distributed

Given that the assumptions are violated, you can not expect to valid results.


Instead of formulating the problem with Venn diagrams you could also look at a simple two by two table. Usually the problem is formulated graphically in a different way (see picture below from Wikipedia page). If you are interested in just predicting the occurrence of the value 1 you are just focusing on the sensitivity (or true positive rate) of your classification algorithm.
This is rater straightforward within a ROC analysis framework, you could just select a minimun value for the classifier threshold. However, this comes at the cost of very low specificity. You should also consider the cost-benefit ratio of your sensitivity/specificity results.

https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity


Can closer points be considered more similar in T-SNE visualization?

I would present t-SNE as a smart probabilistic adaptation of the Locally-linear embedding. In both cases, we attempt to project points from a high dimensional space to a small one. This projection is done by optimizing the conservation of local distances (directly with LLE, preproducing a probabilistic distribution and optimizing the KL-divergence with t-SNE). Then if your question is, does it keep global distances, the answer is no. It will depend on the "shape" of your data (if the distribution is smooth, then distances should be somehow conserved).

t-SNE actually doesn't work well on the swiss roll (your "S" 3D image) and you can see that, in the 2D result, the very middle yellow points are generally closer to the red ones than the blue ones (they are perfectly centered in the 3D image).

An other good example of what t-SNE does is the clustering of handwritten digits. See the examples on this link:https://lvdmaaten.github.io/tsne/


how to adjusting already built ML predictive model

This depends on your model type:
Classical using ensemble/stacked models:
If you are using classical machine learning, you could use your old model built on the previous 1 million records, and create a new model on the most recent 500k records and then combine the predictions in an ensemble or stacked approach.
References for Ensemble and Stacking:
https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/
Video Reference: https://www.youtube.com/watch?v=Un9zObFjBH0
AI/NN using transfer learning:
If your are using a NN (neural network) model, you can use the idea of transfer learning. Save your model built on the first 1 million records, then add it as an initial layer to a new NN for analyzing the new data. You can then save the new NN and use it in the next round.
Reference: https://machinelearningmastery.com/transfer-learning-for-deep-learning/
Video Reference: https://www.youtube.com/watch?v=yofjFQddwHE
General guidelines:
If you need to do this updating process many times, you can create a new model on n number of records, drop the oldest data/model off once your new dataset reaches a minimum, and predict only on the last x number of models. n and x are adjusted based on your data, flexibility and need for real-time predictions. If the data is changing over time, then it would be better to only use the latest data, or weight the older data lower and the newer data higher.
Here is a good definition of transfer learning: "Transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task."


Clustering with arrays / vectors as features?

You need to define a distance function that yields the desirable output.

Usually it will be enough if you can construct it such that d(a,b)for your purpose.

Doc2vec vectors are a bit tricky because they have so many dimensions, and a very strange geometry. It doesn't even seem to be clear whether cosine or Euclidean is to be used on these vectors...

Either way, you have to carefully balance the different features. In the other answer, minmax or stddev scaling was proposed. On one hand, that likely ruins the doc2vec properties, on the other hand this will but much more weight on the word vectors than on the other attributes.

For DBSCAN, you can also follow the "Generalized DBSCAN" approach. Here, the idea is just to define different thresholds for different features. Then neighbors must satisfy all thresholds. I.e. have doc2vec cosine less than A, and other features' distance less than B.
This is likely easier than balancing them as factors inside a single distance function.

But nothing saves you from weighting different feature (sets} carefully.


What to do with stale centroids in K-means

k-means finds only a local optima. Thus a wrong number of cluster or simply some random state of equilibrium in the attracting forces could lead to empty clusters. Technically k-means does not provide a procedure for that, but you can enrich the algorithm with no problem.

There are two approaches which I found that are useful:

remove the stale cluster, choose a random instance from your data set and create a new cluster with centroid equal with the chosen random point
remove the stale cluster, choose the farthest distant point from any other centroids, create a new cluster with centroid in that point

Both procedures can lead to indefinite running time, but if the number of this kind of adjustments is finite (and usually it is) that it will converge with no problem. To guard yourself from infinite running time you can set an upper bound for the number of adjustments.

The procedure itself is not practical if you have a huge data set a a large number of clusters. The running time can became prohibitive.

Another procedure to decrease the chances for that to happen is to use a better initialization procedure, like k-means++. In fact the second suggestion is an idea from k-means++. There are no guarantees, however.

Finally a note regarding implementation. If you can't change to code of the algorithm to make those improvements on the fly, your only option which comes to my mind is to start a new clustering procedure where you initialize the centroid positions for non-stale clusters, and follow procedures for stale clusters.


Should I use rescale parameters for data augmentation?

The rescale argument byitself does not augment your data.

If you input pixels are in the range of [0,255], you can rescale to [0,1] using the code in your question. Just remember to do the same on all partitions of your dataset.

If you input is already in the range of [0,1], then obviously you should not rescale.


Constructing graph of crypto financial instruments 2016-2017

Finding correlation - It has been described in the provided reference
Degree of distribution - Every decent package in which you build the network will have a function that will calculate the empirical distribution of the vertex degrees.
Analysis of cliques and independent sets in the market graph - This is a problem of Graph Clustering. There are many methods available. Start by reading this.


What is the difference between sentiment analysis (e.g. on Facebook) and Twitter sentiment analysis?

There is nothing specifically called Twitter sentiment analysis, rather its colloquial. Sentiment analysis is a common term and could be applied on any dataset depending upon the use-case considered.

I have not came across any facebook api for scraping posts (maybe there are privacy concerns, as sometimes you cannot see fb pages when you are not an user), nevertheless there are several apis to scrap tweets from twitter, even specifying person, geographic location, language.

here is a github gist for reference, it uses python tweepy module to grab current tweets.


Semantic segmentation: mean IOU in presence of missing classes

You somehow misunderstood how mean_IOU is calculated.

First, mean_IOU is not calculated per image and then averaging over all images, but calculated per class for images and then averaging over classes. That means for a single image, if some classes are missing (in both ground truth and your predicted labels), you don't need to worry about the 0/0 situation. You need to only update the confusion matrix, until you finished with all images, then calculate the mean_IOU.

In the extreme case, some classes are missing in all images. In that case, you should adjust you num_class parameter. Even if you don't, the missing classes will be treated as IOU=0 instead of 1 (sad...)

For example, suppose your image has 5 pixels and you have 3 classes. The ground truth vector is always [0,0,1,1,1] and you always predicted [1,1,0,0,0] (note that you predicted class 0 and 1 inverted, and class 2 is missing). In this case, the mean_IOU is 0, not 1/3.

Therefore, you should understand that the IOU is already being normalized by the sum of the numbers of pixies in (the union of your predicted pixels and the ground truth pixels) across all you images. So I don't think it's really necessary to re-weight the IOU any more.


Bad Input Shape -- How to interpret and Diagnose; Also side ML question

According to doc GaussianNB can handle multiple classes, but its fit function still accepts a one dimensional array. This array is allowed to contain multiple classes like: [0, 1, 2, 3,...]
When I changed your train labels as below fit function worked:
train_labels = np.random.randint(0, 9, 25000)

So you can create an array for labels. If each sample only belongs to one class then you can label this sample with corresponding class, but if there are some samples that have multiple classes at the same time then you should use one of them, or create groups that consist of these classes, and label sample if it belongs to this group or not.


How to cluster and visualize 3D data in python

The following steps is one method to achieve your result. I used Wolfram Language but the method can be applied by any language with the right libraries.

FindClusters
for category A data (dataA),
calculate the
ConvexHullMesh
for each of these clusters,
for each point in category B (dataB)
calculate the
RegionDistance
to each of the category A hulls,
and
Pick the
dataB points by their nearest category A hull.

We can collect related 3D example data from "AdministrativeDivision" Entity object properties.
dataA =
  Select[FreeQ[_Missing]]@
   EntityValue[
    EntityClass["AdministrativeDivision", {"ParentRegion" -> Entity["Country", "UnitedStates"]}]
    , {"GiniIndex", "TotalVotingRate", "HomeOwnershipRate"}];
First@dataA

{0.4776, 56.3712%, 70.7%}

I used FindClusters with the "MeanShift" method to cluster. Two clusters were found.
clusters = FindClusters[dataA, Method -> "MeanShift"];
Length@clusters

2

The list of ConvexHullMesh for each cluster is obtained by
hulls = ConvexHullMesh /@ clusters

These can be visualised with their internal points by combining aListPointPlot3D of clusters with a Graphics3D of hulls (with low Opacity to make them transparent) with Show.
cp =
 Show[
  ListPointPlot3D[
   clusters
   , PlotStyle -> ColorData[110]
   , PlotTheme -> {"Web", "FrameGrid"}
   , BoxRatios -> Automatic]
  , Graphics3D[
     {Opacity[.1]
     , MapIndexed[
       {ColorData[110] @@ #2, EdgeForm[{Thin, Opacity[.1], ColorData[110] @@ #2}], #1} &
       , hulls]}]
  ]

For category B example data we need points outside of the hulls of the clusters. We can create a Cuboid around the RegionUnion of hulls and hollow out the volume of hulls by taking the RegionDifference. This region can be visualised with RegionPlot3D.
With[
  {ru = RegionUnion[hulls]}
  , rd =
   RegionDifference[
    Cuboid @@ Transpose[
      MapAt[Ceiling[#, 0.01] &, {All, 2}]@
       MapAt[Floor[#, 0.01] &, {All, 1}]@
        RegionBounds@ru]
    , ru]
  ];
RegionPlot3D[rd
 , PlotStyle -> Opacity[.1]
 , Axes -> True]

Then we can generate RandomPoints inside this region for dataB.
SeedRandom[19283745]
dataB = RandomPoint[DiscretizeRegion@rd, 20];

The dataB points can be combined with the dataA cluster plot with Show.  All of the dataB points are outside of the dataA hulls.
Show[
 cp
 , ListPointPlot3D[dataB
  , PlotStyle -> Black]
 ]

Now that we have example category B data (dataB) we can calculate the RegionDistance of each point to each of category A's hulls. Then by Ordering these distances the First entry gives the hull the point is closest to.
nc =
 First /@
  Ordering /@
   Transpose@
    Through[
     Function[r, RegionDistance[r, #] &, Listable][hulls][dataB]
     ]

{2, 1, 2, 1, 2, 2, 1, 1, 2, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 1}

Lastly we Pick the dataB points by their closest hull and combine their plot with the dataA cluster plot with Show. dataB points have been coloured to indicate their closest dataA cluster.
pncB = Pick[dataB, nc, #] & /@ Range@Length@hulls;

Show[
 cp
 , ListPointPlot3D[
  pncB
  , PlotStyle -> ColorData[104]
  , BoxRatios -> Automatic
  ]
 ]

Hope this helps.


Try this:
t = tensor([0.1, 0.2, 0.3, 0.4, 0.5, 0.6])
t.index_copy_(0, tensor([2, 4]), tensor([0.7, 0.8]))

Reference:
torch.Tensor.index_copy_


Train and predict on different data structures

From what I understood from your problem, the chronicles are a way to model relationship between events and if so they are extracted from the sequential data (as a higher interpretation of data). I had a similar project involving Episode Mining in sequential database involving alarms.

The problem is whether you are trying to extract chronicles or use existent ones to predict something in a sequence (which is like using a pre-trained model).

Training a model mostly involves adjusting weights to inputs and applying (non-) linear functions on them in order to output the result in a certain format. (This is a very broad and abstract way of describing most ML algorithms)

So to answer your question, No you cannot do it in this broad way but you could try to make it so that the inputs are homogenous.

Building a  model that extract from the sequence a chronicle would help or doing the exact opposite, generating a sequence with the chronicles and then perform machine learning.

Use simple Apriori/FPGrowth to extract rules (old way of doing things)
Build a (Hidden) Markov Model to estimate each state probability, in order to predict outcome.
Use RNN to learn how a sequence behaves (LSTM or GRU) with the fact that it's time-dependent, the RNN will remember important states and forget unimportant ones (which is similar to chronicles but in a black box kind of way).
Create something hybrid or completely new.


How choosing a value for random_state argument in sklearn.model_selection works?

In the documentation (for example for KFold) you can find the following explanation:

random_state : int, RandomState instance or None, optional, default=None
If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random. Used when shuffle == True.

Obviously, the way you use it falls into the first use case. Thus, the number is used as a seed for the pseudorandom number generator. When you have the seed hardcoded as a parameter, you use the same seed for a second time. This will result in the same sequence of pseudorandom numbers, thus not random at all. Because of this, use numeric seeds only for testing purposes when you want to reproduce your "randomness". For learning experiments and production purposes, you can just leave the default None value.


"The New Science of Retailing: How Analytics are Transforming the Supply", Book by Ananth Raman and Marshall Fisher has some very good information of data science applied to Retail. Although, they don't address your case directly, they do have a chapter on store level execution that might help, and another chapter on assortment planning which might have relevant information.


Well, thats a question that institutes at univiersities, among others, are trying to solve since several decades and where millions of money currencies are spent in each country every year.

Short answer: for now, nothing general is known to perform good.

Long answer: it depends completely on your image domain. Do you want to recognize suns in a picture? That possible up to some false discovery rates. Do you want recognize familiar faces? Possible, see automated passport controls at airports. Emotion recognition? Good luck with that! Grouping by persons? Possible, but statues will be included. etc...

Something really general:  Grouping by colors? Totally viable! But will not mean much. Something that includes some hint to the structure? Well, the easiest way is to compute color histograms and use them as feature vectors for (like suggested) nearest neighbor calculation - (watch out for normalization, color space, and other sources of information distortion)

Basic answer: take some Computer Vision courses either online (e.g. https://youtu.be/skaQfPQFSyY?list=PL7v9EfkjLswLfjcI-qia-Z-e3ntl9l6vp)

There is also an intersting TED talk to that topic https://youtu.be/40riCqvRoMs

On the other side: don't let yourself down! Understand the #basics of data processing and break the structures! Develop that algorithm we are all hoping to use some day!


If the goal is to predict actual revenue, you want to predict the unadjusted rate. You should include month as a variable so your model has a chance of getting the seasonality right.

If the goal was more performance monitoring (e.g. "We did better in March than February"), then you would want to be analyzing the adjusted rate. You see this all the time in the US news when they forecast adjusted home sales, car sales, or unemployment.


How Does Cross-Entropy Work With Softmax Activation Function?

As you mentioned, (output-expected)
see, this value will be less than 0 (negative as output<1 and expected=1)
Now w_new = w_old - learning_rate*derivative
now based on derivative, w_new can be greater or lesser than w_old.

I guess when you are assuming with negative derivative, weights decrease is not correct.
Also even if weights decrease the output will decrease is also not correct.


Resampling for imbalaced datasets: should testing set also be resampled?

You are right! As you said, the final goal of the model is to work with real life data distribution, therefor the test data has to stay unchanged.

The goal of resampling is not to have the same number of objects from all classes, but to fit model to work with your data. So an intuitive example might be: if you have k objects of the first class and 5*k of the second, but your model learn data well, then resampling is not necessary. It's the reason, why we shouldn't use it during test (we have already trained model)

One more thing that could be useful to remember is that F1 score could be calculate in multiclass task in different ways (it could account label imbalance), here is an example from sklearn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


Association rule learning for multi-classification suggestions

Assuming there are many repetitions of each attribute and class, an embedding space can be learned. Attributes and classes that co-occur together will be projected into nearby space. One example is affinity weighted embedding.

Then prediction becomes approximate nearest neighbor search. For a given set of attributes, find the nearest class(es). One example is locality-sensitive hashing.

The strategy of combining learned embedding spaces and approximate nearest neighbor search scales very well for prediction. Search time can remain constant if you are willing to accept a looser definition of "approximate" as the data grows.


Over-fitting issue in a classification problem (unbalanced data)

I'm working on a similar classification problem using Quinlans C5.0 and in my experience there are couple of tweaks inside that might help you.

First, I believe using such big number (trials=500) for boosting is
affecting the overfitting, since the model will repeatedly try and
improve on the misclassified instances in every iteration. Try using
a smaller boosting size, or even turning it off - it might help with
the overfitting problem.
There are a couple of other parameters you might find useful. The minimal leaf size (minCases in C5.0Control() is something to you want to keep as big as possible when avoiding overfitting issues.
Another one is the confidence factor (CF) which lets you control
the severity of pruning of the tree, meaning lower factor levels will likely prune away the leaves which overspecify the classification.
Finally, you might try using the fuzzyThreshold control option which lets you 'soften' sharp thresholds with an inbetween grey area.

These are all very well documented in the C5.0 vignette as well as on the informal tutorial page.

Hope this helps you a bit.


Object detection model using images which have single instance of class per image-can it detect multiple instances in a single image?

I believe that the resulting accuracy of your model is due to a mix of reasons. Firstly, the data-set is really small, so you cannot expect to have an efficient model just by using your 200 images.

Proposed Solution #1: Data Augmentation. Enlarge your data-set by processing your images, crop, rotate, etc Examples with code. As for labeling the bounding boxes, you could do it manually or by creating functions to transform the co-ordinates based on the image processing method.

Proposed Solution #2: Also, try a different approach. I think that a version of the Yolo algorithm would be more appropriate for multiple instance detection YOLO-read_the_docs_link because R-CNN's search algorithm is fixed, no learning happens during selection, which might lead to bad region proposals resulting to fewer detection and low probability outputs.

  Why does the model detect objects more efficiently when zoomed in closer?

The obvious reason is that it is easier for your model to detect them, given the fact that the data-set is small. Also, it depends on the labeling, if your ground truth contains only large bounding boxes, then it would detect bigger pomegranates more efficiently. Adding cropped instances of your data-set to the train set might help with this issue too.


I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$, under mild assumptions on the activation function.

  But the main problem is that the theorem has a very important
  limitation. The function needs to be defined on a compact subsets of
  $\mathbb{R}^n$ (compact subset = bounded + closed subset). But why
  is this problematic? When training the function approximator you
  will always have a finite data set. Hence, you will approximate the
  function inside a compact subset of $\mathbb{R}^n$. But we can always
  find a point $x$ for which the approximation will probably fail. That
  being said. If you only want to approximate $f(x)=x^2$ on a compact
  subset of $\mathbb{R}$ then we can answer your question with yes.
  But if you want to approximate $f(x)=x^2$ for all $x\in \mathbb{R}$
  then the answer is no (I exclude the trivial case in which you use
  a quadratic activation function).

Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $\sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $x\to 10000$ if you don't use enough terms in your Taylor expansion.


Why does horizontal lines in plt.plot(feature, '.') mean that the data have been properly shuffled?

Shouldn't an index have only one value in the feature axis?

Yes, that's correct. On the graph given as example this is not visible because there are too many row indexes (50000). As a consequence it's impossible to distinguish a particular index from its neighbors, but if the X axis was stretched long enough one would see a single feature value for every index.

One horizontal line should mean that the feature values for all indexes have been uniformized, not randomized?

I think there could be two different confusions here:

An horizontal line means that a single feature value is distributed uniformly across the indexes, which is equivalent to saying that the indexes are random for this feature value. In other words, the chance that this feature value appears at a particular index is the same as at any other index. This is what the author means: the order (indexes) is random for any feature value.
The values for all the features have not been uniformized, this can be seen from the fact that vertically the density of the points is different around the middle (say 0.4-0.6) and the extremes (say 0-0.2 and 0.8-1). Of course this would be more visible with a standard histogram, which would show a kind of peak in the middle but with two high bars at the extremes for 0 and 1 (it can be seen from the continuous lines for these two features values that they appear much more frequently).

One may also note on this graph that there is some kind of underlying discrete distribution of the values: very clearly for values 0 and 1, but also from all the white horizontal lines which show that some values seldom exist in the data.


Neural network regression is not dynamic enough to predict target range?

I've solved it myself. Actually the whole premise was incorrect. It was tempting to look at the plot and think that it just takes a linear transformation to change the slope of the line, but that's because one axis of the plot is the target data!
My claim was that the model should stretch the data along the y-axis (around the mean) because it's the obvious thing to do. But is it really? Have a look at these 4 examples. According to my "obvious fudge factor", the two points on the right should have been stretched up, and the two on the left should have been stretched down.

But then what would that mean for all the other points that were on target? It would shift them off target and overall the model would perform worse. Here's what would happen to some of the points stretched up away from the mean.

The tempting human thing to do, which is what got me, is to include the x-axis in your perception of the problem. But we have to remember that it's not there for the model.
Every time the model predicts deviations from the mean it needs to have a pretty strong reason to do it. If it hasn't actually found the real reason to do so, then it shouldn't, otherwise the overall performance would be worse.
As the model gets better, we don't just see a better R^2 (dots line up better), but we also see the slope increase, because the model gets more confident about deviating from the mean.


Display Images (url) Inside Pandas Dataframe

Actually the solution2 worked; I just had to be a bit more patient. I am posting it here in case someone have difficulties, like me, getting this to work:

import pandas as pd
from IPython.display import Image, HTML

def path_to_image_html(path):
    '''
     This function essentially convert the image url to
     '' format. And one can put any
     formatting adjustments to control the height, aspect ratio, size etc.
     within as in the below example.
    '''

    return ''

HTML(df.to_html(escape=False ,formatters=dict(column_name_with_image_links=path_to_image_html)))

Note that df is your actual dataframe name, column_name_with_image_links is the column name that contains all image urls, and path_to_image_html is the above function.


You could use the following function to determine how much memory your model requires:

def get_model_memory_usage(batch_size, model):
    import numpy as np
    from keras import backend as K

    shapes_mem_count = 0
    for l in model.layers:
        single_layer_mem = 1
        for s in l.output_shape:
            if s is None:
                continue
            single_layer_mem *= s
        shapes_mem_count += single_layer_mem

    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

    total_memory = 4.0*batch_size*(shapes_mem_count + trainable_count + non_trainable_count)
    gbytes = np.round(total_memory / (1024.0 ** 3), 3)
    return gbytes

I copied this from ZFTurbo's answer on this post.


How does a query into a huge database return with negligible latency?

Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, and that make it feasible to have such low latency on queries from all these TBs of data in different machines:

distributed computing: by being distributed does not mean that the indexes are simply distributed in different machines, they are actually replicated along different clusters, which allows for lots of users performing different queries with low retrieval time (yes, huge companies can afford for that much of machines);
caching: caches tremendously reduce execution time, be it for the crawling step, for the retrieval of pages, or for the ranking and exihibition of results;
lots of tweaking: all the above and very efficient algorithms/solutions can only be effective if the implementation is also efficient. There are tons of (hard coded) optimizations, such as locality of reference, compression, caching; all of them usually appliable to different parts of the processing.

Considering that, lets try to address your questions:

  but I imagine it infeasible for the results of every single possible query to be indexed

Yes, it would be, and actually is infeasible to have results for every single possible query. There is an infinite number of terms in the world (even if you assume that only terms properly spelled will be entered), and there is an exponential number of queries from these n -> inf terms (2^n). So what is done? Caching. But if there are so many queries/results, which ones to cache? Caching policies. The most frequent/popular/relevant-for-the-user queries are the ones cached.

  wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs

Nowdays, with such highly developed processors, people tend to think that every possible task that must finish within a second (or less), and that deals with so much data, must be processed by extremely powerful processors with multiple cores and lots of memory. However, the one thing ruling market is money, and the investors are not interested in wasting it. So what is done?

The preference is actually for having lots of machines, each using simple/accessible (in terms of cost) processors, which lowers the price of building up the multitude of clusters there are. And yes, it does work. The main bottleneck always boils down to disk, if you consider simple measurements of performance. But once there are so many machines, one can afford to load things up to main memory, instead of working on hard disks.

Memory cards are expensive for us, mere human beings, but they are very cheap for enterprises that buy lots of such cards at once. Since it's not costly, having much memory as needed to load indexes and keep caches at hand is not a problem. And since there are so many machines, there is no need for super fast processors, as you can direct queries to different places, and have clusters of machines responsible for attending specific geographical regions, which allows for more specialized data caching, and even better response times.

  Does MapReduce help solve this problem?

Although I don't think that using or not MapReduce is restricted information inside Google, I'm not conversant about this point. However, Google's implementation of MapReduce (which is surely not Hadoop) must have lots of optimizations, many involving the aspects discussed above. So, the architecture of MapReduce probably helps guiding how the computations are physically distributed, but there are many other points to be considered to justify such speed in querying time.

  Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches?

The graph below presents a curve of how the kinds of queries occur. You can see that there are three main kinds of searches, each of them holding approximately 1/3 of the volume of queries (area below curve). The plot shows power law, and reinforces the fact that smaller queries are the most popular. The second third of queries are still feasible to process, since they hold few words. But the set of so-called obscure queries, which usually consist of non-experienced users' queries, are not a negligible part of the queries.

And there lies space for novel solutions. Since it's not just one or two queries (but one third of them), they must have relevant results. If you type in something much too obscure in a Google search, it shan't take longer to return a list of results, but will most probably show you something it inferred you'd like to say. Or it may simply state that there was no document with such terms -- or even cut down your search to 32 words (which just happened to me in a random test here).

There are dozens of appliable heuristics, which may be either to ignore some words, or to try to break the query into smaller ones, and gather the most popular results. And all these solutions can be tailored and tweaked to respect feasible waiting times of, say, less then a second? :D


Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

The power to model non-linear decision boundaries comes directly from the non-linear activation function. You can understand this when you see that concatenating N linear transformations (i.e. dense layers) is equivalent to a single linear transformation. This is the mathematical proof with 2 linear layers:
$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$
(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)


The shape should be 3d array --> (samples, timesteps, features)

It needs to add return_sequences=True in first two RNN layers.

model=Sequential()
model.add(SimpleRNN(1000,input_shape=(1,320*15),activation='relu', return_sequences=True))
model.add(SimpleRNN(1000, return_sequences=True))
model.add(SimpleRNN(1000))


Error rate of AdaBoost weak learner always bigger than 0.5?

As far as I understand, weak learners of AdaBoost should never yield a error rate > 0.5

This is only true for binary classification problems.  The simplest possible classifier would pick the majority class, ensuring that accuracy is always >= 0.5
For a problem with $N$ classes, the worst error rate for a weak learner will be $\frac{N-1}{N}$.

  Is it possible that my threshold for the error rate instead is 0.9 (accuracy 0.1), as I have 10 classes and literature mostly focuses on binary cases?

Yep that's exactly right.  Your weak learners will have, at worst, an error rate of 0.9.  Unsurprisingly, most of your learners have a better error rate better than random guessing, which explains the good performance of your ensemble.

Edit:

  Also the trees get worse from iteration to iteration

This is a common feature in boosting algorithms.  Each iteration, training samples are reweighted such that the mistakes from previous iterations are given more importance.  The net effect is that the learners in later iterations are specialized for classifying the examples that earlier learners got wrong.  Since later learners are specialized for a few hard-to-classify examples, they will have worse performance on the entire dataset.


RNN time-series predictions with multiple features containing non-numeric features and numeric features?

Possible ways to expose these categorical variables as part of a time-step to be fed to an RNN include producing One_Hot Tensors, Hash Table Representations, or Embedding Layer Outputs from the categorical fields and concatenating all of the feature tensors at each time-step to feed to your RNN.

If, instead of discrete categories, you're looking to interpret the strings that make up the fields as sequences themselves, you could encode them numerically at a word- or character level and pass them through a subsidiary RNN to produce the internal representation at each timestep. This would be appropriate only for cases where you have a very large variety among strings in those fields.

Apologies for the Tensorflow links if you're not using that framework.


Why are ROC curves better for imbalanced datasets?

The AUC reaches 0.5 because of this:

$Sensitivity = \frac{True Positives}{True Positives + False Negatives}$

$Specificity = \frac{True Negatives}{True Negatives + False Negatives}$

In your case:

$Sensitivity = \frac{99%}{99%+0%}=1$

$Specificity = \frac{0%}{0%+1%}=0$

$1-Specificity = 1$

Remember that the X axis is 1-Specificity.

The point $(1,1)$ is in the diagonal line that makes the AUC equal to 0.5

If you try this exercise with any other rule like "Saying 'yes' to dementia for all patients" you will get the point $(0,0)$ and joining these points you will have the diagonal line.


What does the Ip mean in the Bayesian Ridge Regression formula?

$\mathcal{N}$ does indeed denote a (multivariate) normal / Gaussian distribution. $I_p$ is just an identity matrix of dimension $p$. So this a matrix with $\lambda^{-1}$ along the diagonal. Read this as the covariance matrix, so this is a spherical Gaussian (0 covariance between different dimensions) where each variable has variance $\lambda^{-1}$.


Pandas datetime error when reading from excel file

Using pandas, first make sure you have a datetime column:

df['DT'] = pd.to_datetime(df['DT'])

To remove the milliseconds, a possible solution is to use round to obtain a specified frequency (in this case seconds).

df['DT'] = df['DT'].dt.round(freq='s')

Depending on the wanted final result, ceil (to always round up) or floor (always round down) could be more suitable.


Multiclass classification with large number of classes but for each user the set of target classes is known

so it sounds like you have historical transaction data. If you do, you probably want to train a model using store label as your dependent variable, and all the others independent variables.

set.seed(100)
items <- c("milk", "cheese", "steak", "apple", "eggs")
df <- data.frame(trans_id = 1:100,
                       cust_id = sample(1:10, 100, replace = TRUE),
                       item = sample(items, 100, replace = TRUE)
                       )

df$store <- as.factor(sapply(df$cust_id, function(x) {
    sample(letters[cust_id:(cust_id+2)],1)
  }))

head(df)
##  trans_id cust_id   item store
##1        1       4 cheese     e
##2        2       3 cheese     f
##3        3       6   milk     f
##4        4       1 cheese     f
##5        5       5  steak     d
##6        6       5  apple     f

#train General linear model
glm(formula = store ~., family = binomial(link = "logit"),
     data = df)

##all:  glm(formula = store ~ ., family = binomial(link = "logit"), data = df)
##
##Coefficients:
##(Intercept)     trans_id      cust_id   itemcheese
##   1.320095    -0.007399    -0.066363     0.024632
##   itemeggs     itemmilk    itemsteak
##   0.084018     0.047022     1.069399

library("e1071")
#train naive bayes
naiveBayes(formula = store ~., data = df)

Depending on business constraints you may or may not want to include customer id as a predictor variable (include if you will only be predicting stores for a finite known pool of customers; don't include if your model needs to generalize across customers). Similarly, if this were a real data set, you probably wouldn't include a transaction id as a predictor variable, because it won't generalize to new cases (transaction ids are unique)


However, some of the parameters cannot be added. In our example, such a parameter can be the plantation age, plantation location, cultivar of the fruit, etc

Can you explain why these can't be added?

There exist many ways to encode categorical information, such as plantation location, such that a ML model can interpret it. In my use-cases, in supply-chain, I deal with thousands of unique location observations, where in I use a geo-encoding API to return latitude/longitude pairs to feed into my models.

In your case, where in you only have 6-factorial observations (that may be wrong), what I would do is to generally categorize your categorical features. So for location, instead of "Florida," I would say "South-East," and then lump all south-eastern locations together. Since you'll have low cardinality in feature size, binary or one-hot encoding may be useful for you.


Make a new column equal to „company“. Use the grep to replace military by „military“, use the grep to replace private by „private“. Change the new column to a factor.


I have tried something similar once. I used an approach which you would not expect, but which gave some surprisingly good results.

I used the NMT (Neural Machine Translation) model in Tensorflow. There are some examples online for Vietnamese->English translation. I changed it to translate from "old job title" to "new job title", and trained it on my dataset. This is easy to do and you don't need to modify any code, except maybe tweak the model a little (e.g. length of document is now shorter than in the machine translation example).

The drawback of this approach is that only the most recent job title is taken into account when using it for inference. However you could try concatenating all previous job titles and other data (schools etc) as the input text, to produce a single job title as output, and this way you could make sure you are making use of their entire career path.

I would suggest tweaking aspects of the architecture (number of layers, number of previous jobs used, off-the-shelf word2vec vs domain specific trained word2vec etc) until you get the best performance against whatever evaluation metric you are using.

The other approach you could take would be to take the doc2vec of each title, and train some kind of RNN/LSTM to predict the next vec given all previous vecs. Then you need a postprocessing stage on the output to convert the output vec to a text. Unfortunately I don't know of a shortcut here to avoid building your model architecture from scratch, so this approach is more work.

Of course there are other ways of solving the problem but since no-one else has answered yet I thought I'd give my suggestion.


How to save Tensorflow predictions to data frame?

Thanks to Ankit Seth I found the answer.

#this returns output layer of forward prop with trained parameters of the model.
pred =  forward_propagation(X, parameters)
predictions_sigm = pred.eval(feed_dict = {X: X_test})
predictions_sigm = sigmoid(predictions)

#pred.eval returns linear regression layer for each class separately, so we need to pick index of maximum of them. I don't use softmax layer, since the output should be the same.

#init list of classes
y_list = []

for i in range(predictions_sigm.shape[1]):
    class_list = []
    class_list.append(predictions_sigm[0][i])
    class_list.append(predictions_sigm[1][i])
    class_list.append(predictions_sigm[2][i])
    class_list.append(predictions_sigm[3][i])

    #get index of maximum value of sigmoid. it will correspond to class
    y = np.argmax(class_list)
    y_list.append(y)

   #then append y_list to dataframe
   df['predicted_y'] = y_list


Which Big-Data Frameworks have most simple interfaces?

I think that it is impossible to answer this question comprehensively, at least for the following reasons:

big data frameworks have different goals and target different knowledge domains, so the comparison simply doesn't make much sense;
most big data frameworks (and other programming frameworks, for that matter) have multiple interfaces, and frequently those sets of interfaces are significantly different (the intersection is small), so there is a risk of comparing apples and oranges;
trying to compare anything (in this case, interfaces), using simplicity as a criterion, involves a significant amount of subjectivity - what one person perceive as very simple, another person might find quite complex;
the variety and the number of big data frameworks is mind-boggling (for example, see https://github.com/onurakpolat/awesome-bigdata); the same applies to a related topic of machine learning frameworks (for example, see https://github.com/josephmisiti/awesome-machine-learning);
corollary from the points above: a comprehensive comparison (considering all the above-mentioned issues) would go far beyond the scope of a single answer on this site, in volume and effort - it would be more like a long research paper, a book chapter or even a book.


Determine useful features for machine learning model

I suggest taking a look at this page for some more ideas:

Feature Selection

That being said a couple of ideas that come to mind quickly, is to:

use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.
Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.
Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.

Generally, these methods will be "expensive" as you are fitting multiple models to get you where you need to go.


Is there any use to running Pandas on Spark?

There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects.

The only interest I have found using Spark with pandas is when you want to load a local CSV / Excel dataset and then transform it into a spark dataframe. The "createDataFrame" method handles this approach.

>>> sqlContext.createDataFrame(pandas.DataFrame([[1, 2]])).collect()
[Row(0=1, 1=2)]


Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its values in a specific convenient range. We don't follow this practice for theoretical guarantees but for practical reasons. The same goes with clipping $Q$ value functions in Q-learning combined with NNs. Of course, there are some drawbacks with these approaches but in general the algorithm behaves better as the backpropagation does not lead your network weights to extreme values. Please take a look at this excellent post by Andrej Karpathy (I attach the part related to your question as a blockquote) which gives additional insights:

  More general advantage functions. I also promised a bit more discussion of the returns. So far we have judged the goodness of every individual action based on whether or not we win the game. In a more general RL setting we would receive some reward $r_t$ at every time step. One common choice is to use a discounted reward, so the “eventual reward” in the diagram above would become $R_t=∑^∞_{k=0}γ^kr_{t+k}$, where $\gamma$ is a number between 0 and 1 called a discount factor (e.g. 0.99). The expression states that the strength with which we encourage a sampled action is the weighted sum of all rewards afterwards, but later rewards are exponentially less important. In practice it can can also be important to normalize these. For example, suppose we compute $R_t$ for all of the 20,000 actions in the batch of 100 Pong game rollouts above. One good idea is to “standardize” these returns (e.g. subtract mean, divide by standard deviation) before we plug them into backprop. This way we’re always encouraging and discouraging roughly half of the performed actions. Mathematically you can also interpret these tricks as a way of controlling the variance of the policy gradient estimator. A more in-depth exploration can be found here.


Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True.

One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a transformer which you can apply to your training and test set separately if you specify the total number of categories. This doesn't work with get_dummies ,for example, if the training set misses categories present in the test set.

You can still delete categories by simply deleting columns from the resulting numpy array (e.g. using n_values_ or feature_indices_ to see which columns correspond to the same feature). Some models work regardless, for example tree-based models. Also, L1 regularization can often set redundant features to zero (see Lasso regression).


Methodology for parallelising linked data?

I'd look into applying some kind of sliding window to you data so you can process one subset at the time. By adjusting the window overlapping with previous chunks you can control how much context have on each batch.

The implementation details will depend on what kind of data structure you use, but since you mentioned a grid in which you want to smear the data, a convolution+pooling approach might do the trick. It's very easy to run in parallel and can be super efficiently on modern GPUs.

Also, you example suggests that the result of your chunk processing would be some kind of subsampling. If the data is too large you could subsample it first. There are many ways to do it depending on which criteria is acceptable for discarding data points so you'll have to dig into it to figure out which method is best for your use case


How should the hyper parameters be defined for the other algorithms defined for sklearn_crfsuite.CRF

C1 and C2 are coefficients for L1 and L2 regularization, respectively. You can use the same definition of params_space regardless of the optimization algorithm.


You are correct that "stacking LSTMs" means to put layers on top of one-another as in your second image.

The first picture is a "bi-directional LSTM" (BiLSTM), whereby we can analyse a point in a series (e.g. a word in a sentence) from both sides. We care about the context of that point.

The most common example I know of is within NLP. Here we want to know the representation of a word in a gap, how it is found between other words. If we have the entire sentence, we can look at the words before and the words after our word. In this case, we could use a bi-drectional LSTM to process the sequence in the opposite direction, which your first diagram shows.

Let's play a game, and say you need to guess the missing word in this text snippet:

  i need to review an __________ ...

What could it be? "article", "iPad", "aerial image" ?

Here is the solution:

  i need to review an article, ...

It was incredibly hard to get that right - perhaps impossible! Well, maybe not if you have some context with it. How about I give you both sides of that snippet:

  i need to review an ________, for tomorrow's newspaper.

A BiLSTM would be fed the sentence from both sides, thus letting it see some more context to understand each word.

Have a look at this article, which eventually get the bi-directional networks.
Here is a similar question to yours with a few nice answers.

In time-series data, such as device readings from IoT devices or the stock market, using such a bi-directional model wouldn't make sense, as we would be violating the temporaneous flow of information i.e. we cannot use information from the future to help predict the present. That isn't a problem in text analysis, voice-recordings or network analysis on sub-network traffic flow.


Interpreting fraction of zero weights in TensorFlow

If you are performing linear (logistic) regression your weights are simply your $\beta_i$. If none of them are $0$ that simply means all features are 'important' to some degree.


How to extract characteristics from text using machine learning?

What you need to look for is called "Named Entity recognition". From Wikipedia

  Named-entity recognition (NER) (also known as entity identification,
  entity chunking and entity extraction) is a subtask of information
  extraction that seeks to locate and classify named entity mentions in
  unstructured text into pre-defined categories such as the person
  names, organizations, locations, medical codes, time expressions,
  quantities, monetary values, percentages, etc.

There are already trained models for that, but most of them are for generic usage. For example in Python

import spacy

nlp = spacy.load('en_core_web_sm')
doc = nlp('European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices')
print([(X.text, X.label_) for X in doc.ents])

The output is

[('European', 'NORP'),
('Google', 'ORG'),
('$5.1 billion', 'MONEY'),
('Wednesday', 'DATE')]

Source of code: TowardsDataScience

In your case, you have to either train an NER yourself for phone specifications or find one that is available in public.


Aspect based sentiment analysis using machine learning approach

Aleksandar Blekh has given some really nice links about the big picture of how to do sentiment analysis; I'll try to provide some links to software and talk about the nitty-gritty of how to make it work. I'll point you to the example using scikit-learn (http://scikit-learn.org/stable/ ), a machine learning library in Python.

You would first want to take your dataset and load it into scikit-learn in a sparse format. This link (http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html ) gives examples of how to load text in a bag-of-words representation, and the same module (scikit-learn.feature_extraction.text) can also count n-grams. It then describes how to run Naive Bayes and SVM on that dataset. You can take that example and start playing with it.


Prediction for Current month based on last month's Labels

This is a common problem. In a perfect world, training data is perfectly representative of the data that will need to be predicted. However, we don't live in a perfect world and we need to work with approximations. You don't have data about Feb-20 but you have data about Jan-20, Dec-19, etc. You can still use this data! Nothing is stopping you.


How to set batch_size, steps_per epoch and validation steps

batch_size determines the number of samples in each mini batch. Its maximum is the number of all samples, which makes gradient descent accurate, the loss will decrease towards the minimum if the learning rate is small enough, but iterations are slower. Its minimum is 1, resulting in stochastic gradient descent: Fast but the direction of the gradient step is based only on one example, the loss may jump around. batch_size allows to adjust between the two extremes: accurate gradient direction and fast iteration. Also, the maximum value for batch_size may be limited if your model + data set does not fit into the available (GPU) memory.
steps_per_epoch the number of batch iterations before a training epoch is considered finished. If you have a training set of fixed size you can ignore it but it may be useful if you have a huge data set or if you are generating random data augmentations on the fly, i.e. if your training set has a (generated) infinite size. If you have the time to go through your whole training data set I recommend to skip this parameter.
validation_steps similar to steps_per_epoch but on the validation data set instead on the training data. If you have the time to go through your whole validation data set I recommend to skip this parameter.


Find Cluster Diameter and Associated Cluster Points with KMeans Clustering (scikit learn)

For the time being, I've prepared a workaround solution:

#iris example
iris = datasets.load_iris()
x = iris.data
y = iris.target

estimator = KMeans(n_clusters=3)
y_kmeans = estimator.fit_predict(x)

To get the clusters' radii you can use the following code snippet:

#empty dictionaries

clusters_centroids=dict()
clusters_radii= dict()

'''looping over clusters and calculate Euclidian distance of
each point within that cluster from its centroid and
pick the maximum which is the radius of that cluster'''

for cluster in list(set(y)):

    clusters_centroids[cluster]=list(zip(estimator.cluster_centers_[:, 0],estimator.cluster_centers_[:,1]))[cluster]
    clusters_radii[cluster] = max([np.linalg.norm(np.subtract(i,clusters_centroids[cluster])) for i in zip(x[y_kmeans == cluster, 0],x[y_kmeans == cluster, 1])])

It will give you this:

Please note K-means:

Implicitly assumes all clusters have the same radius
Separates the data into Voronoi-cells (which can be seen from here as well).
Cluster points (circles) can overlap (it is how it is defined).

If you want to relax the shape of the clusters (not strictly spherical or circles like K-means), you should perform Gaussian mixture models.

Appendix (To Reproduce the above Visualization):

#Visualising the clusters and cluster circles

fig, ax = plt.subplots(1,figsize=(7,5))

plt.scatter(x[y_kmeans == 0, 0], x[y_kmeans == 0, 1], s = 100, c = 'red', label = 'Iris-setosa')
art = mpatches.Circle(clusters_centroids[0],clusters_radii[0], edgecolor='r',fill=False)
ax.add_patch(art)

plt.scatter(x[y_kmeans == 1, 0], x[y_kmeans == 1, 1], s = 100, c = 'blue', label = 'Iris-versicolour')
art = mpatches.Circle(clusters_centroids[1],clusters_radii[1], edgecolor='b',fill=False)
ax.add_patch(art)

plt.scatter(x[y_kmeans == 2, 0], x[y_kmeans == 2, 1], s = 100, c = 'green', label = 'Iris-virginica')
art = mpatches.Circle(clusters_centroids[2],clusters_radii[2], edgecolor='g',fill=False)
ax.add_patch(art)

#Plotting the centroids of the clusters
plt.scatter(estimator.cluster_centers_[:, 0], estimator.cluster_centers_[:,1], s = 100, c = 'yellow', label = 'Centroids')

plt.legend()
plt.tight_layout()
plt.savefig('kmeans.jpg',dpi=300)


How to compute document similarities in case of source codes?

Why not use all non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, I have a tendency in python to copy entire import blocks from one script to another, whether or not I actually need all the imports or not. If someone uses a copied import or include statement, then their imports will all be in the same order. If you track the order of imports, you may be able to find patterns.

There are also issues of whether you use spaces or tabs for indentation, preferred number of spaces or tabs, etc. For languages with braces, you can see if braces are used following an if statement, or at the start of the next line. I believe this will give you enough 'handprints' to identify people distinctly.

Once you do this, you can use multilevel clustering to attempt to assign documents first to a group of possible originators (i.e. all originators who use a certain IDE or text editor). Then you can go through and look for certain patterns within each group, clustering again within each cluster.


Train MLP Neural Network on time series data?

So I posted this same question on machine learning mastery post about removing trends and seasonality difference transform for time series data. And Jason Brownlee responded to my questions:

Yes, you can use MLP, CNN and LSTM. It requires first converting the
data to a supervised learning problem using a sliding window:

https://machinelearningmastery.com/time-series-forecasting-supervised-learning/

Then evaluating models in a way that respects the temporal ordering of
obs, called walk-forward validation:

https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/

You can see tens of tutorials and my book on this here:

https://machinelearningmastery.com/start-here/#deep_learning_time_series
MY REPLY TO THIS POST
Ben February 12, 2021 at 8:09 am #

Cool thanks for all the info. So if I used multivariate sliding window
for MLP NN, is Ok when training the model that shuffle_data == True?
or should I not shuffle training data…? Thanks so much!

Jason's REPLY TO MY QUESTION ABOVE
Jason Brownlee February 12, 2021 at 1:36 pm #

Yes, as long as all data in the training dataset is in the past
compared to the test set.


How to do hidden variable learning in Bayesian Network with Python?

While this model could be implementable in the libpgm library (it seems to have quite rigid interface tailored to several special models, though), it will not allow you to reproduce the results of this paper. In the paper authors actually do something way simpler than Bayesian Inference – they only perform maximum-aposteriori (MAP) inference, which is way easier, but does tell you how confident the model is.

You don't have to code optimization algorithms for MAP inference by yourself, these days there are powerful and general packages that do almost everything for you. You can use automatic differentiation features of TensorFlow / PyTorch – they will compute the gradients and run optimization algorithm for you. You can also use numpy.autograd to get gradients for free, and then run any optimizer you like (check out scipy.optimize). The autograd + scipy.optimize pair is particularly neat if you wanna use more advanced optimization methods, e.g. 2nd order ones.

All these methods will give you values (point estimates) of $w, \theta, z$ that you can use to make inferences. However, if you want to have more than point estimates, and would like to do find all possible values (along with their probabilities) and maybe Bayesian Model Averaging, you'll need more powerful libraries enabling probabilistic programming. There are Edward (built upon TensorFlow), Pyro (built upon PyTorch), Stan and others.

If you feel overwhelmed by the choices, I recommend going with PyTorch.


Neural network with flexible number of inputs?

Yes this is possible by treating the audio as a sequence into a Recurrent Neural Network (RNN). You can train a RNN against a target that is correct at the end of a sequence, or even to predict another sequence offset from the input.
Do note however that there is a bit to learn about options that go into the construction and training of a RNN, that you will not already have studied whilst looking at simpler layered feed-forward networks. Modern RNNs make use of layer designs which include memory gates - the two most popular architectures are LSTM and GRU, and these add more trainable parameters into each layer as the memory gates need to learn weights in addition to the weights between and within the layer.
RNNs are used extensively to predict from audio sequences that have already been processed in MFCC or similar feature sets, because they can handle sequenced data as input and/or output, and this is a desirable feature when dealing with variable length data such as spoken word, music etc.
Some other things worth noting:

RNNs can work well for sequences of data that are variable length, and where there is a well-defined dimension over which the sequences evolve. But they are less well adapted for variable-sized sets of features where there is no clear order or sequence.

RNNs can get state-of-the-art results for signal processing, NLP and related tasks, but only when there is a very large amount of training data. Other, simpler, models can work just as well or better if there is less data.

For the specific problem of generating MFCCs from raw audio samples: Whilst it should be possible to create a RNN that predicts MFCC features from raw audio, this might take some effort and experimentation to get right, and could take a lot of processing power to make an RNN powerful enough to cope with very long sequences at normal audio sample rates. Whilst creating MFCC from raw audio using the standard approach starting with FFT will be a lot simpler, and is guaranteed to be accurate.


Predicting future value with regression Model

You should distinct between a time series prediction, where from a known history of some attribute the future is predicted and model prediction where based on the predictor variables the target variable is calculated.

In your case you could combine both approaches, i.e. use time series prediction on the customer balances and apply the regression model to calculate the profit on the result.


GridSearchCV using Random Forest Reg Pipeline

The problem seems to be that your pipeline uses a fresh instance of RandomForestRegressor, so your param_grid is using nonexistent variables of the pipeline.  There are two choices (I tend to prefer the second):

Use rfr in the pipeline instead of a fresh RandomForestRegressor,  and change your parameter_grid accordingly (rfr__n_estimators).

Change param_grid to use the lowercased name randomforestregressor__n_estimators; see the docs on make_pipeline:

it ... does not permit naming the estimators. Instead, their names will be set to the lowercase of their types automatically.


How to version data science projects with large files

I have used dvc.  It has data versioning though I do not use that often. I tend to use the makefile (pipeline) features.


You might need an encoder-decoder framework that can be implemented using two LSTM/GRU models. The Encoder takes the first phrase in, outputs the last (summary) hidden layer, it is taken by the first Decoder state, alongside BOS token, outputs the second token, uses it to predict the third, and so on until you reach the end of the (response) phrase.


Bagging: Also known as Bootstrap Aggregation is an ensemble method. First, we create random samples of the training data set (sub sets of training data set). Then, we build a classifier for each sample. Finally, results of these multiple classifiers are combined using average or majority voting. Bagging helps to reduce the variance error.

Boosting provides sequential learning of the predictors. The first predictor is learned on the whole data set, while the following are learnt on the training set based on the performance of the previous one. It starts by classifying original data set and giving equal weights to each observation. If classes are predicted incorrectly using the first learner, then it gives higher weight to the missed classified observation. Being an iterative process, it continues to add classifier learner until a limit is reached in the number of models or accuracy. Boosting has shown better predictive accuracy than bagging, but it also tends to over-fit the training data as well.

Algorithms work on these techniques

Bagging: Random Forest

Boosting : Ada Boost, Gradient Boosting, XGBoosting, etc


Naive Bayes for Categorical Features (Non Binary)

In a recent scikit-learn release (v0.22.1), the developers have added Categorical Naive Bayes to their list of Naive Bayes implementations:

https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.CategoricalNB.html

You could also use my implementation of categorical and/or Gaussian Naive Bayes:

https://github.com/remykarem/mixed-naive-bayes


How to find average lag time with variance & confidence of two time series

In a first approach, What you are looking for might be lagged correlation. You shift the trailling time serie by some variable time t and you look at correlation with the first time serie while varying the lag. The value of the lag which yield the biggest correlation is a good candidate for an estimation of time delay betweeen the two.
If you want more advanced statitics like variance (meaning that you have a variable lag) it might get more complex as you would have to identify which peak in the first time serie correspond to wich one in the second. Depending on your format of your time serie it might be easy or not. I would suggest to try a simple approach first : for each peak of your first time serie try to find the corresponding one in the second (first peak that appears after it for exemple). It might get complicated if some peaks overlap or the delay is too variable. Another option is to use more complicated things like Dynamic Time Warping Algorithms.


How to extract true positives data (complete row with data) after training and testing from test dataset?

One thing you can do is browse your prediction vector, get the indexes of "1" responses, and then check those indexes in y_test. if your y_test[index] is also a "1" class, then select the row by index in X_test
I tested this, it works for me. In my case, my X and y are pandas.DataFrame.
    import pandas as pd
    from sklearn.linear_model import LogisticRegression
    import numpy as np

    X_train = pd.read_csv("saves/cv_sets/X_train1.csv", sep=";", encoding="latin1")
    X_test = pd.read_csv("saves/cv_sets/X_test1.csv", sep=";", encoding="latin1")
    y_train = pd.read_csv("saves/cv_sets/y_train1.csv", sep=";", encoding="latin1")
    y_test = pd.read_csv("saves/cv_sets/y_test1.csv", sep=";", encoding="latin1")

    clf = LogisticRegression(class_weight="balanced", solver='lbfgs', C=0.1)
    model = clf.fit(X_train, y_train)

    pred = model.predict(X_test)

    pred1 = np.where(pred==1)

    TP_Indexes = []
    for k in pred1[0]:
        if(y_test.iloc[k][0] == 1):
            TP_Indexes.append(k)

    X_test_TP = X_test.iloc[TP_Indexes]


Pandas column count mismatch after insert into dataframe

Can you share some of your data frame i would recommend using merge instead of insert if you have a common column let's say ID for example your code will look like this:

mergedStuff = pd.merge(df1, df2, on=['id'], how='outer')
mergedStuff.head()

as simple as that


Chicago Taxi dataset is another option with large number of records and multiple feature types: categorical, numerical and buckets


Difference between interpolate() and fillna() in pandas

fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to the index of rows of a pd.DataFrame or on the name of the columns in the form of a python dict.

But interpolate is a god in filling. It gives you the flexibility to fill the missing values with many kinds of interpolations between the values like linear (which fillna does not provide) in the example provided below and many more interpolations possible. For example

>> import pandas as pd, numpy as np
>> df = pd.Series([1, np.nan, np.nan, 3])
>> df.interpolate()
0    1.000000
1    1.666667
2    2.333333
3    3.000000
dtype: float64

Pandas documentation on fillna and interpolate is very clear on this.


Can I save only some VGG19's layers into a .H5 file?

after creating the model you can create another model as below ( I created model till 8 layers)

model = Model(vgg19.input, vgg19.layers[8].output)
model.save('./style_transfer/st.h5')

You can also use post-training Quantization techniques to reduce the size of the model to deploy in mobile/IoT devices. please check tensorflow documentation here


How to treat Compass data in random forest regression

One option is to map the ordinal categories to numerical degrees. This would allow the machine learning model to learn the relative relationships. The machine learning model might interpolate values which may or may not be useful.

One common mapping is:

n  ->   0
ne ->  45
e  ->  90
se -> 135
s  -> 180
sw -> 215
w  -> 270

The book "Statistical Analysis of Circular Data" by Fisher goes into greater detail about more options.


When I think about how to implement a certain ML or data mining process in an OOP (like Java), I usually go to see how smarter people than me designed their system.
In this case, I'd see how Weka, RapidMiner, JAVA ML or others decided to tackle this problem.

In your case of feature/attribute selection, i'm adding a link to Weka's API.
If you want to better understand how it's done, you should download Weka and play with the source code.

http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/AttributeSelectedClassifier.html

Without getting into specific implementations, I think that a linked list of any sort would do the trick, since you want to select a subset of all features.


What you have is a cyclical feature, i.e. a feature which takes values which repeat cyclically in time. With that trivial encoding, for any algorithm, Sunday will be one day from Saturday, but 6 from Monday. It should be one day, also.

The proper way to handle cyclic features is to encode them with a sine and a cosine variable. So instead of a feature weekday, you should have weekday_sin and weekday_cos, which are calculated as:

data['weekday_sin'] = np.sin(2 * np.pi * data['weekday']/7)
data['weekday_cos'] = np.cos(2 * np.pi * data['weekday']/7)

Although I can't assure you that this will work in your case, it is definitely a conceptual error you're dealing with.

For more in-depth explanation with some plots which make it a lot easier to understand, you can read this: https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning

Hope this helps!


I've found a dataset the meet the specification here:

https://www.kaggle.com/atulanandjha/lfwpeople/data#lfw-funneled.tgz

Some people in this dataset have only very few image.

We can create a new one dataset using the following bash command, in order to exctract only the people that contains more than 5 images:

# Extract only the people that have more than 5 photos (-gt 5)
for i in $(ls); do a=$(ls $i |wc -l); if [ "$a" -gt 5 ]; then echo $i ; fi ; done > people_ok
# Create a directory for store the images
mkdir -p /tmp/faces
# Copy the filtered directory in the new one
for i in $(cat people_ok  | xargs echo -n) ; do cp -r $i /tmp/faces/ ; done


How to manually calculate AUC and Accuracy

For accuracy, TP+TN/total is it right way to calculate?

If your problem is binary classification, then yes.

  How to calculate AUC using some formula? What are the parameters
  required and what formula to use?

Check this question.


Welcome to the community.

Definition: Let's define the feature/examples (examples are also called data points). Usually in tabular dataframes when folks doing data analytics, data science etc., features are in columns, and data points of each particular feature are populating the the rows. Here is a simple dataset snapshot showing this definition:

Variance: We are interested in measuring the statistics for each feature. Because in Machine Learning paradigm, those features that have very narrow variance often are not that informative for predicting the target. In this definition it would be along the rows. For example, income feature, we would like to know what is the mean of income, or variance of income for all data points (along rows).

Please NOTE this is just a convention that is very common among Python users and many standard libraries/tutorials out there. As long as you know what you are doing, you can swap these axes. I have seen that some old teaching materials having an opposite definition, meaning that features are in rows, and data points are spread in columns like your definition. If you follow this convention, obviously you have to calculate the variance along column axis, not rows anymore.


Create a new column in Pandas Dataframe based on the 'NaN' values in another column

Try this -

Column is your column with NaN values and column B is the new column

df['B'] = df['A'].isnull()*1


How to deal with time series of multi-source energy in order of classifcation?

OK, let us say you have 24 data points per day, in a year you would have 8700~ data points. As an initial analysis you would likek to classify the data points in 2 classes, 'summer energy', 'winter energy'. Notice I am refering to data points and not time series. For this analysis, on way is not to take a time series view but a collection of datapoints view. Once you do that, it becomes a unsupervised classification problem. You could use K-Mean, where the value of K is 2. You could also use neural network based models like Adaptive Resonance Theory networks. As for feature selection, take all the features that you receive from the sensor. It could additionally help to center and scale the data.

In my opinion, if you want to classify the data into 2 classes, timeseries analysis is not immediately required, you could employ the method mentioned above and see how the results are.

EDIT 1:
As I undestand (and assume), you have 4 sensors placed at 4 points of a single entity, e.g. an office. Each sensor output per hour is one of your dimensions. Hour of the day is also your data dimension. As an example, if you have 4 sensors, let us call them A,B,C,D, then your data point will look like:

Datapoint 1::
Vector position 0: value of A sensor, 1.0
Vector position 1: value of B sensor, 1.5
Vector position 2: value of C sensor, 0.5
Vector position 3: value of D sensor, 3.5
Vector position 4: hour of day (24),  14

You will have 720 such data points for one month. You should apply any clustering algorithm on the data points.

Few suggested clustering algorithms (there are many more):
1) K-Mean with K value as 2
2) ANN based approach

EDIT 2:

You could explore the ANN approach, you could make a 3 layer feed-forward network with 4 inputs and 2 outputs. The inputs would be the data point I mantioned above, however, it would be best to normalize the values ( here are some techniques) between 0 and 1. The output in your case will be 2 neurons, one representing summer and another winter, the input vector would be the datapoint vector and the output a vector of length 2, eg [0,1] for winter ideal valu & [1,0] for summer ideal value.

In another approach you could use K- Nearest Neighbour, in this case data normalization may not be strictly (technicaly) required.

For ANN library, you could use R/Python/Java, reference to one Java library is here.

For KNN, there are many library options, one reference is here


How to get a dataframe values in one single column for the following dataset?

You are not getting 2 columns. You are getting 2 rows.

The function alpha = (np.log([dataset.m38,dataset.m78])/np.math.log(38,78.7)) returns a 2-dimensional array. Putting in:

         m38     m78
      0  4.4717  4.8745
      1  4.4569  4.6491
      2  4.5101  4.7262
      3  4.4407  4.8234
      4  4.1184  4.3862

gives us: [[0.41174795 0.41083658 0.4140986  0.40983552 0.38912198]
           [0.43545842 0.42244323 0.42696487 0.43256132 0.40644075]]

This is due to the fact that np.log([dataset.m38,dataset.m78]) returns a 2-dimensional array.

Regarding the flipped columns/ row issue: Once your function doesn't return a 2-dimensional array, it will most likely return a Pandas Series which will have one column and 8000 rows.

If you want to turn that into a DataFrame, you can use
alpha = pd.DataFrame({'alpha':alpha})

Think about what your formula does. Putting in [dataset.m38,dataset.m78] will do the calculation for both columns seperately resulting in 2 different arrays.


Which ML algorithm to use if we have categorical data, numeric data, derived data (derived from) other variable in our data set?

This question is way too vague, you just describes pretty much a standard situation. Which algorithm / method you use depends entirely on your objective and data. You should provide more information if you want a specific answer. Since your dependent variable is binary you are interested in classification, popular algorithms are Random Forests, Logistic Regression, K-means, etc.


How is the concept of data different for different disciplines?

Data is, at it's most basic reduction, a raw element of something. Data is a raw "thing" that exists in any form from which we can analyze it and construct intelligence. When I was an Intelligence Analyst, we used to define data as "anything and everything that could be used to construct a hypothesis."

Thus, data for any discipline is interchangeable; as a sociologist, I have a vector of discrete variables indicating ethnicity, as an economist I have a vector with housing prices, and as an anthropologist I have a vector of tablet names used in some long-gone civilization.

Data is data.


Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN

Policy distillation is an excellent way to efficiently reduce number of parameters and maintain the quality of the original (large) network.

According to the paper, while having orders of magnitude less numbers, distilled models can actually improve score from 84% up to 124% and above.

The basic idea is that we train a new, smaller network from scratch, to predict what the original (large) network would produce.

An analogy would be: it's a lot easier to assign the desired mindset to a young child, than to update an old person's way of thinking.


Inputs required for Random Forest Regressor and ways to improve performance

A random forest trained on data whose labels are all positive integers cannot produce any prediction less than 1. So yes, the model will be less accurate.

I would suggest including a random subset of the rows with 0 sold, so that the forest can learn a (hopefully representative) pattern to them.  I'd also probably weight them higher in the fitting algorithm, to reproduce a more faithful average of items sold (in each leaf).


Dataset for Named Entity Recognition on Informal Text

As I understand it, these are the properties that you're seeking in a sample dataset:

Text data
It should be informal, i.e. have typos, slang, and basically something not professionally edited
Something other than Twitter (I don't blame you, Twitter is a useful yet way overused example datasource in text mining)

Here are some recommendations:

Emails from the SpamAssassin corpus -- note that both "ham" (non-spam) and spam datasets are available
microblogPCU data set from UCI, which is data scraped from the microblogs of Sina Weibo users -- note, the raw text data is a mix of Chinese and English (you could perform machine translation of the Chinese, filter to only English, or use it as-is)
Amazon Commerce reviews dataset from UCI
Within the bag-o-words dataset, try using the Enron emails
The Twenty Newsgroups dataset
This nice collection of SMS spam
You can always scrape (extract) your own text data from the Internet; I'm not sure which language or statistical package you're using, but XPath-based packages are available in R (rvest, scrapeR, etc) and Python to accomplish this


Stability of value function approximation in policy gradients

The short answer is yes. Take a look at Asynchronous Methods for RL. In a similar way that sampling from experience replay helps breaking the correlations, asynchronous methods are based on having multiple agents interacting with multiple instances of the same environment.

The Process:
Each agent (called worker) collects its own experience up to a specified timestep $t_{max}$, which is stored in a batch. Then a master network performs a training update by using this batch. After the update each worker resets to an identical network to the master and they start the task all over again. The update can be synchronous (as I describe here using a batch) or asynchronous by training multiple agents with their own parameters and then update asynchronously the master network.

Why this is helpful? The experience of each agent is independent of the experience of the others and thus more diverse which helps keeping correlation down.

A couple of additional comments on your question:

The goal in Policy Gradients (PG) is to optimize the expected reward w.r.t some parameters which is analogous to the estimation of the gradient of the policy function (parametrized by these parameters).
In RL, and especially in Policy Gradients, you are dealing mainly with 3 things: state sample correlation, bias and variance of your gradient estimator.
The introduction of the baseline is to reduce variance in the gradient estimator without introducing any bias. The baseline is called control variate and is used for reducing variance in Monte Carlo estimators (our gradient estimator here) and has zero mean.


I have used tesseract for similar tasks. I can give you few recommendation. You can choose the best that works for you.

Extract the parameter values by finding the exact locations they appear.

If the parameters always appear in the same place you can find those coordinates and extract parameters from there. You can get the location information by getting the TSV output produced by Tesseract while performing OCR.

Looking at the line / block / paragraph numbers produced by tesseract for each text

Since the values and parameter pairs are separated by lines or to separate sections, I think tesseract will do a reasonable job in giving your parameters either different line numbers or block/paragraph numbers. You can get this information if you get the TSV output from Tesseract using the create_tsv=1 configuration I believe.

Using regex to match parameters

It seems that your parameters are pretty consistent (i.e. either categorical ON/OFF or a number). In that case what you can do is, while reading the output from tesseract, match the description in front of the value. For example, as soon as you encounter AHU ON/OFF STATUS text, look for the word either on or off.

And you can use these in combination to make the results even more accurate. Hope these suggestions help.


How to convert Keras h5 to PyTorch pth format?

You must be careful with the versions from TF and PyTorch (as some commands may be different).
Basically you must:
1 - Know your layer and activation structure on Keras:
You can get the layer information with:
model_keras.summary()

If you can't get info about activation functions, try:
for layer in model_keras.layers:
    print(layer.output)

2 - Build a model on PyTorch that has the same layer structure (and activation) as on Keras
3 - The recently created PyTorch model (let's say model_pyt) has different weights and biases from your model on Keras, so you must copy those weights and biases from the Keras model to PyTorch model:
BE CAREFUL HERE, PYTORCH WEIGHTS ARE TRANSPOSED IN RELATION TO KERAS WEIGHTS
Then, for example, copying a weight would be like:
model_pyt.layer1.weight.data = torch.tensor(model_keras.layers[0].get_weights()[0].T)

And for the bias:
model_pyt.layer1.bias.data = torch.tensor(model_keras.layers[0].get_weights()[1])

repeat that for all layers.
Then, your PyTorch model has the same architecture and weights as the Keras model and might behave in the same way.


Data leakage and predictive models: should we use past predictions as a feature?

It depends on the business problem you are solving. You might see data leakage if your initial models were not so good. For the weekly problem it will be good to get the real ground truth (even a fraction of the week's data will be good) and add it for reevaluation of the model and possible retraining. Now if you are thinking that past purchase will make the customer return back (yes if the products are good and quickly consumed making the customer repeatedly want it again) then yes adding every week's ground truth will definitely greatly increase reduce the misclassification for returning customers for same products. But as I said in my first sentence if you only include predictions of past week into this week the model might go wrong since first iterations errors will start influencing the later iterations and the model will move away from good prediction as time passes by. If you include predictions of past week into this week and the customer's returning behaviour has got nothing or very less to do with the current behavior (like non-chronic patient visiting the hospital for Outpatient visits for minor ailments) then you are introducing data leakage which will give wrong predictions as time passes by.


How do I get a word frequency list? Can't export from Word Cloud

Ad 1) Word Cloud has a silly bug that causes the visualization sometimes not displaying. Press Regenerate word cloud and the visualization will appear.

Ad 2) This is currently not possible in Orange, but it's surprising how we haven't thought about it yet. We will make necessary modifications and the export should be available in the next release.


Standard deviation isn't an outlier detector. It can't tell you if you have outliers or not.

What it will do is effectively remove outliers that do exist, with the risk of deleting a small amount of inlying data if it turns out there weren't any outliers after all. As such, I think it's useful as a "quick-and-dirty don't want to spend too much time on this problem" method of ensuring your dataset only contains inlying data points. As long as you understand the risk of potentially binning 1% of inlying datapoints, I think it's fine.


Document similarity: Vector embedding versus BoW performance?

Here some points on which we can focus -

1.) Averaging the words vectors lose the order of words, making it very similar to the concept of Bag of Words. That is why with less Data Bag of words approach perform better than word2vec.
it has been found that training word2vec with potentially large data , outperform Bag of Words models. Google's results are based on word vectors that were learned out of more than a billion-word corpus.

2.) Published literature, distributed word vector techniques have been shown to outperform Bag of Words models. In this paper, an algorithm called Paragraph Vector is used on the IMDB dataset to produce some of the most state-of-the-art results to date.

3.) In your case Document size increase with time. So a good approach would be not to cluster each word rather than clustering documents. Word2vec algorithm has power to learn embedding for each document , which we call Doc2vec. it will cluster similar documents. and using cosine similarity you can find similar documents.


K-means clustering for one class classification

The K-means algorithm has the capacity of retrieving which are the "boundaries" your data has for knowing the only-class, is possible that you don't find the only-class boundaries to be the same boundaries your k-means algorithm found. This is the risk of comparing k-means with the one class classification: Clustering can be looking different things from one class classification.
This answer may guide you to another solution.

How to perform model selection for One-Class Classification?


I have done this before and didn't find a default implementation - the StratifiedKFold and RepeatedStratifiedKFold are only documented to work with classes.

The way I ended up doing it was not quite as you are thinking with quartiles/deciles, but rather using a histogram (it matched my needs).

NumPy has a nice method for doing this, with many different formulas for computing the bin sizes that you can play around with to be match your data i.e. if it is normally distributed or not. I can't post the entire method code, but here is the gist:

samples_per_bin, bins, = np.histogram(data, bins='doane')    # Doane's method worked best for me

min_bin_size = samples_per_bin.min()

# compute the maximum batch size possible, using all samples from the bin with lowest population
n_bins = len(samples_per_bin)
max_batch = min_samples_single_bin * n_bins

I then put the data into a Pandas DataFrame and added a column indicating which bin each data point was in - finally doing something like this to perform the sampling from each of the bins:

df.groupby('bin_name', group_keys=False).apply(
            lambda x: x.sample(n_per_group, replace=True))

Obviously you can allow duplicates in a batch or not by changing the replace argument. It might be necessary if you want a larger batch size while forcing stratification. It is definitely a limitation of my approach, which you might be able to overcome by using quartiles etc as you suggest.


The fast hist mode is available in newer versions of XGBoost. You can find infos on https://github.com/dmlc/xgboost/

For small data sets (100k lines), you wont feel much gain in training speed.


Keras exception: Error when checking input: expected dense_input to have shape (2,) but got array with shape (1,)

Yoy always need to pass the data for prediction in batches, although this batch is of size one (one sample). Try changing this line:

x = np.array([house_size, bhks])

into this:

x = np.array([[house_size, bhks]])

This should work.


How to find which patch in orignal image does an activation correspond to in vgg net after the final pooling layer

Tracing activation through a neural net is complex. OpenAI Microscope helps visualize it.