Why does "Depth = Semantic representation" in convolutional neural networks?

The point of the layer depth and gradual pyramidal reduction is to build up a hierarchy of spatially invariant representations, each more complex than those of the prior levels. For example, at the lowest level, a convolutional may be able to pick out noteworthy arrangements of pixels; at the next level, it can condense these into particular spots, basic shapes, edges, etc.; then at higher levels it can recognize increasingly large and complex objects.  I'll borrow an example from Gerod M. Bonhoff's thesis1 on Hawkins' Hierarchical Temporal Memory (HTM), which is a closely related concept, one that also makes use of receptive regions to build invariant representations. At higher levels, the filtering process allows a convolutional or HTM to assemble individual lines and shapes into objects like "dog's tail" or "dog's head"; at the next stage they can be recognized as a "dog" or perhaps a particular variant, like "German shepherd."

This is made possible not only by the stacking of multiple layers, but the divisions of neurons within them into separate receptive regions. The receptive regions mimic actual neuronal "cell assemblies" and cortical columns that learn to fire together in groups; this enables clustering around particular types of objects, while the additional layers allow them to be related together into objects of increasing sophistication. The decrease in spatial dimensions in the example you cited reflects the narrowing of the receptive regions as we move up the pyramid; the third dimension (i.e. depth within a layer, as opposed to the depth of the layers) increases in tandem so that we can provide a wider choice of spatially invariant representations to select from at each stage i.e., each filter in the output volume depth-dimension learns to look at something different. If we simply narrowed the pyramid at each stage along every dimension, eventually we'd be left with just a narrow range of objects to choose from; taken far enough, it might just leave us with a single node at the top reflecting a single yes-no choice between "is this a dog or not?" This more flexible design allows us to choose more combinations of the previous layer's spatially invariant representations. I believe that this also enables a convolutional net to take into account various orientation problems, including translation independence, by adding more cell assemblies/columns to deal with each reorientation of an invariant representation.

As this excellent tutorial at github explains,

  First, the depth of the output volume is a hyperparameter: it corresponds to the number of filters we would like to use, each
  learning to look for something different in the input. For example, if
  the first Convolutional Layer takes as input the raw image, then
  different neurons along the depth dimension may activate in presence
  of various oriented edged, or blobs of color. We will refer to a set
  of neurons that are all looking at the same region of the input as a
  depth column (some people also prefer the term fibre).

This type of design is inspired by various biologically plausible structures found in actual organisms, such as the eyes of cats. If what I've said here isn't clear enough to answer your question, I can add a lot of further detail, including more examples, some based on actual organs of that kind.

1See pp. 26-27, 36 76 Bonhoff, Gerod M., Using Hierarchical Temporal Memory for Detecting Anomalous Network Activity. Thesis delivered March, 2008 to the faculty of the Air Force Institute of Technology at Wright-Patterson Air Force Base, Ohio.


What scale does LightGBM use for output?

The underlying model will be a stepwise function. I don't see any garanty that it will work better (or worse) with the transformation, in the general case. This may be different depending on you variable (for binary variables you may want to work directly on the linear predictor).

However, in practice, if you know there is an underlying link transformation, I would suggest you to use it. This will often ease the model intepretation.


The diagram below Example 2 in this CS231 lecture is a good visualization of how filters work: http://cs231n.github.io/convolutional-networks/. Since each layer of a ConvNet has a depth corresponding to the number of filters (in an input RGB image, for example, the depth of the input layer would be 3), each filter of size $k\times k$ applied to a layer of delth $d$ slides over a volume of $k\times k\times d$ (note that the filter stays at a constant depth).


What's the best way to detect bible verse mentions in a text?

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper
Each sentence in the text could be compared to the 10 Bible verses. The bible verses could be ranked based on similarity.


Which classification algorithm to choose for classifying driving patterns (GPS coordinates) and mapping them to drivers?

I have done some previous work classifying vehicles (heavy or light) based on their driving behaviors. This required calculating speed and accelerations, which you can easily do by using numerical formulas such as the five-point stencil. You already know that points are separated by 0.25 seconds and the distances can be calculated using the haversine formula. More features can be derived such as the driving range, maximum and average speeds, number of left and right turns, hard breaks and sharp accelerations. Try plotting the polylines on Google Earth (or a similar mapping tool) to see if there is a distinctive geographical pattern (are they far apart from each other, or all in the same region?). If patterns are visible, a clustering algorithm may help.


Clustering is unsupervised which means that you do not know the classes and / or have no examples of correctly labeled texts.
Assuming you have some labeled texts then we are talking about a classification problem.
Your current approach of doing separate binary models for each label is very basic but still sound. What is your validation metric and score for each of the models? If the performance is already good then you might not need something else.
Another approach would be to train one model that predicts all classes and then to output the multiclass probability prediction. This would give you information about the best fitting classes but you would have to fit your loss metric accordingly, accuracy wouldn't make sense here for example.
Another approach would be to make a new factor variable which encodes all possible combinations of classes and train one model on this. However I suspect that the model would perform badly due to imbalanced cases and a high complexcity.


In orange, how do you split a given table at a certain row id?

Actually, I can add in a string column and should still be able to do the row selection trick even with PCA.  In the file widget you set that column as text / meta and it shows up in the forward row selection widget.

I think this will work.


How to select topology for neural network?

You problem is linearly separable so you can use a single layer perceptron.
Hidden units are necessary only for non linear problems (xor is a classic example).

As for general guidelines, I don't think anyone has ever needed more than one hidden layer. Also the number of neurons in the hidden layer doesn't need to exceed the number of inputs.


What type of machine learning are able to return feature importance?

You can calculate a feature importance for any classifier. Take a look at lime or shap
Shap unifies seven different methods (cited from the shap GitHub page):

1 LIME: Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why should i trust you?: Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.

2  Shapley sampling values: Strumbelj, Erik, and Igor Kononenko. "Explaining prediction models and individual predictions with feature contributions." Knowledge and information systems 41.3 (2014): 647-665.

3  DeepLIFT: Shrikumar, Avanti, Peyton Greenside, and Anshul Kundaje. "Learning important features through propagating activation differences." arXiv preprint arXiv:1704.02685 (2017).
4 QII: Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016.

5  Layer-wise relevance propagation: Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.

6  Shapley regression values: Lipovetsky, Stan, and Michael Conklin. "Analysis of regression in game theory approach." Applied Stochastic Models in Business and Industry 17.4 (2001): 319-330.

7  Tree interpreter: Saabas, Ando. Interpreting random forests. http://blog.datadive.net/interpreting-random-forests/

Depending on the classification algorithm you could calculate the feature importance by using Gini or gain based methods, too ..


How come same cluster category be separated?

This KMeans Clustering is a representative of points in the space based on 200 Features and their length. I think there is some gap in your perspective and the actual clustering.
 Other than "Tram", most of the other relevant features are uncommon in both the Rail Cluster. So, two different blobs are created in space.
See this image, data are at the same place on the Tram dimension but other Features have created two different groups.

 These are the approaches, you may take -
 - Clean all the features for duplicate/similar meaning in the domain context e.g. rail, railway
 - Try different cluster count e.g. 5 and see if these two merges
 - Even if these are not merging then there is some message and you should figure it out. e.g. gvg, combino, streetcar, city


ffill missing data based on the sort order of another column

Since you mentioned that there is a linear relationship between Age and Bandwidth, you could train a simple linear model i.e linear regression on your dataset to estimate the bandwidth based on Age. Once you have figured out the relationship you can use it to impute the missing bandwidths.


How to use ontologies for text classification?

Since no one answered my question, I'll describe what I found here for others.

Similar to what I described above in the question, you go over pre-processing, then look for the terms in the ontology and replace them with the whole branches from the ontology. After that, you train your model.

Notes:

Since you have the ontology with all the relevant terms, you can leave the stopwords in while pre-processing. Stopword removal can be performed via lists or using sklearn or Spacy. In addtion, Spacy performs lemmatization, which can be done via ontologies too (more accurate, but probably slower).
Use SPARQL for working with ontologies and pattern matching. Spacy does patter matching too; also, Lucene and several other tools. However, I haven't used them with ontologies. Similarity algorithms can also be used.
I found Random Forests to work best among traditional supervised methods. However, probably using neural networks might produce better results. Please keep in mind you can use alternative methods for text classification without ontologies, such as word embedding and topic models. Spacy seems to be a good tool for text classification too. It has nice documentation.


How to get an intution for generating Keras CNN layers for a model

Layer Choices : I believe this is an issue that requires domain expertise and experience. Neural network in general is still considered a black-box algorithm. As far as I understand what we try to do is improving model based on previous discovery of what actually works for our cases. Basically as you are more familiar with various model you may understand what works and what doesn't work and you will improve your model based on this. For example, you may want to check a more recent papers such as efficientnet, which is a very light yet powerful image classification model. They combined techniques and ideas that people have proven to work before and we have this amazing model. As for simple models like what you did above you parameter search should suffice. You might also consider a more sophisticated approach is using automl approach such as autokeras.

Debugging : I believe the tools you are looking for is model summary and visualization https://keras.io/visualization/. The first one is simple you simply call model.summary() which I believe you have already used on your code, this will get you a summary of the model and if you already understand how some CNN building block works you should have already get the idea how the shape changes and hence you can judge based on this whether or not it satisfy your expectation. As for your model it is still fairly easy to debug, since it is sequential and I believe the points that I mentioned above are enough.


Adding more emphasis on most recent data in CNNs

One approach is to use Keras - sample_weight parameter in fit() method.

That parameter specifies which steps are more important.


1) How does batch normalization layer work with multi_gpu_model?

For N GPUs, there are N copies of model, one on each GPU. For each copy, forward and backward passes are executed for a sub-batch (each sub-batch is 1/Nth of a batch). This means, batch normalization is actually sub-batch normalization, there is no access to the rest of batch.

# This `fit` call will be distributed on 8 GPUs.
# Since the batch size is 256, each GPU will process 32 samples.
parallel_model.fit(x, y, epochs=20, batch_size=256)

  2) Which batch normalization parameters are saved when saving a model
  (since when using multiple-gpus in Keras original model must be saved,
  as suggested here)?

One unified (template) weight is saved instead of N different weights. Each GPU calculates a different gradient due to receiving a different sub-batch. Then, either (1) weights are updated separately on each GPU and become synchronized periodically, or (2) N outputs/gradients are aggregated on the template model (on CPU), and then new weights are broadcasted back to GPUs. In both cases, there is a unified (template) model, although in the first case, the template model may not always have the latest weights since it is kept out of synchronization and gets updated occasionally.

Extra remarks

Some researchers have proposed a specific synchronizing technique for batch normalization to utilize the whole batch instead of a sub-batch. They state:

  Standard Implementations of BN in public frameworks (suck as Caffe,
  MXNet, Torch, TF, PyTorch) are unsynchronized, which means that the
  data are normalized within each GPU.

Because of weights synchronization we cannot expect a linear speed-up w.r.t. number of GPUs.
Some technicalities of multi_gpu_model are discussed in this github issue. multi_gpu_model has a speed gain when weights are sparse (in comparison to Dense layers), otherwise  weights synchronization becomes a bottleneck.
Also, here is an example of GPU-GPU weight synchronization flow from Nvidia:


How to predict unknown unknowns in machine learning

If you want to predict wether a bird is a bird of one of your two classes or unknown you need three classes: $[bird A, bird B, unknown]$.
For the unknown class you need data from birds which are neither $bird A$ nor $bird B$. You should make sure the number of rows for each of the three classes is roughly the same.
If you don't have data of birds which are neither $bird A$ nor $bird B$ you can use anomaly detection to detect wether a bird is $unknown$ before predicting if it is $class A$ or $class B$.


Shuffling begins by making a buffer of size BUFFER_SIZE (which starts empty but has enough room to store that many elements). The buffer is then filled until it has no more capacity with elements from the dataset, then an element is chosen uniformly at random. This means that each example in the buffer is equally likely to be chosen, with probability 1/BUFFER_SIZE. Then, a new example is loaded to fill the slot in the buffer that was emptied. This continues until there is nothing left to load.
A uniform shuffle would be what you would think of as truly random: any sequence of examples is equally likely. If the buffer is smaller than the size of the dataset, this is not possible. Here is an example of why: imagine I have ten examples, labelled 1 to 10, and a buffer of size 2. First I load examples 1 and 2 into the buffer, then I have no more space so I must randomly select something from the buffer. Therefore, my random shuffle always begins with example 1 or 2: not uniformly random!
If you have a buffer as big as the dataset, you can obtain a uniform shuffle (think the same process through as above). For a buffer larger than the dataset, as you observe there will be spare capacity in the buffer, but you will still obtain a uniform shuffle.


BM25 is usually used in information retrieval. In this task, you have a query and a lot of documents(maybe millions), and then you want to find a subset of these documents that are most relevant to your query. A ranking of a set of documents will be provided from the most relevant to the least.
If by efficient you mean fast in a computational way. I would say BM25 is pretty fast with respect to other algorithms that are using deep neural networks.
But if your asking if BM25 results are promising or not. This is debatable as BM25 is being used for a long time. People usually use it for the first step of ranking, then they do re-ranking with other powerful tools. It doesn't mean that BM25 is giving the best answers. But when you are dealing with thousands or millions of documents, This is a good choice in order to select only a subset of documents that have high scores with BM25 then later use a more accurate algorithm to rerank the results of BM25.


Why labels are not used in outlier detection algorithms?

Outlier detection can be performed in unsupervised fashion if there are no labels, or supervised fashion if there is a training set where outliers have already been marked as such by an "expert".

Unsupervised outlier detection is often (always?) based on density. Algorithms will call observations outliers if they are too far away from most of the other observations. This is useful to detect cases which are likely to be either errors or exceptional cases; and eventually, for instance, to remove the errors from the dataset.

Supervised outlier detection is less common. Indeed, in real world applications, it is rather easy to determine what should be the normal situations (e.g. some equipment mode of operation), but quite complex to define abnormal situations. You could have examples of abnormal situations, but probably not all possible abnormal situations. Then, a supervised learning outlier detection (basically a binary classification problem with normal and abnormal classes) will learn how to detect known abnormal situations, but will likely fail on unknown ones.

This is why there is the semi-supervised outlier detection (called novelty detection in the scikit-learn example). The training dataset contains only normal points, the model is trained to define the boundaries of the normal domain. Then, at classification step, the algorithm can predict if an observation can be deemed normal or not. Semi-supervised outlier detection techniques are often based on the likelihood that observations were generated by the same unknown process which generated the learning dataset (see Bayesian networks or Gaussian mixture models for instance).

Notes:

Unsupervised and semi-supervised techniques may be combined. For instance, you could use a first unsupervised approach to remove supposed outliers, then train a semi-supervised model on the remaining observations. The resulting model will likely be more robust than if outliers had been labelled as such, and a supervised model had been trained on the full dataset.
For semi-supervised or unsupervised outlier detection, the decision boundary position depends on model hyperparameters (expected fraction of outliers, distance to inliers, acceptable likelihood, etc.)


Generate 3D "matrix" with Pandas, based on comparing two dataframes [Python]

Pandas does not provide a 3D data structure per se (Panel has been deprecated and removed) - however, it is possible to express this kind of data using the long format (also known as EAV) with three key columns (or index levels), instead of the wide format you are suggesting.
Starting from your definitions, I believe the following achieves your goal:
(
    df_C
    # Transform to long format (two columns: former column names under `variable`
    # and corresponding values under `value`) plus the original index.
    .melt(ignore_index=False)
    # Join with the other dataframe, similarly transformed. join() implicitly joins
    # on indexes, so this will generate all combinations of the `variable` column values.
    .join(df_F.melt(ignore_index=False), lsuffix='_C', rsuffix='_F')
    # Make the index a regular column.
    .rename_axis('index')
    .reset_index()
    # Your rules can be expressed by multiplying the two value columns and examining the sign.
    .assign(combined=lambda df: df.value_C * df.value_F)
    .assign(output=lambda df:
        # Uses the Pandas nullable boolean type (three values: True, False, NA).
        pd.Series(pd.NA, index=df.index, dtype='boolean')
        # If combined is positive, both values were non-zero with the same sign.
        .mask(df.combined > 0, True)
        # If combined is negative, both values were non-zero with opposite signs.
        .mask(df.combined < 0, False)
        # If combined is zero, either of the values was zero, and the NA is retained.
    )
    # Remove intermediary values. The first three columns can also be transformed
    # to a MultiIndex.
    [['index', 'variable_C', 'variable_F', 'output']]
)


I got some pointers (below links) to my doubts. Hope it helps other future readers as well.

https://www.quora.com/How-do-you-take-a-machine-learning-model-to-production/answer/H%C3%A5kon-Hapnes-Strand
https://www.quora.com/Do-most-machine-learning-algorithms-run-in-batch-or-do-they-run-every-time-they-get-a-new-bit-of-data/answer/H%C3%A5kon-Hapnes-Strand

Will keep updating this space as and when I get some good answer(s) regarding this.


data pre-processing before image classification

Why are you convinced that the issue is in pre-processing? It could be in many other spots or a combination of steps in your process.

If I were you, I would revisit WHY you're using a decision tree. There are many other approaches to image processing and neural networks are far superior - in most cases - to decision trees.


What is fractionally-strided convolution layer?

Here is an  animation of fractionally-strided convolution (from this github project):

where the dashed white cells are zero rows/columns padded between the input cells (blue). These animations are visualizations of the mathematical formulas from the article below:

A guide to convolution arithmetic for deep learning

Here is a quote from the article:

  Figure [..] helps understand what fractional strides involve: zeros
  are inserted between input units, which makes the kernel move around
  at a slower pace than with unit strides [footnote: doing so is
  inefficient and real-world implementations avoid useless
  multiplications by zero, but conceptually it is how the transpose of a
  strided convolution can be thought of.]

Also, here is a post on this site asking "What are deconvolutional layers?" which is the same thing.

And here are two quotes from a post by Paul-Louis Pröve on different types of convolutions:

  Transposed Convolutions (a.k.a. deconvolutions or fractionally strided
  convolutions)

and

  Some sources use the name deconvolution, which is inappropriate
  because it’s not a deconvolution [..] An actual deconvolution reverts the process of a convolution.


How do I fit a curve into non linear data?

Here is a simple solution that you can compute yourself.
It is trivial to solve your equation for $γ$ as a function
of the other values.  We get
$$
γ = \sqrt{(\biggr( \frac{C}{v(ω)} \biggl) ^2 - {(ω^2-ω_0^2)^2}) / {ω^2}}
$$

It is almost true that you can substitute in your values and get
several estimates for $γ$, which you can simply average to get a good estimate.
I say almost true because as $ω$ approaches zero, both the numerator and denominator
go to zero, so the calculation becomes unstable. As long as you stay away from
$ω=0$, this should give good results.   @Esmailian provided some nice data in his
answer. Using his data I get the following estimates of $γ$.

6.012568  5.999664  5.994524  5.991173  6.009153  5.996748  6.019433
5.998072  6.001754  5.993656  6.002409  5.985938  5.996226  5.998363
6.474674  6.525602 14.140690  5.999124  6.000368  5.998427

Three of his $ω$'s are less than 1. These are the ones that give the estimates of $γ$

6.474674  6.525602 14.140690

The mean of the remaining estimates is 5.999859. As long as you have enough
$ω$'s that are not near zero,  this should provide a good estimate of $γ$.


Linear regression is not a classification technique. Most of this is meaningless as a result. Did you mean logistic regression? You do not need OneVsRestClassifier explicitly if you set LogisticRegression to handle multi-class classification internally with 'ovr' (one vs rest again) or 'multinomial' (softmax loss)


Multi-Label Loss function and model training

In a normal multi-class setup we use softmax over the last layer because we know that the sum of probability of the classes is 1 as only one of these classes is the actual answer. After that, we pass the output to a suitable loss function like BCE (binary cross-entropy).
For multi-label, we know that each class can be the output so the sum of probability of these classes will not necessarily be 1! However, the individual probability of each class still needs to be between 0 and 1 to be a valid probability distribution. Therefore, simply apply a sigmoid layer on top of each class output and pass this to BCE loss which treats each class as a separate mini BCE loss, and finally adds these losses across all the classes to get the final BCE.
For e.g., in Pytorch, you just apply sigmoid and pass it to the BCE such that the dimensions of the output passed to the BCE layer are (batch size, num of total class).
Look at the BCE loss section in the pytorch docs, where the dimensions of the input to BCE loss is (N, *) which implies the dimension which I just pointed out: TORCH.NN | Pytorch Docs.


Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

Without a reproducible exmple is difficult but I would go with something like this:
df = pd.DataFrame({"key_column":["key1","key1","key2","key3"],"name_column":["NAME1","NAME2","NAME3","NAME4"]})

df.groupby("key_column").name_column.apply(lambda x: "/".join(list(x)))

Result:
key_column
key1    NAME1/NAME2
key2          NAME3
key3          NAME4
Name: name_column, dtype: object


My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out.

Secondly, I don't think that you need to assume that the list of attributes is not finite. A standard and intuitive approach is to represent each attribute as an individual dimension in a vector space. Each product is then simply a point in this space. In that case, if you want to dynamically add more attributes you simply have to remap the product vectors into the new feature space (with additional dimensions).

With this representation, a seller is a point in the feature space with product attributes and a buyer is a point in the same feature space with the preference attributes. The task is then to find out the most similar buyer point for a given seller point.

If your dataset (i.e. the number of buyers/sellers) is not very large, you can solve this with a nearest neighbour approach implemented with the help of k-d trees.

For very large sized data, you can take an IR approach. Index the set of sellers (i.e. the product attributes) by treating each attribute as a separate term with the term-weight being set to the attribute value. A query in this case is a buyer which is also encoded in the term space as a query vector with appropriate term weights. The retrieval step would return you a list of top K most similar matches.


Do we also preprocess the target?

Sometimes. The most common example is scaling the target variable for regression, which is beneficial for some algorithms. Another example is when the distribution of the target variable is skewed and is processed into being normally distributed by log-transformation.


As far as I know text anonymization is mostly considered a manual pre-processing step, I'm not aware of any reliable fully automated method. The reliability of the process is usually crucial for legal and ethical reasons, that's why there must be some amount of manual work.

That being said, the process can be made semi-automatically, especially if the scope of the information to be obfuscated is not too large. In your case a NE tagger could probably be applied to capture a large part of the entities.

Once all the entities have been annotated in the original data, it's straightforward to replace them automatically with a placeholder. This can be done while keeping the original and anonymized version aligned (typically using a unique id for every entity).


Data preprocessing, relative scale problems in features of same type

If want you want is to scale both columns using the maximum of both and the minimum of both. You can do it like this:

df = pd.DataFrame(data=data, columns=['F1', 'F2'])
columns = ['F1', 'F2']

ma = float(np.amax(df[columns].values))
mi = float(np.amin(df[columns].values))

for column in columns:
    df[column] = df[column].apply(lambda e: (e - mi) / (ma - mi))

According to the documentation of sklearn that is what MaxMinScaler does, if you want to use predefined values for max and min, you can just change ma and mi. If you have more columns you just have to change the list of columns. For rescaling to an arbitrary range (max, min) read the sklearn documentation on MaxMinScaler.


pass variable length argument to mstats.kruskalwallis

i solved using

def kruskawallis_test(column):
    k_test=train.loc[:,[column,'SalePrice']]
    x=pd.pivot_table(k_test,index=k_test.index, values='SalePrice',columns=column)

    for i in range(x.shape[1]):
        var[i]=x.iloc[:,i]
        var[i]=var[i][~var[i].isnull()].tolist()

#     s=""
#     for i in range(x.shape[1]):
#         s=s+'var{}'.format(i)+','

    m=()
    for i in range(x.shape[1]):
        m=m+(var[i],)

    args=(m)

    H, pval = mstats.kruskalwallis(*args)

    return pval

i did
m=()
    for i in range(x.shape[1]):
        m=m+(var[i],)

args=(m)

by adding as many vars[i] in wanted to pass as argument of an empty tupple which i later passed to

mstats.kruskalwallis(*args)

*args helped mr pass variable length parameter.

please reply if you can suggest a better way.


Dataset of extremely low-dimensional images for PCA

Actually in your case I guess the pure images are not that important. The features that you extract from them are important because if your feature space is constructed base on intensity of images at different picture elements, pixels, then you will need so many coefficients. As an easy solution, use MNIST digits and use shape features to extract features from the images of numbers. You can use plausible number of features and then use PCA for the data that is in the new feature space that you have just constructed. In this case smaller number of coefficients will be needed if the features are fine.


How to add more weight to certain features?

The features should be first normalized before doing any kind of combination.
There are two ways to combine: concatenation and addition. You can try these both. Addition is only possible if the features sizes are same, though.


Important disclaimer: I'm not a statistician and I'm not sure about my interpretation!

I also thought at first about the duplicates, but I think the problem might be with this assumption:

  In my head, the training should be the data points , while the test should be  (as in  is the expectation of the test data).

Specifically the last part: in principle the test set is made of points from the same distribution as the training data, with the same risk of noise. In other words, the test set $t$ is similar to $T$: $t=\{(x_i,g(x_i)):i=1,\dots,m\}$ (and not  $t=\{(x_i,f(x_i)):i=1,\dots,m\}$).

  If 'test' here means the data , we must conclude that  contains  duplicates: otherwise we could fit a polynomial of degree +1 through all the test points and get this to zero.

Importantly, the test set $t$ is different from the training set $T$, and the estimated function $\hat{f}$ is based only on the points in $T$. So this way it makes sense that even a perfect estimate $\hat{f}=f$ might not be able to predict the true (noisy) value for every point $x\in t$. That could explain the non-zero minimum test MSE.


LSTM - Incorporate word embedding in layer with multiple records in same date

Your phrases are very short, you will be fine with a bag of words encoding instead of sequence.

2-3. If you want total cost over day prediction, you can just sum all the features including BoW per date and then do time series prediction on the resulting data.

Try just LSTM predicting next date from the sequence of past dates


Conceptual question on CNN and any multi layer neural network (Part 2)

There's an amazing book called "Deep Learning with Python" by Francois Chollet that you could refer to. To answer your question:

You usually add high number of layers and check where the validation accuracy halts and training loss decreases rapidly. This means your network starts to overfit and it's easy to fine tune your network from there on with the help of regularization.
You determine the layers of your network depending upon the complexity of the problem and the resolution of the images as well. Bigger pictures need larger networks. The filters need to be increasing as the network increases and the size of the feature map should decrease in an ideal network. I don't know much about strides though. Max_Pooling is the most common solution instead of strides.

I could go on about Max_Pooling but that won't answer your question.


What does dimension represent in GloVe pre-trained word vectors?

Glove creates word vectors that capture meaning in vector space by taking global count statistics. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words probability of co-occurrence. while optimizing this, you can use any number of hidden representations for word vector. In the original paper, they trained with 25, 50, 100, 200, 300. These dimensions are not interpretable. after training, we are getting a vector with 'd' dim that captures many properties of that word. If the dimension is increasing, the vector can capture much more information but computational complexity will also increase. please go through this blog.


There are several ways to do this, here are a couple of options:

Calculate different lag values (difference between now and t time units)
Calculate a linear regression for different time windows and store the slope and the bias
You can also involve higher order models to describe what is happening, if you think for example that the acceleration also matters you could use a 2nd or 3rd degree polynomial over the past couple of observations.


A simple way to go would be to use Option 1. Thus, each row could be uniquely identified and you could perform classification easily. You could additionally add a column which mentioned the week in which they churned. This would be very similar to a Type 2 Slowly Changing Dimension. Go wide with your dataset, ie: keep adding as many columns as necessary. That way, even if you have to look at it in Excel before creating a classification model, it'll be easier. If you are using a Random Forest classification algorithm or any other tree based algorthim, this option would make good splits on nodes.

Option 2 would be possible, however, it would result in a very deep dataset which would grow over time if you decide to update it frequently.


1) Is this even an appropriate algorithm to apply to this sort of task?

No, you have selected an evaluation algorithm from chapter 9 of the book. None of the algorithms in chapter 9 are control algorithms. They are all designed to estimate the value function for a policy supplied as input. The corresponding control algorithms are discussed in chapter 10.

The current draft of the book does not give the corresponding TD(0) control algorithm with linear estimator. However, that algorithm does exist and might be suitable (with caveats). In fact in your case it could even have benefits over action-value based methods, because you reduce the scope of estimates needed by a factor of 3. This is something that you can take advantage of only if you have a full model of the environment, so can look ahead one time step to determine the best action. Without a model of the environment, or if you don't want to use the model in your agent, then you must use an action value based algorithm like Monte Carlo, SARSA or Q Learning.

  2) If yes to 1), how do I choose an action?

Well it was a no, but you could use the control version of TD(0). Then you have the problem of using your state value function to figure out the policy. The rule here is that to use state values you need to use a model of the environment. Specifically you need to be able to predict the next state and immediate reward given the current state and action. In the book, this is usually represented by the transition function $p(s′,r|s,a)$ which gives the probability of each possible successor state and reward. In a fully deterministic game, the probability is just 1.0 for one target state and reward caused by each action. In your case you have new rocks appearing randomly at the top. To be complete you would probably have to model this in detail (which would be painfully slow). However, given the very low influence of this top row, and how little planning the agent can do to deal with it, I'd be tempted to just sample it.

Assuming you want to choose the greedy action, then you can find a policy by taking $argmax_a$ over the next step. When you have a state value function, then you have to run the step forward in simulation to figure out the expectation over each action. This is the same calculation for greedy policy as used in dynamic programming (back in chapter 4 of the book):

$\pi(s) = argmax_a \sum_{s'} p(s',r|s,a) (r + \gamma \hat{v}(s',\theta))$

Of course this is a lot simpler if you used action values istead (e.g. in SARSA):

$\pi(s) = argmax_a \hat{q}(s,a,\theta)$

. . . so despite the fact this is less efficient, you might want to use action values for less effort in this part.

One additional thing you are likely to have problems with: Your choice of state representation does not have good linear relationship with the true value function. The linear estimator is quite limited in what it can do. A better estimator here would be a neural network with at least one hidden layer.

However, you can probably tweak the representation slightly to get something that will work a little bit with a linear estimator - the trick is to have the rocks part of the state represented relative to the agent - i.e. don't have a grid of absolute positions of rocks, but make the grid relative to the agent. That way, a rock directly above the agent will always contribute the same to the current state value, which is important. Without this tweak, and using a linear approximator, your agent will not learn optimal control, it will instead learn a kind of fatalistic "with these many rocks, I have roughly this long to live" value function and probably just take random actions (if the distribution of rocks is not even it might learn to move to a particular column . . .)


This type of scatter plot is best performed in seaborn for Python.

# example from: https://python-graph-gallery.com/?s=scatter
# library & dataset
import seaborn as sns
df = sns.load_dataset('iris')

# Use the 'hue' argument to provide a factor variable
sns.lmplot( x="sepal_length", y="sepal_width", data=df, fit_reg=False, hue='species', legend=False)

# Move the legend to an empty part of the plot
plt.legend(loc='lower right')

The keyword hue will designate which variable is the one which is used for labeling the data points. Also, have a look at the link provided for more examples.


How to use hashing trick with field-aware factorization machines

One option is to use xLearn, a scikit-learn compatible package for FFM, which handles that issue automatically.

If you require feature hashing, you can write a custom feature hashing function:

import hashlib

def hash_str(string: str, n_bins: int) -> int:
    return int(hashlib.md5(string.encode('utf8')).hexdigest(), 16) % (n_bins-1) + 1


Geolocation Based Anomaly Detection in IPs Using Isolation Forest

IsolationForest doesn't work on Euclidean distance. Hence [0,0] is almost as good as [100,100]
It builds random Trees on the dataset and expects that the Outlier will singled-out very early in the Tree while the Inliers will go deep. With that logic, it can figure out the Outlier.

The IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.
 Since recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.
This path length averaged over a forest of such random trees, is a measure of normality and our decision function.
Random partitioning produces noticeably shorter paths for anomalies.
[scikit-learn doc]

Your data is very clean and unique. If we observe here, we can differentiate [0,0] and [100,100] in a single split. So the depth of Node will be same for both cases.
You may try a euclidean distance-based Model i.e. LocalOutlierFactor
from sklearn.neighbors import LocalOutlierFactor

model = LocalOutlierFactor(n_neighbors=10,novelty=True).fit(dataset)

model.predict([[0,0],[100,100]])

Output - array([ 1, -1])
Scikit-Learn guide for Outliers/Novelty - Link


Doc2Vec - How to label the paragraphs (gensim)

Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time.

The TaggedDocument constructor takes a list of tags. (If you happen to limit yourself to to plain ints ascending from 0, the Doc2Vec model will use those as direct indexes into its backing array, and you'll save a lot of memory that would otherwise be devoted to a string -> index lookup, which could be important for large datasets. But you can use string doctags or even a mixture of int and string doctags.)

You'll have to experiment with what works best for your needs.

For some classification tasks, an approach that's sometimes worked better than I would have expected is skipping per-text IDs entirely, and just training the Doc2Vec model with known-class examples, with the desired classes as the doctags. You then get 'doc vectors' just for the class doctags – not every document – a potentially much smaller model. Later inferring vectors for new texts results in vectors meaningfully close to related class doc vectors.


Is there enough datapoints to make a reasonable predcitction?

Having enough data points is an empirical question. Split the data into train and test. See if the model trained on the training data can predict the test data.
For small data sets, pick algorithms that are less likely to overfit. One example is a decision stump.


Bias-variance tradeoff in practice (CNN)

Normally, the training loss is lower than the validation one. This does not indicate any overfitting. Indeed, it is even suspicious when you training loss is higher than the validation loss. From other hand, worsening of the validation accuracy while improving on the train set definitely tells you that you overfits.

Generally speaking, overfitting means bad generalization, memorization of the training set rather than learning a generic concepts behind the data. Besides the metrics during the training you can find it out by trying your model on external datasets from a similar but not the same domain/distribution. Very poor accuracy will indicate the overfitting that might be hidden by a validation set very similar to the training set.


Text classification with neural network number of input neurons

Yes, actually what usually people do is to map the unique tokens in a space with fixed dimensionality, obtaining what is called "words embeddings".

And actually, using already trained word embeddings like GloVE is usually it's a best practice: those vectors are trained on huge datasets like Wikipedia or Common Crawl. A nicety of those vectors is that, thanks to the way they are built, they include also the relation between the words, a sort of semantic.
This is the way I definitely suggest you to start.


Which machine learning technique for product ranking/scoring

So I am assuming you just want to be pushed in the right direction. There are 2 different ways you can go about this.

Netflix up until very recently did all its recommendations using classical algorithms and setups, see paper on their architecture.

For this type of light recommendation problem I would recommend using something from PredictionIO. It is very versatile and can be used to classify using a variety of inputs. It's also not very hard to learn.

You can also solve this problem using neural nets, it can be viewed as a recommendation by classification. Deep learning is all the jazz now and you can utilize these breakthroughs in the recommender space.

Youtube is the big one when it comes to deep neural nets applied to recommendations, see this paper.

They split their system into 2 separate neural net models. One for candidate generation, and then another for producing the actual recommendations.

Spotify also did some awesome stuff applying Convolutional Neural Nets to the actual audio streams with some equally interesting results:
http://benanne.github.io/2014/08/05/spotify-cnns.html

As far as implementing something like that goes I would look for examples and build in python using either tensorflow or theano and keras. It wouldn't have to be too 'deep'.

Hope that helped!


I think you're in a pretty difficult state. I think the class imbalance techniques you're using are fine. I would advise you to try framing your problem in a hierarchical classification manner.

Level 1 Classifier

classifier_A between class 0 & class (1 + 2 + 3)

Note: Apply all sort of class imbalance methods to get good accuracy here

Level 2 Classifier

classifier_B between class 1, class 2, class 3

Final pseudo model

if predict_classifier_A(x_test) == "class 0":
   result = "class 0"
 else:
   result = predict_classifier_B(x_test)

I'm not sure will it work or not but its worth a try. Let us know if it worked better.


How to use hours of the day as a continuous feature?

Great question.

I'd say you should either use fourier transforms or sine / cosine transforms as they should get 0 to be right next to 23. The below post is talking about transforming day features, but should be easily applicable here.

Transforming day features

Transforming hour features


Normalization is rather common. I think you might want to consider the distribution of your data and whether you want to do anything with your outliers. Some popular normalization techniques: min-max and z-score. You will see tons of resources by googling them. From the limited amount of time I work with income data, sometimes you might want to test with a log scale as well as the top 1%-ers tend to skew the normalization as well.


Big array and MemoryError: Unable to allocate memory (in Python)

quick fix would be to change the data format - I can't see how your data looks like so my suggestion stay theoretical without example

float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.

Timestamp - decides what precision of time is needed - if you are fine with days - don't store hours, minutes and others

alternatively usse DataTable https://github.com/h2oai/datatable - Pandas is not the most efficient library when it comes to work with big data. however, I think the first 2 points will fix your issue. Because (157673, 157673) is not too big but 185. GiB sounds too much for it !


rectangular markers in bubble plot (Python)

I can imagine a transposed boxplot() working well... in a figure with seven subplots...

https://github.com/steadfordium/ga_pre/blob/master/bar_as_rectangular_bubble.ipynb


What if we input sequence data to feedforward network?

A finite impulse response (FIR) RNN can be expressed as a directed acyclic graph (DAG), hence it can be represented as a FFNN. So, you could theoretically make an equivalent FFNN to an RNN in this situation.
In terms of performance between non-equivalent models, an RNN would probably be better because, as they are inherently designed for sequential data. In the case of weather stat prediction, the temporal nature of the sequence elements would need to be learned by a FFNN, whereas an RNN would have in-built knowledge of such relationships. The RNN would also be able to more easily take advantage of "memory" via the common gating mechanisms as seen in LSTMs and the like.


How to restrict the columns to be passed to final classifier in PMML Pipeline

After digging down too much and some help from sklearn2pmml creator, I
managed to filter the final columns to be passed to the classifier.
Note : Here recorder is DataFrameMapper object.

1.Getting categorical column indexes.
cat_cols = [recorder.transformed_names_.index(c) for c in categoricalCols if c in recorder.transformed_names_]

2.Adding ColumnTransformer to filter those column with the help of their indexes.
pipeline = PMMLPipeline([
    ("mapper", recorder),
    ("select", ColumnTransformer([("drop", "drop", cat_cols)], remainder='passthrough')),
    ("classifier", xgb.XGBClassifier())
])

3.Fitting the Data to the pipeline.
pipeline.fit(X_train,y_train)

4.Creating PMML file out of Pipeline.
out_file = "XGBoost.pmml"
sklearn2pmml(pipeline, out_file)


This is probably irreversible operation unless the pre-convolution data was not full rank. But note that you reduced the dimensions of your signal, so the convolution was probably cropped to "valid" information (which doesn't need padding)

If you have at least the image or the filter, recovery might be possible, since convolution can be inverted using deconvolution but note that:

Convolution is defined as
$$ f(x) \circledast g(x) =  h(x)  = \int_{-\infty}^{\infty}f(x-t)g(t)dt $$
The some integral transforms (such as Fourier and Laplace) have the property that:

$$ T\{f(x) \circledast g(x)\}(s) = T\{h(x)\}(s) = T\{f(x)\}(s) \times T\{g(x)\}(s) $$

This is true for the Discrete Fourier Transform and Discrete Convolution as well, so to find your image $i$ from the filtered image $j$ by a filter $h$ given $I$,$J$ and $H$ as the Discrete Fourier transform of $i$,$j$ and $h$, respectively. Let $F\{.\}$ denote the discrete fourier transform and $F^{-1}\{.\}$ denote its inverse transformation:

$$ i = F^{-1}\{I\} = F^{-1}\{\frac{J}{H}\} $$

The discrete fourier transform is implemented efficiently by SciPy(signal module), FFTW, NumPy(fft module) and probably Theano. Having a lot of wrappers arround.

Note 1: Is important to notice that you will need at least an estimation of your filter, there are a lot of algorithms to do so.

Note 2: Deconvolution is very sensitive to noise, you can check on this class on Digital Image Processing to understand image filtering, mainly the part on Wiener filters.

Note 3: Image Deconvolution is implemented on scikit-image (e.g. Unsupervised Wiener) and on OpenCV using many algorithms (also on matlab in Image Processing Toolbox).


Underlying model for prediction using different prediction variables

Have had the same problem before.  Used Weka tool (http://www.cs.waikato.ac.nz/ml/weka/).

The approach:
• convert time of day into a nominal (non-numeric), eg bins numbered 0-47
• convert day of week into a nominal (non-numeric)
• use M5Tree to construct a model

The rules in the tree will pop out things like "if 0 < bin < 5:30am_bin" then ... which discriminates start of day

It is better with more than 1 month of data.  There will only be 4 samples for certain days of the week, for example.  So more months would be better. And in such case again converting month to a nominal value (non-numeric) lets the rules pick up seasonal and monthly changes.


Is there a difference between Vectorization and Featurization?

It's hard to say for sure without having the article in hand.  At risk of putting words in the author's mouth, I would guess that the two terms do mean the same thing if they are being used interchangeably.  Featurization is a common-ish term for feature engineering or feature extraction.  In that context,  you typically start with some low-level, noisy representation of the data, then pass it through some algorithm to transform it into a "feature vector" (a high-level representation of the data).  So I guess it might be tempting to call that process "vectorization" as well.  Once again, hard to say for sure without taking a look at the article.
Outside of a given article, I would not want to use them interchangeably.  Usually "vectorization" refers to modifying an operation so that it works with vectors rather than scalars.  (this is what numpy.ndarray is all about).


Using word embeddings with additional features

My first comment would be that you have to remember that Tree-based models are not scale-sensitive and therefore scaling should not affect model's performance, so as you well mention it should a problem with the feature itself.
If anyway you want to scale all your features you could use MinMaxScaler with the min and max values, being the min and max fo the Glove Vectors so that all the features are on the same scale


How do I get the name of Sagemaker Estimator's job

The _current_job_name contains the name of the job. So in the example from the question:

print(mnist_estimator._current_job_name)

would print

foo-2018-01-10-20-13-57-893


why variational inference in variational autoencoder if dealing with simple graphical model

I'm not sure if I get the question correctly, but we use variational inference, because we do have many dependencies. X and Z are usually high-dimensional vectors with complex relationships.


How to build Generative Model when we have more than one variable

First, I'd like to clarify. The maximum likelihood function you gave there is NOT with respect to "one feature". Or at the very least, it's not meaningful to think of this as a "feature" because in your example, you would actually evaluate the loglikelihood function at observed values of y not x. You aren't using your variables in x to explain anything about y. The resulting maximum likelihood estimate you would get for $\mu$ would be $\bar x$, the sample mean of your target variable y.
Now, suppose you actually do want to use x (your features) to predict y as in the case of supervised learning. Then, as I alluded to in your comments you need to specify two things - the predictor or model function, denoted typically as $\hat f(X)$ and the "link function", denoted as $g$.
$\hat f(X)$ is a function of your predictor variables such that:
$$g(E[Y|X]) = \hat f(X)$$
In your case, $E[Y | X] = \mu $ since you have a Normal distribution. Hence,
$$g(\mu) = \hat f(X) \rightarrow \mu = g^{-1}(\hat f(X))$$
Now, in terms of choosing $\hat f(X) $, this depends on your goals and how complicated you wish to go. Regardless, this is a function that can take on any real number. In the standard case (say in linear regression) you set $\hat f(X) = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$ There are other examples where writing out $\hat f(X)$ is impossible or tiresome, for example in the case of gradient boosting trees or deep neural networks. Other algorithms may set $\hat f(X) = B_{0} + h_1(x_{1}) + h_2(x_{2}) + h_3(x_{3})$ where $h_{i}$ are smooth functions. It really depends on again how complicated you wish to get and how interpretable you need your models to be.
With respect to g, the "link function", this is almost always chosen depending on the range of the response variable or the range of the parameter that you are linking with ($\mu$ in your case). In your case, since the normal distribution can take on any real number, most of the time $g$ is chosen as the identity function since $\hat f(X)$ naturally can take on any real number already. This leads to:
$$\mu = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$$
Finally, when dealing with your likelihood function:
$$p(Y) = constant * \prod_{i = 1}^{n} e^{\frac{-1}{2}\frac{(y_{i} - (B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}))^2}{\sigma^2}} $$
Solving this equation will provide you with the ordinary least squares estimates which I am sure you have seen before. Of course, choosing a different $\hat f(X)$ or a different $g$ will likely change everything and often you may not even get nice-looking closed-form solutions that come out with ordinary least squares. This motivates different numerical optimization methods. However, the "ingredients" are the same.
I hope this helps.


Your problem is that neural networks work poorly when the input is not scaled to a simple range. A usual choice is to scale and offset each column so that it has mean 0 and standard deviation 1.

In your case, x1 and x2 vary from 0 to 49999 and roughly 10 to 50009. This range for inputs will causes lots of numeric issues.

With a balanced dataset as you have, 51% accuracy is basically just guessing (within experimental error), so the network has learned nothing. Try again with scaling - e.g.

x1 = (x1 - 25000) / 14433
x2 = (x2 - 25000) / 14433

I have tested your code with this addition, and it gains 100% validation accuracy within the first epoch.

If you want to assess other values in testing later you will need to scale them in the same way.

Note your predictions may be off in testing when x1 and x2 are not close to 10 apart, because you have only trained with examples which are close to exactly 10 apart. How the network behaves when this is not the case - e.g. for inputs of x1 = 100 and x2 = 1000, or x1 = 90 and x2 = 15, may not generalise well compared to the original comparison function.


Try this R code:

library(igraph)
dfr <- data.frame(idMovie1=c(2,1,4), idMovie2=c(3, 2, 3), strength=c(23,80,10))

igr <- igraph::graph.data.frame(dfr)
plot(x = igr,
     edge.curved=FALSE, edge.width=log(edge_attr(igr)$strength), edge.label=edge_attr(igr)$strength,
     main="Graph of Movie Strengths")

Works only for small daasets. visualizations get ugly quickly.


How should I treat these non-English documents in the NLP task?

You can use these tips :

  Should I exclude them for the corpus and from training the model?

You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised.

  should I manually translate them (Requesting natives from each
  language to translate it for me) and include them in the final corpus?

You should do this only when you need the 500 docs as they hold significant information. I would not recommend this method personally.

  should I use Google translate/DeepL to translate these non-English
  documents into English and then include them in the final corpus?

That is something you can do. Using Google Translate could be a help if the structure of the sentences in the docs is simple and sober. You can get good translations without the need of any natives of other countries.

I think you should opt for this method.

Conclusion:

Translate the docs which hold importance using Google Translate.
Else, omit them if you have an extra of 500 docs. They willl not affect the model's performance significantly.

Tip:

I don't think that you can paste the 500 docs in the Google Translate console. This will be time consuming if the docs belong to different languages. Hence, try to omit these documents. Otherwise, you will require a mini application which could use some translation API to transform the docs.


Predict the next player's option for betting based on playing history?

this is a multi class classification problem.

There are many approaches you can use to solve this problem.

You can first try Gradient Boosted Trees with libraries like xgboost or lightgbm. Both of those can be used in R or Python.

You can also use neural networks for multi class classification problems.

But there are many ways to solve a multi classification problem.

You can turn it into a multi binary classification problem. Since you have 300 options, you can train 300 binary classification algorithms on your data.

You can also try K nearest neighbours.


predicted labels or predicted probabilities in a ROC plot?

Use the predicted labels.
Explanation - The ROC curve show possible classification performance at different setups.
Where 'classification performance' is a factor of True-positive-rate and False-positive-rate. And 'different setups' are thresholds used to determine a predicted label(class) based on predicted probability.
The latter is the reason why you should use predicted labels and not predicted probability.

Some additional context:
In my experience, ROC curve can be useful for three things:
1) Determining optimal setup(probability threshold) for a given model. Based on possible setups and problem needs.
2) Evaluating overall model performance and different models comparison. This can be achieved using the AUC.
3) Understanding classification model behavior and identifying possible abnormalities. For example, if the curve has a step like pattern its possible your model is too simple.

Here is a short nice explanation on the ROC curve.


This doesn't seem like a great task for regex--even your pattern would miss very close typos like COWID-19 or potential OCR mistakes like C0VID-I9. Instead, I'd suggest using the stringdist package to do fuzzy matching, perhaps stringdist::afind to find approximate matches of "COVID-19". You can read a bit about it here.
This will let you select from a variety of string distance algorithms and set a maximum distance. You could then, e.g., correct matches to "COVID-19" and proceed with your analysis.


How to get xgbregressor feature importance by column name?

If you have X_train Dataframe then you can take columns from there and loop it with feature_importances_

for col,score in zip(X_train.columns,model.feature_importances_):
    print(col,score)


Exploratory Data Analysis on dataset divided by winners and losers

So, the question asks that during Exploratory Data Analysis whether it is better to analyse the data with both winning and losing tennis players in one data frame or split them into two. In this question, I am going to assume that you will be using this data in a classification task (do correct me if I am wrong!): to discriminate between winning and losing tennis players.
I believe the best approach would be to analyse with both winning and losing players together. The main reasons for it are:

When you are running this data through a model, you won't split the data by class (winning vs losing players)
More importantly, in the context of feature engineering, the usual rule of thumb normally is the you choose features which can greatly discriminate between the classes. So you can only do that by reviewing the feature distributions by classes to see if adding those features to the model will improve generalisation performance of the model.


How is a single element of the training set called?

The term you are looking for is "Example".
Source: Martin Zinkevich, Research Scientist at Google (http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf)

Instance: The thing about which you want to make a prediction. For example, the instance might be a web page that you want to classify as either "about cats" or "not about cats".

Label: An answer for a prediction task  either the answer produced by a machine learning system, or the right answer supplied in training data. For example, the label for a web page might be "about cats".

Feature: A property of an instance used in a prediction task. For example, a web page might have a feature "contains the word 'cat'".

Example: An instance (with its features) and a label.


Tensorflow API: What does the metric `tf.keras.metrics.TopKCategoricalAccuracy` do?

Result of tf.keras.metrics.TopKCategoricalAccuracy() will be between 0 & 1. Default value of the argument k is 5.
The result is 1 because for both the samples, the actual value is within the top 5 predictions.


How to add confidence to model's prediction?

It is common to say the error term follows a standard Guassian distribution. If you assume that to be true, then your squared errors follow a Chi-squared distribution:

  In probability theory and statistics, the chi-squared distribution (also chi-square or χ2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

Have a look here for some ideas about how to implement a quasi-confidence metric, based on your (mean squared) errors. It assumes the errors follow a chi-aquared distribution and then uses the normalised RMSE to define a set of confidence boundaries for a given confidence level,  $\alpha$, as follows:

$$
\left[\sqrt{\frac{n}{\chi_{1-\frac{\alpha}{2},n}^{2}}}RMSE,\sqrt{\frac{n}{\chi_{\frac{\alpha}{2},n}^{2}}}RMSE\right]
$$

See the link for the steps involved. Here is the coded simulation taken from that post, with some added comments (requires python 3):

from scipy import stats
import numpy as np

s = 3                           # a constant to scale the random distribution
n = 4                           # number of samples/states per prediction
alpha = 0.05                    # confidence interval

# distribution with confidence intervals ɑ = 0.05
c1, c2 = stats.chi2.ppf([alpha/2, 1-alpha/2], n)

# we will take this many samples (this pre-allocates the y-vector)
y = np.zeros(50000)

# Loop over each sample and record the result mean sample
# This would be your prediction vector - here it is random noise
for i in range(len(y)):
    y[i] = np.sqrt(np.mean((np.random.randn(n)*s)**2))

# Use the chi-squared distributed confidence intervals to see when predictions fall
# finds percentage of samples that are inside the confidence interval
conf = mean((sqrt(n/c2)*y < s) & (sqrt(n/c1)*y > s))

print("1-alpha={:2f}".format(conf))

Here is another answer on CrossValidated, which gives more information around the area.

Additionally, if you assume your predictions lie within a Gaussian distribution, you could use the variance of your predictions as the confidence (welcome to Bayesian learning!).

There are packages that will help you do this, such as BayesOptimization. There are lots of examples on that webpage. Essentially, you will be able to make predictions and automatically get robust estimates for condifence... and some cool plots to show where your model is quite sure, and where it isn't:


How to increase accuracy of classifiers?

Dimensionality Reduction
Another important procedure is to compare the error rates on training and test dataset to see if you are overfitting (due to the "curse of dimensionality"). E.g., if your error rate on the test dataset is much larger than the error on the training data set, this would be one indicator.
In this case, you could try dimensionality reduction techniques, such as PCA or LDA.
If you are interested, I have written about PCA, LDA and some other techniques here and in my GitHub repo here.
Cross validation
Also you may want to take a look at cross-validation techniques in order to evaluate the performance of your classifiers in a more objective manner


Do you think it's normal for data science projects to have some amount of "what should we do" -time? Or does it mitigate by experience?

This is really about differences in the expectations of your employers and you.

In general every job depends on continued learning and making time to expand your knowledge. Data science as a very heavily knowledge and methodological based occupation and therefore life-long learning is even more of a thing.

But how and when to do that learning? In my experience most employers expect you to do that sort of learning on the side or with irregular, slotted and scheduled training.

However I think it is quite normal that a big part of learning is during projects / on-the-job like you describe. Experimenting with new methods, reading up on other solutions, etc.

At the end you have to balance these expectations and just make sure, that your "research time" is goal-oriented and efficient. Don't spend hours reading up on "interesting untested solutions" but also don't worry if you spend time on datascience.SE to look up solutions and guidance.


Formal definition for parameter setting in data mining context

First part about parameter setting regards to the fact that you don't need to define any parameters of the model, which is the case in per se linear regression which takes the form of ŷ =θ0+θ1x where $θ_0,θ_1$ are parameters. Decision trees create those n-dimensional boundaries for future classification (regression) based on certain criteria.

If you are familiar with statistical inference, a good analogy would be good old parametric vs non-parametric tests.


Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.

Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. However, its a good practice to remove any redundant features from any dataset used for training, irrespective of the model's  algorithm. In your case since you're deriving new features, you could use this approach, evaluate each feature's importance and retain only the best features for your final model.

The importance matrix of an xgboost model is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees. The second column is the Gain metric which implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.


Document embedding vs locality sensitive hashing for document clustering

Locality sensitive hashing (LSH) is a search technique. With it, similar documents get the same hash with higher probability than dissimilar documents do. LSH is designed to allow you to build lookup tables to efficiently search large data sets for items similar to a given item. It is also a probabilistic method in that it can generate false positives and false negatives. While there are ways to train LSH, most LSH is untrained. That's because LSH has been studied more in the search setting than the machine learning setting.

Embeddings are a machine learning technique to capture semantic information for use in some downstream task, such as clustering or classification. Typically semantically similar items get similar (but not the same) embeddings. Embeddings are trained from data. There are many unsupervised algorithms (word2vec, glove) and there are supervised methods too (auto-encoders, hidden layer output from deep models).

Embeddings can be used to map items into a space where near neighbor search would find semantically similar items. However, on large data sets you would still need to index the data to search efficiently, which raises the possibility of doing LSH on embeddings. That way you get the benefit of a trained model that learns the distribution of your data set and the benefit of a fast lookup table.


Usually you can only predict with the variables you have trained on. But in a case like this, I would suggest you check the Multicollinearity of these missing variables with the ones you will have. May be they are already highly correlated with the features you already have. In that case, you can just model using the available features.

If that is not the case and the missing variable looks like a significant variable in your regression, then you might need to pick some other variables which can help you indirectly derive the missing features. For example may be category helps you derive the rating with some accuracy.


Why don't the dimensions in this linear regression equation match up?

My guess is that $x$ here refers to a single datapoint with shape $m$x$1$. So $h_w(x)$ and $w^Tx$ both have shape $1$. In the case of multiple datapoints given as a matrix $X$ of shape $n$x$m$, the equation becomes:

$h_w(X) = Xw$


is it possible to do feature selection for unsupervised machine learning problems?

Take a look at these links-

https://stats.stackexchange.com/questions/108743/methods-in-r-or-python-to-perform-feature-selection-in-unsupervised-learning

http://www.jmlr.org/papers/volume5/dy04a/dy04a.pdf

https://www.researchgate.net/post/What_is_the_best_unsupervised_method_for_feature_subset_selection


Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?

Extract features such as variation ratio
Cluster the extracted features instead of the raw data


Establishing data science programs as an independent discipline

Advantages of setting up a dedicated data science as an independent discipline, which I can think of:

There would be more and more data scientists who can cater to the growing demand for them, in companies and industries in almost every domain.
The concept of statistics, algorithms (ML would be included in this), and Big Data don't need to exist as chapters in separate domains. Now, they can exist as part of the data science workflow in this program.
Data Scientists would be more aware how data science would be applicable in the domain of business. The concepts of business knowledge for data scientists is often found wanting. Data scientists with a knowledge of Business Statistics is in huge demand, and a domain dedicated program would help serving the needs.


ValueError: Error when checking input: expected the_input to have 3 dimensions, but got array with shape (14174, 1)

Your inputs have different length so as suggested by @skrrrt, you should pad your data and apply a mask into your model.
The following pads all your input with 0. values so that all sequences have the same length.
from tensorflow.keras.preprocessing.sequence import pad_sequences
padded_inputs = pad_sequences(X, padding="post", dtype='float')

You can choose which value to use for padding with the parameter value=0.0 (documentation)
Then, add a masking layer just after your input layer in your model.
    # Main acoustic input
    input_data = Input(name='the_input', shape=(None, input_dim))
    masked_input = Masking(mask_value=0.0)(input_data)
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(masked_input)

Refer to this tutorial for more information.


Is K-NN applicable for binary variables?

Yes, you just have to find a suitable distance metric, instead of using the default Euclidean distance. Euclidean distance will work, but it loses a lot of its positive points when used on a non-euclidean space.

For you specific case, the Jaccard distance basically measures how many 1's are equal on both instances, ignoring the dimensions where both are 0's. This gives an interpretation like "if a program has a course, but the other doesn't, then they are dissimilar". Jaccard index is very useful in high-dimensional boolean matrices, such as generated by one-hot encodings.

Other more intuitive choice is the perfect match distance, which would simply measure how many dimensions are different for the two instances and can be easily computed by $\sum_{i=1}^n |X_i - Y_i|$. In this case, the interpretation becomes similar to "if both programs have a course, or both program don't have a course, then they're similar".

However, be careful with the K value of your K-NN. You only have two instances of class B, so you will have to choose 1 or 2 (technically at most 3) for the value of K.


ValueError: No gradients provided for any variable

I found the solution to my problem.
First of all, I had to declare a loss when compiling the model :
model.compile(
    optimizer='adam',
    loss='mean_absolute_error',
    metrics=['mean_absolute_error']
)

In addition to that, I changed the monitor for the checkpoint:
import tensorflow.keras.callbacks as C

checkpoint = C.ModelCheckpoint(
                filepath='best_pool.h5',
                save_best_only=True,
                monitor='val_mean_absolute_error',
                mode='min')

Changing the monitor gave a better result in this case.


Dropout backpropagation implementation details

As given in the links, the answer is yes! note that you divide the mask by p so that you won't need to multiply by p in the test time and since this is a coefficient for the new activation, it will come out of the derivative in chain rule in backprop.


Why the VC dimension to this linear hypothesis equal to 3?

The VC dimension for the classifier depends on the dimension of space that your data points belong to.

In your problem, if your mean $x\in \mathbb{R^2}$, then the VC dimension is 3. For $\mathbb{R^2}$, you can always shatter any three general position points ("general position" means they do not coincidentally lie on the same line). For instance, consider three points (0,0), (0,1), (1,0). No matter how you assign labels to them, you can always seperate them by a line. If you consider point points with label (0,0), (0,1), (1,0), (1,1) with label [+,-,-,+], this is the "xor" case which cannot be seperated by a line.

For $x\in \mathbb{R}$, then the VC dimension is 2 because you cannot separate +-+.

In general for $x\in \mathbb{R^d}$, the VC dimension for a linear classifier is $d+1$.


There could a skewed power envelope or non-stationary data. As a result, off-the-shelf feature scaling could attenuate the signal.

There are feature scaling techniques that tend to work better for audio signals, examples include: RMS level (Root Mean Square Level), Cepstral Mean Subtraction (CMS), RelAtive SpecTrAl (RASTA), kernel filtering, short time gaussianization, stochastic matching, and feature warping.

You should make sure you understand your raw data and the assumptions of each feature scaling technique before application. Accuracy-driven machine learning might lead to the wrong conclusions.


Is 10,000 images for one class of spectrograms good enough for music classification model?

Agreed with Emre. One thing that can be helpful when asking the question is to look at comparable datasets since there's usually some around. For spectrograms for example, there are 150,000 samples w/ 12 labels here, 2890 samples here, and 1 million samples with various labels here.

Find a few examples and gauge a rough order of magnitude of samples per class given how close the tasks are to yours. That should give you at-least a starting point. Then, if lets say one set is very large, usually someone will have written a paper using that set, and that can help start both the architecture and size jumping off point (and maybe a baseline for transfer learning) of your task.


LFR as synthesized data for dynamic community detection methods

I think the dataset is generated using dynamic network generator made available at this address which is dynamic version of static LFR. It's an excellent dynamic networked data generator.


First of all, don't use classification accuracy as a metric. Use precision, recall or F-score. They are best suited for multiclass unbalanced datasets.

Secondly, if you want to enrich your dataset with synthetic points of the minority class, a common way is to use the SMOTE algorithm.

Another way of creating more synthetic samples of your classes would be to use Generative Adversarial Networks (GANs). You will train a Generator and a Discriminator network based on the samples that you have. See this for further info.


Predicting a final exam score from arbitrary set of practice exams

I think that the most natural way to design this problem is indeed with missing values, and of course that means using a learning algorithm which can deal with these. Additionally to NB I think decision trees (at least some implementations) can handle missing value as well.


Effect of weights on the Louvain communities detected

Yes, weights are interpreted as a strength of connection in the context of modularity.  This is perhaps easiest to see using the interpretation of unweighted modularity as

The fraction of edges within communities minus the expected such fraction in a random graph with the same degree sequence.

Switching the adjacency matrix to the weighted adjacency matrix to get the weighted modularity just changes this to

The fraction of total weights within communities minus the expected such fraction in a random weighted graph with the same total weights at each vertex.

That said, you do have to be careful.  There are community detection methods that do treat weights as distances:
SO: meaning of weights in community detection algorithms
It may be worth running a simple example when first using a specific algorithm implementation; e.g., with $G=C_4$, weight one perfect matching very much higher than the other, the communities should end up being two $K_2$s, and you want to know whether they contain the high- or low- weight edges.


Identifying potential customers based on their Rank and Value

Do you have some data on past clients ? For example, do you have a dataset with past clients Rank and Value, and a variable indicating if the client was good or not (binary variable 1 if the client is good and 0 if it's bad, or a variable to use as indicator, like profit done on the client for example).
If you have such a thing, your problem is a basic classification problem, where your threshold is not 0.5 (the probability you want to be "Good Client" is not 0.5, as it is on default cases), but where you just select the 100 customers with the best probability (closer than 1).
If you don't have such dataset, and such variable indicating if the client was good or bad, based on what you know from the past, then you can't create a model since you don't know how to train it.


No. From this correlation matrix you cannot draw the conclusion that

  as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college

The reason is that correlation is a measure of association between single pairs of variables. The conclusion you draw above - on the contrary - is based on a combination of three different variables plus the outcome variable.

If you want to get an estimation of the probability that a student will be admitted to college based on her gpa, gre and prestige the right way is to create a logistic regression model. Here's an example in R (provided that admit is a binary variable, with admit=1 indicating that the student is admitted and admit=0 that she is not admitted)

model <- glm(admit ~.,family=binomial(link='logit'),data=data)

With this fitted model you can then compute the probability that a student is admitted given her particular combination of gpa, gre and prestige.


How to load and run feature selection on a dataset with 5,000 samples and 500,000 features?

5000 samples and 500,000 is not that big - it all depends how much memory you have. Also remember you can always and always optimize your data format. e.g. if they are float64 - do they need to be ? if they are categorical, how they are encoded ? (one character or a 20 character word?) and such. so Yes, if you can load the data into memory good for you if not here are the suggestions:

if you only and only have 5K samples - you must not use all for feature selection.
you can drop features that have very low variance - in an extreme scenario if the variance of a column is 0 - for sure it is useless.
there is something called feature-screening proposed by Fan et. al from Princeton https://orfe.princeton.edu/abstracts/feature-screening-distance-correlation-learning - in short: you can lower your dimension by using a univariat model and then afterwards use multivariate-feature selection models.


After giving much consideration, let's review the mechanism of Random Forest Regression (RFR):

So this idea or process of averaging models is a technique called Ensembling. Additionally, Random forest is a Supervised Learning algorithm which uses an ensemble learning method for classification and regression. Random forest is a bagging technique and not a boosting technique. It uses a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

If you check sklearn documentation, the metrics for evaluation are:

MSE (default) ---> Root of MSE ---> RMSE (optional)
MAE           ---> Percentage form of MAE ---> MAPE (optional)

Additionally, MSE is recommended to use on continuous data via this post
You can use the method of score(X, y):
Return the coefficient of determination R^2 of the prediction.
The coefficient R^2 is defined as (1 - u/v), where:

u is the residual sum of squares ((y_true - y_pred) ** 2).sum()
v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum()

The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse).
A constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0.
The problem with this question is OP didn't share info about the evaluation of used metrics on the train and test datasets except for his own defined metric formula and some SKlearn metrics as follows:
#OP's defined metric #MSE
average_mean_error = (SalesPrice - predPrice) / Salesprice

result_rfbl.insert(2, "predResidualValue", rf.predict(X_test), True)
result_rfbl.insert(2, "predPrice",  round(result_rfbl["predResidualValue"] * result_rfbl["NEUPREIS"], 2), True)
result_rfbl.insert(2, "SalesPrice", round(result_rfbl["RESTWERT"] * result_rfbl["NEUPREIS"], 2), True)

# absolute prozentualer Fehler #MAPE
result_rfbl.insert(2, "ERROR", abs(result_rfbl["SalesPrice"] - result_rfbl["predPrice"])/result_rfbl["SalesPrice"], True)

I assume that his model seems to be overfitting If validation MSE higher than the training MSE and get a very decent result with the training MSE nevertheless I don't want to judge by the cover of the book. If it is the case, RFR-model needs to apply feature engineering or RFR-analysis and Regularization like Ridge (L2 regularization) & Lasso (L1 regularization).
To understand better the reason behind the difference between manual defined MSE and Sklearn MSE, I draw your attention to minimum weighted MSE calculation for the first Split of a Decision Tree on SalesPrice from this source.


A policy is a state-action mapping.  A 'state' is a formalism used in AI that represents the state of the world, i.e. what the agent's idea of the world is.  The action is, naturally, what action it should take in that state.  A policy just maps states to actions.

One of the basic problems in AI is how to maximize reward over time at some task.
 One strategy for the agent is to try to understand the system and predict the results of their actions, and the reward that would follow.  Another strategy is for the agent to try lots of things and record the results.  Either of these (eventually) allow the agent to calculate a good policy - and once it's calculated, the difficult computational work is done and the agent just has to 'look up' what action to take in each state.


I guess what you are looking for is a crawler, which can crawl through the web pages and scrape the data for you into a csv file.

I really don't like the idea of giving away the full code. So, I would tell you how it's done. I would use Python for the example.

Define the url of the web page. Leave the last part (page=), as it can go into a loop, so that as it needs to be incremented after every page gets scrapped.

Now, identify the xpaths of the data which you want to scrape. Store each of them in an array, and keep writing(and appending into a csv)

This is the code of one such scrapper which I have written for a project of mine. Refer to it in case of a road-block. I have used the lxml library of Python.
You can use other fancy libraries like Scrapy, BeautifulSoup. But, they are just fancy wrappers over lxml, so it's faster.


From the Keras documentation, you can load the data into Train and Test sets like this:
(X_train, y_train), (X_test, y_test) = mnist.load_data()

As for cross validation, you could follow this example from here.
from sklearn.model_selection import StratifiedKFold

def load_data():
    # load your data using this function

def create model():
    # create your model using this function

def train_and_evaluate__model(model, data_train, labels_train, data_test, labels_test):
    model.fit...
    # fit and evaluate here.

if __name__ == "__main__":
    n_folds = 10
    data, labels, header_info = load_data()
    skf = StratifiedKFold(labels, n_folds=n_folds, shuffle=True)

    for i, (train, test) in enumerate(skf):
        print "Running Fold", i+1, "/", n_folds
        model = None # Clearing the NN.
        model = create_model()
        train_and_evaluate_model(model, data[train], labels[train], data[test], labels[test])


What you are looking for can be found in KONECT (the website is down as I'm writing this but it should be fixed soon!). It's almost the most comprehensive data collection for network analysis. But the question is which one is more standard to use?

Well, there is no clear answer except of Zachary's Karate Club!

If you do a literature review in Community Detection algorithms you'll see that almost all shining papers use different networks. My suggestion is going through what Andrea Lancichinetti and Santo Fortunato did for benchmarking graphs. They proposed some benchmark graph generation algorithms e.g. this one.

Hope it helps :)


Neural network breaks if hidden layers are increased

I found my error. My early calculations only involved one hidden layer, however, I did not include the derivatives of previous weights when calculating the hidden_error. This means that in the scenario of a neural network of:

Input -> H1 -> H2 -> Output

to calculate the backpropagation of the last step (Input <- H1), I did not calculate derivatives of weights_hidden_output (H2 -> Output) and weights_hidden_to_hidden (H1 -> H2).


Weighted Binary Cross Entropy Loss -- Keras Implementation

The code is correct. The reason, why normal binary cross entropy performs better, is that it doesn't penalize for mistakes on the smaller class so drastically as in weighted case. To be sure, that this approach is suitable for you, it's reasonable to evaluate f1 metrics both for the smaller and the larger classes on the validation data. It might show that performance on the smaller class becomes better. And training time can increase, because the model is forced to discriminate objects of different classes and to learn important patterns to do that.


How to remove features from a sklearn pipeline after it has already been fitted?

Standard Scalar trained on 30 features so it expects 30 features only. One simple hack you can do is, you can create a new Standard Scalar and train with those 20 features, and replace your pipeline Standard Scalar with the new one.
For the LogisticRegression, get the non zero weights and set those weights to the new model with 20 features without any training like below.
model.coef_ = numpy_array_of_non_zero_coef
model.intercept_ = numpy_array_of_intercept


In Neural Networks and deep neural networks what does label-dropout mean

I ended up finding the paper you were referencing, maybe next time, add it to your post.

From what I can understand in this paper, label-dropout means that you are dropping the real labels and replace them with others.

The section of the paper you're referring to is explaining everything in great detail, so I'd try to read it carefully. In short, instead of using the ground-truth distribution (i.e. one 1 and a bunch of 0s) in the target layer, they are adding some uncertainty. That way, the model becomes less confident, and therefore is less prone to overfitting and becomes more adaptable.


Returned loss value is different than the loss printed with verbose

The model.evaluate function predicts the output for the given input and then computes the metrics function specified in the model.compile.Based on y_true and y_pred and returns the computed metric value as the output.
You can use  model.metrics_names  property of your model to find out what each of those values corresponds to.


How to choose PCA or KernelPCA a priori?

Lots of the dimensionality problems are trial and errors at first.

Only very few datasets are linear "manifolds" that can be described with PCA. But it's a good start to see if there are disjoint sets, or if you can figure out some structure in 2D or 3D.

KPCA is a good technique, but there are many different kernels you can try. You may want to start with other "better" techniques (better as more deterministic) like ISOMAP, Laplacian Eigenmaps or LLE.

One of the things to remember is that you need enough data to populate the embedded space.


Encoder-Decoder LSTM for Trajectory Prediction

There are multiple questions in your description:

The input to LSTMs are normally continuous representations. In NLP, you normally embed discrete elements as vectors in a continuous representation space and then you pass these vectors to an LSTM. You already have continuous representations, so you just pass your 2-dimensional vectors as input to the LSTM.
In NLP neural networks, like most neural networks, the training happens passing as input not one sample, but a minibatch of N samples. This N most of the times is chosen as the maximum number that makes the model, data and intermediate computations fit in the GPU memory.
The 3D array expected as input by the LSTM has shape [N, timesteps, feature]. In your case, this would be [N, 125, 2].
I think you don't need an encoder-decoder architecture, only the encoder. Therefore, a single LSTM would suffice. You would train it to receive any number of input elements and predict the next one. If you want more predictions ahead, you can feed the model's own predictions as input, autoregressively. To find an analogy in the NLP world, your model would be a language model, which receives words (or letters) and generate the following word.

P.S.: I may be wrong about you needing only the encoder instead of an encoder-decoder architecture, as I don't know the specific nature of the predictions you want to make.


Possible to use different learning rate for different neuron in Keras/Tensorflow?

You can follow this thread here: https://github.com/fchollet/keras/pull/3004 Seems at least that the layer-wise tuning should come.


Do we need the outer discount term when implementing REINFORCE algorithm

The answer to your question is already in the text which you copied from Sutton's and Barto's book.
Sutton's and Barto's pseudocode describes the general, discounted, case. While the lecture by Silver talks about the undiscounted case. And in the undiscounted case $\gamma=1$ and therefore $\gamma^t=1$. So you can just omit it. Equation (13.8) in Sutton's and Barto's textbook also provides this simplified equation for the undiscounted case.
Also note that Monte Carlo methods are applied to episodic MDPs only. Therefore, in many cases you might not need the discounted version.


I have a 20.2 GB dataset which is entirely consisting of 1 file. Need help on how to open the file

This file should be a plain text file with n rows and m columns separated by a tabulator. Here is another much smaller file which they offer for download and claim that it has the same data format.

Since the file is too big to fit in memory, you can use the data format HDF5. This will allow you to load slices of your dataset without having to load the whole file into memory. However, first, you need to convert the file to a .h5 file. This can be done with h5py.

I put the following code together right now. It reads the text-file line by line and writes each line to the .h5-file. It works fine on the smaller file which they offer for download. I haven't tested it on a larger file, however. If you try it out, let me know how it goes.

import h5py
import numpy as np

N_COLS = 3  # adjust to your number of columns

n_rows_dataset = 1000

file = h5py.File('/path/to/your/output.h5', 'w')
dataset = file.create_dataset(
    'my_dataset',
    (n_rows_dataset, N_COLS),
    chunks=(5, N_COLS),
    maxshape=(None, N_COLS))

current_row = 0
# this does not load the whole file but creates an iterator instead
for line in open("/path/to/the/large/file"):
    if current_row == n_rows_dataset-1:
        n_rows_dataset = n_rows_dataset + 1000
        dataset.resize((n_rows_dataset, N_COLS))
    dset[current_row] = np.fromstring(line, sep=" ", dtype=np.float64)
    current_row += 1

Note: You can find out the number of colums of your dataset by printing the first lines of your file:

i = 0
for line in open("/path/to/the/large/file"):
    print(line)
    i += 1
    if i >= 5:
        break