To answer the title of your question:

  Is Loss not a good indication of performance?

It is only a relative indicator of performance over the training session.

First, what is loss anyway? In general, the loss is some expression of the difference between the model's predicted output and the target output. Depending on the loss function used (e.g. if log is involved) or on the nominal values of the inputs themselves, the value of the loss can also be very small or very large.

People usually normalise data so the values are smaller, but the point is that you cannot always say that a validation loss of 0.0012345 is actually a good value. Or that 12345 is definitely bad!

Other possible loss functions

Other 3d segmentation models that I have seen it is common to use the Dice Coefficient for your cost/loss. Maybe give that a try.  The Dice coefficient is essentially the same as the F1 score; you are really finding a trade-off in how to penalise a model for its mistakes in classification e.g. of pixels or voxels. Do you want to strongly punich bad cases or rather a more averging approach. As that link points out, it is similar to the difference between the $L_1$ and $L_2$ losses.

There is also the Jaccard index, which is essentially the same as the Dice Coefficient.

The Tversky index is a generalisation of the two - it is an asymmetric similarity measure. :

$$
Tversky(A, B; α, β) = \frac{|TruePos|}{|TruePos| + \alpha |FalsePos| + \beta |FalseNeg|}
$$

The Dice coefficient is this with $\alpha = \frac{1}{2}$ and $\beta = \frac{1}{2}$
The Jaccard index instead has $\alpha = 1$ and $\beta = 1$

This might be a nice approach for your problem to tweak how the loss is computed.
There are plenty of source online to explain more about these well-defined measures and perhaps help you gain an intuition for their results.

Your results

With the losses you posted, it doesn't actually look as good as you explained. I am not sure exactly what your idx values mean, but the loss values are actually going up:

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: import matplotlib.pyplot as plt
In [4]: val_loss = np.array([0.029650183394551277, 0.009899887256324291, 0.0504908040
   ...: 1659012, 0.02019292116165161, 0.04724293574690819, 0.02810296043753624, 0.026
   ...: 42594277858734, 0.029894422739744186, 0.04158024489879608, 0.0457481481134
   ...: 8913, 0.05406259000301361])
In [5]: pd.Series(val_loss, index=range(0, 51, 5)).plot()
Out[5]: <matplotlib.axes._subplots.AxesSubplot at 0x7f4405326390>
In [6]: plt.show()

This suggests you might be overfitting. However, the differences between the predictions and ground truth in the images you show suggest there is a more fundamental problem.


Suggest an order to read data science books

I guess its common to start with
 "An Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani" and then, if you want a deeper picture, move to
 "The Elements of Statistical Learning, by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie".

I also really like the video course based on the first book, taught by Hastie and Tibshirani ( if im not mistaken) - a real gem in my mind.
From my perspective its a good way to get both practical and teoretical background for machine learning.
Link to the course published at R-bloggers:
https://www.r-bloggers.com/in-depth-introduction-to-machine-learning-in-15-hours-of-expert-videos/


how is correct usage of the validation split in neural networks?

I really like this post with the answer on your question.

Once we have the estimated skill, we are finished with the resampling method.
You finalize a model by applying the chosen machine learning procedure on all of your data.

It means that after achieving enough performance of the model, we don't need anymore splitting ("resampling methods"). Then we can merge all available data and repeat learning procedure. We can rely on the final performance (have no qualms about, who model will behave using new data during training), because it has already learned how to generalized input for the particular task (differ cats from dogs for example).

The careful design of your test harness is so absolutely critical in applied machine learning. A more robust test harness will allow you to lean on the estimated performance all the more.


Feature extraction; similarity and classification of accelerometer data

Regarding your first problem:
I would suggest that you do not discard the the amount of values even if it goes more than or less than 250. Instead what you can do is ; aggregate the values of accelerometer over a time intervals and then tie it up to a single action.

Regarding your second problem:
You have around 750 columns of the data. It would be very difficult to use random forest algorithm on it and get higher accuracy on it. You have to apply dimension reduction and then feature extraction techniques. You can go ahead with PCA (principal component analysis) f 750 independent variables. Reduce it down to 2 or 3 variable and check how much  variance these reduced variables can explain. if it less than 60%. You can apply T-SNE algorithm to extract features more on it.

P.S. to check if your reduced variable can explain your dependent variables (like squat, sitting, pushup), you can plot the scatter plot of reduce variable values and then color the values based on your dependent variable. you can click below link to understand what I am saying

https://blog.bioturing.com/2018/06/18/how-to-read-pca-biplots-and-scree-plots/


How to interpret classification report of scikit-learn?

The classification report is about key metrics in a classification problem.

You'll have precision, recall, f1-score and support for each class you're trying to find.

The recall means "how many of this class you find over the whole number of element of this class"
The precision will be "how many are correctly classified among that class"
The f1-score is the harmonic mean between precision & recall
The support is the number of occurence of the given class in your dataset (so you have 37.5K of class 0 and 37.5K of class 1, which is a really well balanced dataset.

The thing is, precision and recall is highly used for imbalanced dataset because in an highly imbalanced dataset, a 99% accuracy can be meaningless.

I would say that you don't really need to look at these metrics for this problem , unless a given class should absolutely be correctly determined.

To answer your other question, you cannot compare the precision and the recall over two classes. This only means you're classifier is better to find class 0 over class 1.

Precision and recall of sklearn.metrics.precision_score or recall_score should not be different. But as long as the code is not provided, this is impossible to determine the root cause of this.


Need help understanding xgboost's approximate split points proposal

I won't go into details but the following should help you grasp the idea.

They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ (sorted), you can try the $10$-quantiles split points $\{x_{10}, x_{20}, \cdots, x_{90}\}$ and have a good approximation already. This is what the $\epsilon$ parameter is doing. They consider a split point when the split has $\sim \epsilon N$ more points under it than the last split point. If $\epsilon = 0.01$, you will end up with $\sim 100$ split points, being larger than $\{1\%, 2\%, ..., 99\%\}$ of the other points. They do not consider a new split when "the sum changes more than $\epsilon$" but when the number of points under the current point is larger by $\epsilon$ than the last one.

Now, if you have a lot of continuous points very that are already well classified, it might be useless to split between them. You want to split the parts of your data set that are very wrong, the ones that are difficult to learn. To do so, they use weighted quantiles. This is where the weights play a role. The first $10$-quantile will not be the first point that is larger than $10\%$ of the points, but the first point that is larger than $10\%$ of the weights.


Visualization Tools that can show proper branch and merge in Graphs

You could use Tensorboard, via the tensorboardX interface.

That allows you to load models from PyTorch (and Chainer, MXNet etc.) into Tensorboard. This will then show you the full graph interactively.

From the homepage:


How can I calculate mean and variance incrementally?

This problem was discussed, with proof and some alternate methods over on math.stackexchange.


How to create vectors from text for address matching using binary classification?

I would rather recommend to consider each address as a separate instance. (Separate row in feature matrix)

Each address can be further broken into attributes like: House no., Street, City, State, Country.

Above attributes can be vectorised based on distance between them. Like vectorise each country from west to east(incremental numeric value). Then vectorise each state in country from west to east. Similarly for street. In case two entities fall on same longitude, assign consecutive numerical values.

Once each attribute is vectorised, similarity between two addresses can be Euclidean distance between vectors of two addresses.

Anyways leaving aside data science approach, this problem can be solved further simply by getting distance between two addresses using Google Maps API.


You can add conditions inside the loop if you want:

for (i in 1:10){
  if (i<=2){print("1-2")}
  if (i>2){print(" >2")}
}


General approach to extract key text from sentence (nlp)

Shallow Natural Language Processing technique can be used to extract concepts from sentence.
-------------------------------------------
Shallow NLP technique steps:

Convert the sentence to lowercase

Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words)

Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text (simply increasing n, model can be used to store more context)

Assign a syntactic label (noun, verb etc.)

Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb

-------------------------------------------
Lets examine the results of applying the above steps to your given sentence Complimentary gym access for two for the length of stay ($12 value per person per day).
1-gram Results: gym, access, length, stay, value, person, day
Summary of step 1 through 4 of shallow NLP:

1-gram          PoS_Tag   Stopword (Yes/No)?    PoS Tag Description
-------------------------------------------------------------------
Complimentary   NNP                             Proper noun, singular
gym             NN                              Noun, singular or mass
access          NN                              Noun, singular or mass
for             IN         Yes                  Preposition or subordinating conjunction
two             CD                              Cardinal number
for             IN         Yes                  Preposition or subordinating conjunction
the             DT         Yes                  Determiner
length          NN                              Noun, singular or mass
of              IN         Yes                  Preposition or subordinating conjunction
stay            NN                              Noun, singular or mass
($12            CD                              Cardinal number
value           NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
person          NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
day)            NN                              Noun, singular or mass

Step 4: Retaining only the Noun/Verbs we end up with gym, access, length, stay, value, person, day

Lets increase n to store more context and remove stopwords.
2-gram Results: complimentary gym, gym access, length stay, stay value
Summary of step 1 through 4 of shallow NLP:

2-gram              Pos Tag
---------------------------
access two          NN CD
complimentary gym   NNP NN
gym access          NN NN
length stay         NN NN
per day             IN NN
per person          IN NN
person per          NN IN
stay value          NN NN
two length          CD NN
value per           NN IN

Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym, gym access, length stay, stay value

3-gram Results: complimentary gym access, length stay value, person per day
Summary of step 1 through 4 of shallow NLP:

3-gram                      Pos Tag
-------------------------------------
access two length           NN CD NN
complimentary gym access    NNP NN NN
gym access two              NN NN CD
length stay value           NN NN NN
per person per              IN NN IN
person per day              NN IN NN
stay value per              NN NN IN
two length stay             CD NN NN
value per person            NN IN NN

Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym access, length stay value, person per day

Things to remember:

Refer the Penn tree bank to understand PoS tag description
Depending on your data and the business context you can decide the n value to extract n-grams from sentence
Adding domain specific stop words would increase the quality of concept/theme extraction
Deep NLP technique will give better results i.e., rather than n-gram, detect relationships within the sentences and represent/express as complex construction to retain the context. For additional info, see this

Tools:
You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort. Below is the sample R code I used for PoS tagging.
Sample R code:
Sys.setenv(JAVA_HOME='C:\\Program Files\\Java\\jre7') # for 32-bit version
library(rJava)
require("openNLP")
require("NLP")

s <- paste("Complimentary gym access for two for the length of stay $12 value per person per day")

tagPOS <-  function(x, ...) {
  s <- as.String(x)
    word_token_annotator <- Maxent_Word_Token_Annotator()
    a2 <- Annotation(1L, "sentence", 1L, nchar(s))
    a2 <- annotate(s, word_token_annotator, a2)
    a3 <- annotate(s, Maxent_POS_Tag_Annotator(), a2)
    a3w <- a3[a3$type == "word"]
    POStags <- unlist(lapply(a3w$features, `[[`, "POS"))
    POStagged <- paste(sprintf("%s/%s", s[a3w], POStags), collapse = " ")
    list(POStagged = POStagged, POStags = POStags)
  }

  tagged_str <-  tagPOS(s)
  tagged_str

#$POStagged
#[1] "Complimentary/NNP gym/NN access/NN for/IN two/CD for/IN the/DT length/NN of/IN stay/NN $/$ 12/CD value/NN per/IN     person/NN per/IN day/NN"
#
#$POStags
#[1] "NNP" "NN"  "NN"  "IN"  "CD"  "IN"  "DT"  "NN"  "IN"  "NN"  "$"   "CD"
#[13] "NN"  "IN"  "NN"  "IN"  "NN"

Additional readings on Shallow & Deep NLP:

Integrating Shallow and Deep NLP for Information Extraction


Has anyone done any benchmarks?

Yes, the 2014's benchmark in question has turned into foundation for db-benchmark project. Initial step was to reproduce 2014's benchmark on recent version of software, then to make it a continuous benchmark, so it runs routinely and automatically upgrades software before each run. Over time many things have been added. Below is high level diff of the 2014's benchmark comparing to db-benchmark project.
New:

continuous benchmark: runs routinely, upgrades software, re-run benchmarking script
more software solutions: spark, python datatable, cuda dataframes, julia dataframes, clickhouse, dask
more data cases

two new smaller data sizes: 0.5GB (1e7 rows) and 5GB (1e8 rows)
two more cardinality factors: unbalanced, heavily unbalanced
sortedness
data having NA

advanced groupby questions

median, sd
range v1-v2: max(v1)-min(v2)
top 2 rows: order(.); head(.,2)
regression: cor(v1, v2)^2
count and grouping by 6 columns

benchmark task: join

Changes (see groupby2014 task for 2014 fully compliant benchmark script):

using categorical/factor instead of character
cardinality of v2 and v3 measures increased
function calls are NA-aware
aggregated columns are named
order of groups is irrelevant whenever possible
extra call to dim()/.shape is included in timings to force lazy evaluation
machine that runs benchmark has 128GB mem (not 244GB mem)
no 100GB (2e9 rows) data size

We are planning to add even more software solutions and benchmark tasks in future.
Feedback is very welcome, feel invited to our issue tracker at https://github.com/h2oai/db-benchmark/issues.

Is pandas now faster than data.table?

According the our results pandas is not faster than data.table.
I am pasting medium size data 5GB (1e8 rows) groupby benchmark plot taken from the report at h2oai.github.io/db-benchmark as of 20210312. Consult the h2oai.github.io/db-benchmark#explore-more-data-cases for other data sizes (1e7, 1e9), data cases (cardinality, NAs, sorted), questions groups (advanced), or tasks (join).
For up-to-date timings please visit https://h2oai.github.io/db-benchmark.


What's the best way to expose the method (say in this case getNextPossibleDestByUser) as a API (pass customer name as input and get the destination as output) ?

Use flask. Pretty easy to build an API.

  Whenever the API is being called , does that mean all the 10 million records gets processed each time ? Are there anyway to optimise that ?

It depends on the exact application but for you case you can take the passenger as an input from the call to your API as POST body or in the arguments and only iterate over the subset of data for that passenger.

  Rather than dataframe (pandas) query approach can I consider some ml models or utility functions from say scikit to solve the same problem ?

If the problem is -- as you illustrated -- simple frequency calculations then better to stick to pandas, in my opinion. If your problem is solved by the simple approach it would not make sense to go for a complicated approach.


I do not understand your problem, the first layer, i.e. input layer, of your neural net must have a size equal to the number of variables you have. The last layer, i.e. output layer, must have a size which is equal to the number of values you want to predict. Please be kind and tell me what you want to know more specifically, if my answer doesn't satisfy you.


The method you have here is for time series, so input and output are a set of values over a time period.

The sum method sum(y) here is a short hand (abbreviation) for the average over the last mentioned equation, it is not a conventional sum.

Also do note argmax k over the sums is done in separate dimension for each k representing the individual class.

So in my understanding sum is needed only because it is a series of data and sum is calculated per class for the whole series.

The y target (n) is 1 in only in one dimension when match happen, so argmax k finds with highest value the correct class.


Why is stochastic gradient descent so much worse than batch GD for MNIST task?

Why is stochastic gradient descent so much worse then batch GD for MNIST task?

It isn't inherently worse. Instead, by changing just one parameter on its own you have adjusted the example outside of where it has been "tuned" to work, because it is a simplified example for learning purposes, and it is missing some features that most users of NNs would consider standard.

The batch size of 1 is performing just fine. In fact, although it takes longer to process the same number of epochs, each epoch actually has more weight updates. You get 100 times as many weight updates, although each one has far more noise in it than a batch size of 100. It is these extra weight updates, plus extra time spent running interpreted Python code for 100 times as many batches, which adds the large amount of time.

The problem with accuracy is that the example network has no protection from overfitting. By running so many more weight updates, the training starts to learn the precise image data to match each digit, in order to get the best score. And by doing that exclusively it learns rules that work really well on the training data but generalise really badly to new data in the test set.

Try batch size of 1 and number of epochs = 3 (I tried this and got accuracy of 94.32%). Basically that is using early stopping as a form of regularisation. It's not the best form of regularisation, but it is quick to try and often effective - the problem is how to tell when to stop, so you need to measure a test set (often separate to final test set, called cross-validation set) at any potential stopping point, and save the best model so far. That will obviously involve adjusting the example code.

Probably the 15 epochs in the original example has been chosen carefully so as to not make overfitting a problem with batch size of 100, but as soon as you change batch size, without any other form of regularisation, the network is very likely to over-fit. In general neural networks strongly tend to over-fit, and you have to spend time and effort to understand and defend against this.

Have a look at regularisation in TensorFlow for other options. For this kind of problem, dropout (explained lower down page in the link) is highly recommended, which is not purely regularisation, but works to improve generalisation for many neural network problems.


Is there a way to drop duplicated rows based on an unhashable column?

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd
    ...:
    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } )
    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ]
    ...: lbls = [ 'x' , 'y' , 'z' ]
    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)

In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))

In [3]: df.drop_duplicates(subset="z_tuple", keep="first")
Out[3]:
   x  y       z z_tuple
0  a  b  {b, a}  (b, a)
1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)

In [5] : df
Out[5] :
   x  y       z
0  a  b  {b, a}
1  b  c  {c, b}
2  b  a  {b, a}


Machine learning classification with time-domain signals how to ignore signal arrival time?

A common approach is what you suggest in 1. - apply time-shift as a Data Augmentation strategy. The augmentation is generally beneficial with deep learning models, and GPUs are fast so the compute time is rarely a big problem.
Another strategy, less common, would be to make sure that the event is always located at the same position inside the analysis window during training. At prediction time one uses a high degree of overlap between analysis windows, to ensure that the model has a good chance of seeing the signal in the correct position.
If you have high quality labels, aligned time-stamps for each event, this is rather straight forward.
A third approach would be to use a combination of a detector model and a classification model. The detector job is to take the input signal stream, and detect the presence of a possible event - then possible events are passed to the classifier. A well designed detector would have a rather predictable trigger time, so the classifier does not have to be time-shift invariant. This is easy enough for signals that have a very clear event vs not-event signatures, but hard for more complex signals or low SNR conditions.


Deep learning - aesthetics data modelling

Modeling aesthetics in media is an example of ordinal classification.
One of the most actively maintained datasets for this can be found is Jen Aesthetics
A relatively recent paper using deep learning towards aesthetics modeling is
this

Prior to deep learning era, research groups were trying to translate methods/guidelines used in the photography community to create/capture good quality pictures. There are several guidelines that you can explore with a bit of search online. One popular example is the 'rule of thirds'. Here the primary subject should not be centered in the image but offset and ideally centered at the intersection of 1/3 and 2/3 horizontal and vertical lines.

This is easy to translate into an algorithm: use salient object recognition or visual attention detection and measure the distance of the center of the salient/attention patch from the 4 rule-of-thirds points. Use this off-set as a feature. The closer the salient patch is to any one of the rule-of-third points, the higher the aesthetic ordinal score for that image.

This is another good paper that explores what makes images popular.

Some researchers have also used the tags or descriptions of photos as features. The objective here is to learn an association between lexical features and image aesthetics. They have sourced their data from online repositories like Digital Photography Challenge.

This subjective task is needless to say very complex. If you plan to address it, I'd recommend beginning with a clear definition of the context within which you aim to address aesthetics.

Ideally, you'd like to map any given image (media) to some value in [0, ..., 1] $\in \mathcal{R}$.
However, this is very difficult unless you have access to a lot of training data. I suggest trying instead to simplify the problem. If you can reliably map images to just two classes, good aesthetics and bad aesthetics.

You can successively generalize from binary classification to full fledged multi-class ordinal classification, for which you'll very likely have to keep increasing the depth of your CNN.

Good luck! Since, there is more to aesthetics than meets the eye! :-)


Converting a Pandas Series string of multiple attributes into individual attributes?

This might Pandas convert a column of list to dummies.
You can try using the MultiLabelBinarizer class from sklearn.


Why 100% accuracy on test data is not good?

I see two ways to go:
1 - There is an error
2 - There is no error.

1 - Look for the error

You probably have commited Data Leakage.  You have added the target in one of the features and the model found out.
The validation is not right, you have a time series and you have done random validation.
Your test has only a few instances or it is unique.
The test is repeated from the train.

2 - There is no error

If the prediction is right and you have 100% accuracy, then no need to do Machine Learning. Open the model find where is taking the decision and don't do machine learning, do classical modeling.

For example if your model is a decision tree, just plot it or print it and get the decision and put them yourself.

This sometimes happens when modeling a previous developed algorithm. The new ML model is able to learn what was going on before.


Sklearn LocalOutlierFactor contamination parameter

You are specifying with a floating point number what proportion of the data you are fitting on is an outlier. If you use 'Auto' it will default to 0.1. Note that in the current documentation, there is a changed note specifying that it will default to 0.2 in a future version.


How to choose dimension of Keras embedding layer?

There is no "Right" answer to this question , but you should take in mind the following guidelines:

Embedding layer is a compression of the input, when the layer is smaller , you compress more and lose more data. When the layer is bigger you compress less and potentially overfit your input dataset to this layer making it useless.

The larger vocabulary you have you want better representation of it - make the layer larger.

If you have very sparse documents relatively to the vocabulary, then you want to "get rid" of unnecessary and noisy words - you should compress more - make the embedding smaller.


The robustness of the Frobenius and L2,1 norms to the outlier

I have only a couple of hints:

Frobenius norm, by definition, takes equal account of all data in matrix (all rows and columns).
Whereas $L_{2,1}$ norm is Frobenius norm but per row instead, so outliers in other rows do not affect (equally) norm of current row.


Model assessment for EXTREMELY imbalanced dataset

I would only look at the precision/recall scores of the undersampled class (the positive class in your case).

Checking the performance on the oversampled class seems quite meaningless, since it is quite easy to achieve very high scores.

Then, balancing the precision/recall with a F-beta score will depend on your specific use case.


How do I train a model with pre-saved weights?

As you said:

  Is it possible to preprocess the data in batches and train a model for the small data set

Yes! In practice, training of Neural Networks always happens with batches. You never fit the whole dataset at once in the model, whatever medium sized dataset could crash any machine.

This is how it works: You extract a slice of your dataframe (the number is usually in the range of 32-256 observations, but it's very task-specific), you train the model on that, and you move to the next until all dataset is completed. This is what an epoch is, in the Deep Learnig jargon.

  ... and then save the weights and then train it again for the next batches with the saved weights?

There's no need to save and reload the model at each iteration, just save it once it was trained for the number of epochs of your choice. If the following day you want to keep training, just reload it and keep iterating one batch at a time.


Python: Detect if data of a time series stays constant, increases or decreases

First i want to calculate the mean offset and the standard deviation
  of the measurement data overall, during halt, increasing and
  decreasing state.

I would recommend checking windows of your dataset (let's say 10 samples in a row). A very convenient method to do that is pandas.DataFrame.rolling(), please take a look at it :)


Choosing an algorithm with normalized data(Classification)

Re-scaling or any other form of standardization/normalization is very useful when dealing with models that are trained with gradient descent: (SVM,NN, LogReg).

This question explains the effect of normalization on the gradient rather well: https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re

Decision trees for example are invariant to linear transformations, therefore feature scaling, in theory, does not effect the model in any way.

In the case of your set-up, you have a binary classification problem.

I would recommend trying out first a linear classifier on your data. Logistic regressions is a good first choice. If you decide that LogReg is not working as well as you would like it to then you can move on to more complicated models. I would recommend using decision trees coupled with a gradient boosting model.

In problems that are similar to what you described, specifically when there is no inherent structure in the data (as there is in image classification, speech recognition, etc.), gradient boosting models tend to outperform neural networks.

Hope this helps.


1) How do I process each sample individually? Do I go about this by setting batch size = 1? Or is there any advantage to disabling automatic batching. If so, how do I go about this.

If you set batch size to 1, then you are effectively disabling batching.  Samples will be processed one-at-a-time and gradients will be computed for single samples.  This is not necessarily a problem - just keep in mind that you'll lose the advantages of mini-batching.

If you are using PyTorch's DataLoader, then just set batch_size and batch_sampler to None.  According to the docs, this will disable automatic batching.

2) How exactly do I delete the sample from the dataset for further epochs. Is there any way to skip this sample in the DataLoader?

You can implement a custom PyTorch dataset.  Internally, keep track of which samples have been deleted.  If you're using an iterable-style dataset, then the next step is pretty easy.  Just skip over the deleted samples in __iter__().

Map-style datasets will be a little more tricky.  Your len() function will need to return len(dataset) - # of deleted samples.  And you will have to find some indexing scheme that prevents deleted samples from being accessed.  One naive idea is simply to reindex after every deletion.  The obvious problem is that the DataLoader's random sampler won't have any way to know this, so a single sample could be processed multiple times in one epoch.  Another idea is to index your samples by some unique key rather than integers.


There is generally a trade of in k-NN:

k should be large enough to cancel out any noise.
k shouldn't be too large to create big biased boundaries.

One rule of thumb is to select an odd k value to avoid ties in binary class problems.
Small and big value of k is subjective to the underlying structure of the data set itself, that's why we check over a range of k values.
Both of the above gets settled if done a good job with cross validation.
So the most important question you need to focus is:

Have you chosen the right metric for model selection, keeping the imbalance of classes into consideration?

I would recommend you to first identify the class that is more important for you to predict correctly. Then, look at the confusion matrix and ponder upon different metrics (eg. recall, precision etc.) before finalizing a model/ k value.


The idea of adding external variables is not new. ARIMAX model (https://robjhyndman.com/hyndsight/arimax/) was specifically created to include them.

If you ask whether it is ok to include the external variables of some particular period then the answer is that you need to understand when they are available for prediction. If you predict a profit of the company at an end of a year based on today's stock price then it is ok. The profit and loss statement appear much later than the information is available in stock price. And stock price should have some info on expected profits based on the financial theory. If you want to predict tomorrow's stock price based on tomorrow's number of sun spots then it is a bad idea. The number of sun spots tomorrow is not available today.

And generally you can use any combination of lags for different variables. Taking today's stock price and last year's profit to predict this year's profit is totally valid and even has financial theory supporting the relationship.


Which libraries in Python are there in NLP to tokenize the Hindi sentence?

There is an Indic NLP library developed by iitB for Hindi texts. you can check out the below links

https://www.cse.iitb.ac.in/~anoopk/pages/softwares.html

https://github.com/anoopkunchukuttan/indic_nlp_library


How an input data flows through an lstm layer cells?

The test data is a 3 step sequence data i.e. 70,80,90.
Following will be the flow -

70 will reach the 1st LSTM layer i.e. to the all 100 in parallel
It will also come back by the Recurrent weight to be multiplied with the next seq.
80 will reach and add up with the Recurrent of 70.
Similarly, 90 will pass through the 1st LSTM layer.
All the seq_step is held at 1st layer because return_sequences=Flase
After the last seq_step, the output will reach the next layer
RepeatVector will duplicate the vector, so the single output will become 2 step input for the next LSTM layer.
When the first of the duplicate will reach the LSTM, it will behave exactly like the previous LSTM layer. In addition, it will pass the output to the next layer too because return_sequences=True here.
Same thing will repeat for the 2nd duplicate vector.
The role of TimeDistributedlayer is to wrap the two incoming data as two steps and hence you will receive a (2,) output.

How vectors look across the Recurrent Layers

This is intuitive for a single Neuron/Single Feature i.e. as depicted below

$\hspace{5cm}$

For a multi-feature, multiple Neuron, it goes like below
 Your case has a single feature for each step and 100 Neurons. The below depiction has 3 features and 4 Neurons. Features behave the way it does in a simple feedforward network.
The only new information here is that recurrent of each Neuron goes back to each of the 100.

$\hspace{7cm}$ Image credit - SO Answer


Open Calais is a free-to-use tool for entity recognition and relationship mapping. It's from Thompson-Reuters so may not be wholly suitable for technical language but worth a try. Has python bindings


Feature selection for gene expression dataset

It would be helpful if you described your dataset more.  Gene expression datasets seem to often have very high dimensionality and Lasso regularized logistic regression is a popular method to approach this problem.  This paper takes it a little further and might help you out:
http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-198

Random forest can generally certainly provide a meaningful importance ranking, but it also depends on what your dataset looks like.

mRMR sounds like it is specifically designed for identifying gene characteristics, so definitely give it a try.

There's also Principle Component Analysis which is also used for gene expression data.

Lots of options, but your questions is not detailed enough to go any further, and providing code as a solution at this point isn't realistic. The documentation for Python scikit-learn has many good explanations and examples.


Best place for Bengin files for training a classifier?

You can use VirusTotal. VirusTotal got a Private API where you can query files (for example only those that has 0 positives) and then download the files, you need to contact them to understand how much it cost.

If you don't need the binary itself and you are using different information from the files structure you can use endgame's dataset


Transforming words in sentences into vector form to prepare a model

An approach would be to sort out all the words in your data according to how often they appear, i.e. their "frequency". After that, pick the "X" most frequent words in your dataset to use them for the classification of your dataset.

Assuming that you are working with Python and Keras, you should use the Embedding layer. For more details about how to use that layer, check this.

Shortly, what this layer does is that it maps the input to a high dimensional vector domain. A word is converted to a real-valued vector and word similarity is evaluated by the "closeness" of two word-vectors in the high-dimensional vector space.

Also make sure that your dataset consists of texts of fixed-length, by truncating long sequences or zero-padding short ones.

After all this is done, you can train a recurrent neural network with LSTM neurons as a text classifier. LSTMs have been proven very successful in text processing due to their inherent memory.

A hands-on Python/Keras tutorial that demonstrates all the above can be found here, I am sure it will be of high help :)


CART algorithm (Classification and regression trees) question

Making an additional split will always decrease $Err$ (until pure leaves are reached), so any negative $\alpha$ will have the same minimizing $T$ as $\alpha=0$.  So we might as well take $\alpha\geq0$.  (But there is no requirement that $\alpha$ be an integer.)

Erm, unless you implement things so that even a pure node can be split, in which case a negative $\alpha$ will make the tree split even these, until every leaf consists of a single sample (and duplicates thereof) or the maximum depth is reached.


Classification algorithm that only matches trained examples

Strictly speaking Machine Learning is not the answer to this problem imho, because a ML method always works by generalizing what is in the data. In other words ML methods are meant to make some guesses while statistically minimizing the risk of error: if you don't want any generalization or risk of error, you don't want ML.
There might be some symbolic methods which could do what you describe, but essentially it's just a very simple deterministic method:

Obtain the frequency for each distinct instance (including label) in the training data in a map
For each distinct instance (only features) keep the most frequent one and discard all the others.
When applying, just look up the instance in the map and assign the corresponding label.

a cumbersome db infrastructure with columns, keys, indexes and other boring things.

There's a confusion here: the question of using a database or not is irrelevant here since it's a matter of how you store the data, it's just not related to using ML or not.
Usual advice: use the right tool for the right job... even if it's boring ;)


How to properly use oversampling without inflating results?

First of all - have you tried to do modeling without oversampling? If you dont have imbalanced classes, why do you need oversampling? You dont always need tons of data to get a good model as a baseline.
Regarding should i apply the SMOTE only in train data - always leave some test data out, dont apply any transformation on the whole dataset (even normalization or missing values imputing). If you have some preprocessing - do it on train data set, train model. Then apply the preprocessing pipeline on test data and predict on it.
simplest models such as a 15-neuron MLP. - it doesnt sound as simplest solution to me. have you tried any other approaches? also, have you checked confusion matrix to see the classification error on test data for each of the class?


Multiple categories within a variable in decision tree

Your intuition that a categorical feature is treated as multiple dimensions (i.e., multiple features) is correct. Millions of categories will be problematic for tree-baed algorithms that search through all features/categories for each split. A random forest would be a good model to try for your project, since it samples the features. Most software implementations let you set the sample proportion, so you could set yours really low if need be to speed up the split searches.

Alternatively you could feed your dataset into a Lasso or Elastic Net model to remove noisy variables and substantially reduce the quantity of categories under consideration. Lasso and Elastic Net have both been shown to do well on datasets where the number of features far exceeds the number of observations (e.g., dealing with genome data).


openai gym - what is an agent I can use with a multi-discrete action space?

OpenAI Baselines - or for me even better, Stable Baselines - has many model options which can handle MultiDicrete Action and/or Observation spaces.  Building a custom gym environment is also quite straightforward.


What is the difference between different batch_sizes in Keras Sequential models?

I need for future prediction using batch_size=1 but affraid of getting bad/wrong result!

Batch size is not connected to the future prediction - it's just the number of samples which will be used in one forward pass to calculate the loss and then in a backward pass to calculate the gradients and parameter updates.

In order to regulate the future prediction (as far as I understand from your question), you just need to define the sample size (or width, or time steps, whatever name you use ...).

Regarding the "important effects of choosing batch_size from 1 to N numbers?" - with a larger batch size your training process might finish faster, but with smaller you might avoid local minimum (but that - local minimum, generally, shouldn't be a problem with deep neural networks).


Recommender systems and Machine Learning

For experience:

observing a set of examples encoded in a rating matrix is good, before doing that perform hybrid filtering(content-based filtering + collaborative), from its results
get insights and do rating matrix if needed.

Performance:

use rmse metric for better understanding of accuracy.


I am currently doing a project with similar industrial image processing task. I am detecting defects in a rectangular product image and from my experience, horizontal and vertical flipping of images has helped me a lot to augment my dataset and increase F1 score. Since I am not sure regarding your picture orientation I am unable to answer regarding other rotations. But if it has equal height and width then rotation of dataset makes sense. Try incorporating residual units into your CNN which has helped me massively since I had to use deeper networks to extract intricate features.


How to treat outliers in a time series dataset?

Decide how auto-correlative your usual event in the time series is.
For example, "I'm tracking temperature over time and it rarely changes more
than 30 degrees F in an hour".
Throw out or smooth any values where the observed value changes
more than that. In other words, "If ever I see the temperature
changing more than 30 degrees in an hour, I'm going to ignore that value and substitute the average of the prior and the next value because that must be a sensor malfunction".

Once you were comfortable with doing that, use something like the standard deviation of the data over a rolling window instead of an absolute, arbitrary value like I did.


Logistics Demand Forecasting with 20k Different Time Series

First of all, there is no guarantee that you can beat baseline with ML, this is something you should take into account.
You also might find a solution for some particular Type Of Demand due to some inner behaviour, but this might not generalize well for the rest of the cases.
I would also try to use Facebook`s Prophet
From the authors:
Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.
It might work well out of the box, without any major tweaks. You can also run in in parallel (because the process of fitting of model is kinda automatic), which is helpful when you have this much time series.
Here is a link on how to implement it efficiently (but note that original docs are quite good as-well)
https://towardsdatascience.com/implementing-facebook-prophet-efficiently-c241305405a3
And a post regarding parallel execution
https://medium.com/spikelab/forecasting-multiples-time-series-using-prophet-in-parallel-2515abd1a245
I would also spend more time analyzing results: can you check the error between different types of demand, search for the particular cases when prophet beats baseline and try to understand why its so and so on.


You can do a pandas.DataFrame.join if you know how this works.

-- Edit: merge is apparently a better choice: See the example at the end.

I think you need an outer join on Keyword.

This should give a new DataFrame, that contains unique rows for the Keyword in both tables. Some entries may be NULL/None. This indicates that in the old or new table, the keyword was not present and you should treat is as a new keyword, or a keyword that has dropped from the list.

Rename the columns in the new table appropriately, and then apply a smart value between columns, taking into account that some values are NULL.

You can do a similar thing in Excel: https://superuser.com/questions/1023123/how-to-simulate-a-full-outer-join-in-excel

Edit:

Minimalistic example:

import pandas as pd

old = pd.DataFrame({'keyword': ['football', 'soccer', 'rugby'], 'position': [2, 1, 3]})
new = pd.DataFrame({'keyword': ['hockey', 'rugby', 'soccer'], 'position': [3, 2, 1]})

old.keyword = old.keyword.astype(str)
new.keyword = new.keyword.astype(str)

old.set_index(['keyword'])
new.set_index(['keyword'])

old = old.rename(columns={"position": "position_old"})
new = new.rename(columns={"position": "position_new"})

print(old)
print(new)

merged = pd.merge(old, new, how='outer', on='keyword')
print(merged)

Output:

    keyword  position_old
0  football             2
1    soccer             1
2     rugby             3
  keyword  position_new
0  hockey             3
1   rugby             2
2  soccer             1
    keyword  position_old  position_new
0  football           2.0           NaN
1    soccer           1.0           1.0
2     rugby           3.0           2.0
3    hockey           NaN           3.0


99% Accuracy is over fitted model and its not the accurate model. There will always be loss associated. secondly you sample size is too small for reaching the global minima. you should also check your ROC-AUC curve for the model.


How to use Cosine Distance matrix for Clustering algorithms like mean-shift, DBSCAN, and optics?

Each of those selected clustering algorithms can be fit using cosine distances in scikit-learn:

from sklearn.cluster import DBSCAN, MeanShift, OPTICS
from sklearn.metrics.pairwise import cosine_distances

# Define clustering algorithms
algorithms = [DBSCAN, MeanShift, OPTICS]

# Placeholder for results
results = dict.fromkeys((a.__name__ for a in algorithms))

# Fit each clustering algorithm and store results
for algorithm in algorithms:
    results[algorithm] = algorithm(metric=cosine_distances).fit(X)


Parameters in GridSearchCV in scikit-learn

Well the error message is quite clear. GridSearchCV accepts only lists. Therefore 'random_state': [7]} will solve the issue.

However when you have only one value with this parameter, it makes more sense put it directly into the classifier as you did with n_estimators.


Assuming that letters are indicative of "motifs" and numbers are considered as digits and not exact numbers, this is what I would do:

First - transform numbers into a digit placeholder (#)

import re
s = re.sub("\d",s,"#")

Then I would transform a string into a bag-of-bigrams vector in the char level

from sklearn.feature_extraction.text import CountVectorizer
vecs = CountVectorizer(s, analyzer="char_wb", ngram_range=(1,3))

After these 2 steps, we got a sparse vector from any string:

'ab123' --> {" a":1,"ab":1,"b#","##":2,"# ":1}

Next we want to convert those vectors to a pairwise distance matrix and cluster by that distance.

from sklearn.metrics.pairwise import pairwise_distances
from scipy.cluster import  hierarchy
def link_cluster(X, threshold=0.1, metric="cosine", algo="average"):
    X = X.todense()
    Z = hierarchy.linkage(X, algo, metric=metric)
    C = hierarchy.fcluster(Z,threshold, criterion="distance")
    return C

Your end result would be

C = link_cluster(vecs)


Detecting specific points in (noisy) dataset

This is a common problem in signal processing. It would be useful to look into the rich history of anomaly detection in signal processing.

One option would be to apply a Fourier Transform which decomposes a function into its constituent frequencies. Define a baseline set of frequencies. Then anomaliness frequencies could be found relative to the baseline.


Classifier design for website screenshots

Yes, instead of image classification you should look to scrape the websites themselves to get to file links or down messages directly.

You could still use an ML to determine down messages with minimal user input but it will be much faster and easier than image recognition.

Image recognition will fail you here because to start you have a population of 100 or so which is a small sample and you'd need to hand label at least 50 to train any kind of viable classifier at which point you might ad well just hand label them all.


I strongly suggest Talking Machines. It's a very well put together podcast from a professor at Harvard. They cater to both machine learning experts and enthusiasts.

Their interviews are often done from NIPS, and the guests are usually top tier practitioners.


Tune hyperparameters for cost-sensitive classification

Tuning the hyperparameters of a model is still more of an art than a science.

Both R (caRet) and Python (sklearn) have frameworks for performing grid searches of hyperparameters using cross validation. You can define custom cost functions in these frameworks, which will allow you to directly assess your cross-validation results using your cost matrix. See the scikit-learn documentation for Python, or the caRet documentation for R. You will get much better results using your cost matrix directly to evaluate your models, instead of using F-1, precision, recall, accuracy, or any other metric.

Other Guidance

It is important to understand what the different hyperparameters actually mean in the context of each learning algorithm to help guide the parameter search, as computational limitations usually strictly limit the number of different models you can build total, limiting the size of the grid you can realistically search.

With such a low percentage of negative examples in your set, you will want to ensure that you use stratified sampling for your cross-validation. This will ensure that each fold has the same percentage of negative examples as the base set. Otherwise, your cross-validation results will likely be biased by the non-uniform negative class distribution.


Convolutional neural network for sparse one-hot representation

I would not apply convolutional neural networks to your problem (at least from what I can gather from the description).

Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern 1 0 1 seen in features 9,10,11 is similar in some way to the same pattern seen in features 15,16,17. Having this assumption built in to the model allows you to train a network with far fewer free parameters when dealing with e.g. image data, where this is a key property of data captured by scanners and cameras.

With one-hot encoding of features, you assign a feature vector index from a value or category essentially at random (via some hashing function). There is no meaning to translations between indices of the feature vectors. The patterns 0 0 1 0 1 0 0  and 0 0 0 1 0 1 0 can represent entirely different things, and any associations between them are purely by chance. You can treat a sparse one-hot encoding as an image if you wish, but there is no good reason to do so, and models that assume translations can be made whilst preserving meaning will not do well.

For such a small sparse feature vector, assuming you want to try a neural network model, use a simple fully-connected network.


k-means is not a good choice, because it is designed for continuous variables. It is a least-squares problem definition - a deviation of 2.0 is 4x as bad as a deviation of 1.0.

On binary data (such as one-hot encoded categorical data), this notion of squared deviations is not very appropriate. In particular, the cluster centroids are not binary vectors anymore!

The question you should ask first is: "what is a cluster". Don't just hope an algorithm works. Choose (or build!) and algorithm that solves your problem, not someone else's!

On categorical data, frequent itemsets are usually the much better concept of a cluster than the centroid concept of k-means.


Does experience with Keras count as experience with Tensorflow?

Currently, Keras is part of Tensorflow, so if you are using the Keras that comes bundled with Tensorflow (tf.keras), technically one is a subset of the other. This was not always like that (see this), so if you are using the old version of Keras that is a separate package, then technically it hides the complexity of Tensorflow, so your experience would be only Keras.
About tutorials not using the Keras API (I understand that you are referring to using model.compile, model.fit, etc), you should understand that tutorials are normally introductory material, which makes using only Keras a sensible decision, avoiding the rougher Tensorflow reality (static graph vs. eager execution, sessions, etc). Bare Tensorflow (no model.fit) is used all over the place for real-world use.
Anyway, in the end, it is up to employers to decide whether your experience fits their needs.


How to convert DataFrame column to Rows in Python?

Try using pandas melt
dataT = pd.DataFrame({"userID":[1,2,3],"date":["2020-01-01","2019-01-02","2020-08-12"],"-7":[0,1,1],"-6":[1,0,0],"-5":[0,0,0]})

Input:

dataT.melt(value_vars= ["-7","-6","-5"], value_name="count")

Output:

Update
By taking the comment by Benji the code would be:
dataT.melt(id_vars= ["userID","date"], value_name="count")

With output:


How does real world machine learning production systems run?

Yes, it is typical to have some persistent representation of the model that is uploaded, and yes, it is typically very big as a file/files. Using pickle is one way to do it, commonly used with scikit-learn, for example. Deep learning frameworks typically have their own formats, but nothing stops you from using pickle with them as well, except that it is more complicated and less efficient as an approach.

I'm not sure I understand the second part of the question, but if you want to modify your model online, nothing stops you from creating a new pickle. It is advisable that you run a new training as a separate batch process in the background, to avoid blocking your site or web service, especially if your web application server is single-threaded. Furthermore, you need to be wary of utilisation of server resources, so you would be better off running batch updates like that in low-traffic periods. This could be on over the weekend, or if your user base is predominantly regional, at night or on public holidays.


Is it good practice to remove the numeric values from the text data during preprocessing?

Is it recommended to discard this numerics before creating a
vectorizer(bow/tf-idf) for any model(classification/regression)
development?

It depends on the problem statement for example year could be significant if you want to find the trend and year has many unique value but if it's constant then you can remove it.
To add to that if you are doing sentiment analysis then numeric variables don't make much sense.


Why does my cost function return double the expected value?

You are supposed to perform calculations on cost function element wise. Try using

np.multiply

instead of

np.dot

which is an inner product for Matrices.


How to create a dummy model in Tensorflow

The reason is that in your model, you have specified the input size to vary, as the parameter of your function specifies. You have to change your code as follows:

def neural_network_model(input_size, output_size):
network = input_data(shape=[None, input_size, 1], name='input')

network = fully_connected(network, 128, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 256, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 512, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 256, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 128, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, output_size, activation='softmax')
network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')
model = tflearn.DNN(network, tensorboard_dir='log')

return model

You have hard coded the output size, which is two, you have to change it to be a variable and during calling, you have to specify the size of your new output, which is four.


Displaying network error as a single value

You can simply sum the absolute m error to have 1 overall error value of the neural network. You can also compute mean square error. Whatever error you choose will not change much, in both cases the smaller the error the better your neural network. Even if by convention we prefer to take the mean squared error.


How does resnet model restores the skipped layers as it learns the feature space?

In the architecture of resnet there are skip connections between certain blocks. Now, the significance of these skip connections is that during the initial training weights are not that significant and due to multiple hidden layers we face the problem of vanishing gradients. To deal with this researchers introduced residual connection which connects the output of the previous block directly to the output of the next block and thus information keeps on transmitting. Like, just see the architecture of the residual network:

  Ref: https://missinglink.ai/guides/pytorch/pytorch-resnet-building-training-scaling-residual-networks-pytorch/

In VGG-19 the information is directly being transferred from one hidden layer and in ResNet the output of one block is being transferred to next block's input as well as to the ouput of the next block, so that if we face the vanishing gradients problem due to the hidden layers of this block network can deal with it.

To your query of why does the activation is not skipped, the output of both previous and next block is concatenated and then passed through an activation layer, that's why it doesn't skips the activation layer.


How to improve tensorflow 2.0 code for policy gradient?

If possible, try to use just tensorflow functions (put the .numpy() part out, for example), and put a @tf.function decorator on top of train() function.

The role of @tf.function is that it transforms a whole function into a tensorflow op. The whole function will be execute an order of magnitude faster than a normal Python function.


Imbalanced classes (balance of train, validation, and test)

If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Is there a loss function that measures the cross similarity between two 2D tensors?

Currently, I've implemented the following solution inpired by N-Pair Loss published from NIPS 2016:

import torch
from torch import nn
from matplotlib import pyplot as plt
import seaborn as sn

class NPairsLoss(nn.Module):
    """
    The N-Pairs Loss.
    It measures the loss given predicted tensors x1, x2 both with shape [batch_size, hidden_size],
    and target tensor y which is the identity matrix with shape  [batch_size, batch_size].
    """

    def __init__(self):
        super(NPairsLoss, self).__init__()
        self.ce = nn.CrossEntropyLoss()

    def show(self, similarity_scores):
        sn.heatmap(similarity_scores.detach().numpy(), annot=True, annot_kws={'size': 7}, vmin=-1.0, vmax=1.0)
        plt.show()

    def similarities(self, x1, x2):
        """
        Calculates the cosine similarity matrix for every pair (i, j),
        where i is an embedding from x1 and j is another embedding from x2.

        :param x1: a tensors with shape [batch_size, hidden_size].
        :param x2: a tensors with shape [batch_size, hidden_size].
        :return: the cosine similarity matrix with shape [batch_size, batch_size].
        """
        x1 = x1 / torch.norm(x1, dim=1, keepdim=True)
        x2 = x2 / torch.norm(x2, p=2, dim=1, keepdim=True)
        return torch.matmul(x1, x2.t())

    def forward(self, predict, target):
        """
        Computes the N-Pairs Loss between the target and predictions.
        :param predict: the prediction of the model,
        Contains the batches x1 (image embeddings) and x2 (description embeddings).
        :param target: the identity matrix with shape  [batch_size, batch_size].
        :return: N-Pairs Loss value.
        """
        x1, x2 = predict
        predict = self.similarities(x1, x2)
        self.show(predict)
        # by construction the probability distribution must be concentrated on the diagonal of the similarities matrix.
        # so, Cross Entropy can be used to measure the loss.
        return self.ce(predict, target)

However, with this loss, the model ends up converging to a scenario where all dense vectors are equal to each other. Which can be seen by executing the following code snippet:

batch_size=7
hidden_size=768
def m_model(scenario=0):
    if scenario == 0: # all equal all
        p1 = torch.ones((batch_size, hidden_size))
        p2 = p1
    elif scenario == 1: # all different all
        p1 = torch.ones((batch_size, hidden_size))
        p2 = -1*p1
    else: # desired case
        p1 = torch.rand((batch_size, hidden_size))
        p2=p1

    return p1, p2

predict = m_model(scenario=0)
target = torch.arange(batch_size)
loss = NPairsLoss(1)

print("Loss:", loss(predict, target))
# Loss: tensor(1.9459), using scenario=0
# Loss: tensor(1.9459), using scenario=1
# Loss: tensor(1.7364), using scenario=2

Any suggestions on how to penalize these scenarios where the similarity matrix has all the same values?


What is the easiest way to identify a gender for a noun (in french)?

You could try to apply a French POS tagger, e.g. as suggested here: https://stackoverflow.com/questions/44468300/how-to-pos-tag-a-french-sentence

Note that a POS tagger usually works better from full sentences than isolated words.


Changes in the standard Heatmap plot - symmetric bar colors, show only diagonal values, and column names at x,y axis ticks

This is how I obtained the desired plot:

def generate_heatmap(X):

    """
    Pearson Correlation Heatmap Plot

    :return:
    """
    #from matplotlib import cm as CM
    from matplotlib.colors import LinearSegmentedColormap

    print("Start Pearson Correlation Heatmap Plot  .. ", datetime.now())

    # get column names
    cols = X.columns
    # define plot for heatmap
    fig, ax = plt.subplots(figsize=(16,16))

    # ------------------------------------------------------------
    # Correlation matrix for heatmap. the tranpose is because we want pxp matrix (rather a nxn)
    corr = np.corrcoef(X, rowvar=False)
    # show only upper matrix triangle - mask out the lower triangle of corr data
    corr = np.triu(corr, k=0)

    # ------------------------------------------------------------
    # Edit graphics of the plot
    plt.title('Pearson Correlation of ' + str(len(cols)) + ' miRNAs', y=1.05, size=15, fontsize=32)

    # bar colors shold be symetric around zero!
    colors = [(1, 0, 0), 'w', (1, 0, 0)]
    cm = LinearSegmentedColormap.from_list('heatmap', colors, N=20)

    # ------------------------------------------------------------
    # Heatmap based on corr matrix we provided
    c = plt.pcolor(corr, edgecolors='w', linewidths=2, cmap=cm, vmin=-1.0, vmax=1.0)

    # ------------------------------------------------------------
    # Editing additional graphics of the plot (if not too big)
    if len(cols) < 50:

        # set axis label names
        ax.set_xticks(np.arange(len(cols)))
        ax.set_xticklabels(labels = cols, rotation=45, fontsize=12, ha='center')

        ax.set_yticks(np.arange(len(cols)))
        ax.set_yticklabels(labels = cols, rotation=45, fontsize=12)

        # show corr values in every cell
        for (i, j), z in np.ndenumerate(corr):
            # in the symetric values, don't annotate the cell with the corr value
            if (i > j):
                continue
            else:
                # va and ha not working, we do +0.4 to overcome the centering of values
                ax.text(j+0.4, i+0.4, '{:0.2f}'.format(z), ha='center', va='bottom', fontsize=11)

    plt.colorbar(c)
    plt.show()

#generate_heatmap(miRNA_data[selected_mir_columns])


Classification model using RNN(action detection)

@CapJS,

Instead of going ahead with an RNN, which helps you model the dependencies and relationship between your content, I would suggest you to take a look at 1D convolution networks to achieve the classification of activity.

There is a nice post talking about something similar : https://machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/

With 1D convolution, you are by design gaining better processing performance and given that you have just 124 samples per activity, you do not need a deep layered network as the data size is still quite small.

Given the data size, I would also suggest you to go ahead with something as simple as logistic regression, random forest approach. Hope this helps.


In supervised learning, what does "Estimating $p(y \vert x)$" mean?

You are correct that
\begin{equation*}
p(y|\mathbf{x})=\dfrac{p(\mathbf{x},y)}{p(\mathbf{x})}.
\end{equation*}
Similarly, we can write the joint probability $p(\mathbf{x},y)$ as follows:
\begin{equation*}
p(\mathbf{x},y)=p(\mathbf{x}|y)\cdot p(y)
\end{equation*}
From the above two equations, we obtain
\begin{equation*}
p(y|\mathbf{x})=\dfrac{p(\mathbf{x}|y)\cdot p(y)}{p(\mathbf{x})}
\end{equation*}
In the context of supervised learning, the variable $y$ is used to denote the class labels, and  the vector $\mathbf{x}$ for measurement vector or feature vector. For the purpose of discussion, let us assume that the class label $y$ takes values in the set $\{1,2\}$, where $1$ denotes  male and $2$ denotes female. Similarly, $\mathbf{x}$ is a measurement vector on two variables say, $(x_{1},x_{2})$, where $x_{1}$ stands for height and $x_{2}$ stands for weight of individuals.

$p(y|\mathbf{x})$ denote the posterior density for $y$ given the observation $\mathbf{x}$. For example, $p(1|\mathbf{x})$ means that given the observation  $\mathbf{x}$, what is the probability that sample belongs to the class males. Similarly, we can interpret $p(2|\mathbf{x})$.
$p(\mathbf{x}|y)$ stands for the class conditional probability density. For example, $p(\mathbf{x}|1)$ denote the probability density for males and $p(\mathbf{x}|2)$ denote the probability  density for females respectively. In supervised learning, these class conditional densities are usually known in advance. Finally $p(y)$ denote the prior probability for the class label $y$. For example, $p(1)$ denotes the probability that an individual/example selected randomly from the population is a male.

Assuming that, the prior probabilities for the classes and class conditional densities are known, Bayes rule says, assign an observation $x_{0}$ whose class label is not known, to the class, say, male, if
\begin{equation*}
p(\mathbf{x_{0}}|1)p(1)>p(\mathbf{x_{0}}|2)p(2).
\end{equation*}
Note that, $p(\mathbf{x})$ is not used in the classifier. Classification rules usually require class conditional densities.


clustering plus linear model versus non linear (tree) model

With regards to the end of your question:

So the work team A is doing to cluster the instances, the tree model is is also doing per se - because segmentation is embedded in tree models.
Does this explanation make sense?

Yes, I believe this is a reasonable summary. I wouldn't say the segmentation is "embedded" in the models but a necessary step in how these models operate, since they attempt to find points in the variables where we can create "pure clusters" after data follows the tree down to a given split.

Is it correct to infer that the approach of group B is less demanding in terms of time? (i.e. the model finds the attributes to segment the data as opposed to selecting the attributes manually)

I would imagine that relying on the tree implementation to derive your rules would be faster and less error prone than manual testing, yes.


Too high performances on a classification problem

As noted by Peter, you should split your data into test and train
sets before fitting the TfidfVectorizer to the training set (not to
all the data).

Pandas read_json() is not reading your data from file the way you
want. You are expecting df["lista_asm"] to be a Series of list
objects (each containing strings). This Series is instead a  Series
of strings (formatted to appear like a list of strings). You must
parse the strings into a list, if that is what you want. But I
believe that the TfidfVectorizer().fit method wants strings, not a
list of strings. (This issue was apparent before your edit)

Similar to the first issue, you should avoid using the
RandomOverSampler() on the test data. If you want to use it, apply it
only to the training data. I suspect that it may be creating
duplicates or near-duplicate records, some of which are later split
into the test set. Try not oversampling at all and evaluate the
performance.


I would call this bad feature engineering, I'm afraid: as the designer of a ML system, one is supposed to analyze their data and find the best way to make the ML system perform as well as possible. In this case by adding a simple feature x % 2 for every instance the decision tree can perform perfectly.

[added] Even in the case of a more complex pattern, if there are such "clusters" of numerical values then there must be a logical explanation why this happens, i.e. something which depends on the task that an expert in this problem can analyze and understand. In most real cases this implies that there are some hidden/intermediate variables, and designing the system so that it represents these variables is key. In other words, the numeric variable is not directly semantically relevant for predicting the response variable, because the assumption when using numeric values is that their order matters (here the numeric value behaves more like a categorical variable).


ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python

Before fitting the model the training data write the below 2 lines:
X_train=np.asarray(X_train).astype(np.int)

y_train=np.asarray(y_train).astype(np.int)


What is difference between feed forward neural network and LSTM?

A feed-forward neural network looks like this:

input -> hidden layer 1 -> hidden layer 2 -> ... -> hidden layer k -> output.

Each layer may have a different number of neurons, but that's the architecture.

An LSTM (long-short term memory cell) is a special kind of node within a neural network. It can be put into a feedforward neural network, and it usually is. When that happens, the feedforward neural network is referred to as an LSTM (confusingly!).

So how does an LSTM work? Have a look at my answer here: Forget Layer in a Recurrent Neural Network (RNN) -

tl;dr: an LSTM cell has three gates, each of which is used to modulate its input in some way: an input gate, a forget gate, and an output gate. It has a "memory" and an output that get modified by the gates. That is, within a single LSTM cell:

(input & previous cell state) ->  (input & forget gates) -> (update cell state)

(input & previous cell state & updated cell state) -> (output gate)

You can stack these cells in many different configurations. So there is no single "LSTM network" - rather a set of many possible architectures that can be built from these basic nodes.

Hope that gets you started!


How to compare Random Forest with other models

We generally rely on sampling to validate the result of our Model.
 We make train/test to test a model on a separate unseen dataset.
 If we are doing a Hyperparameter tuning, we keep another set as a validation set to validate the result of new hyperparameters.
Random forest builds each new tree on a Bagged sample from the original sample(train data).
Bagging means sampling is done with replacement i.e. you pick one data point, put it back and then pick the next.
In this process, many duplicated data is sampled and many data points are not sampled.
~63% of the data points are selected Read here
The other "not selected" 37% data points is called Out of Bag samples.
Hence, the way Bagging, and RF is designed, we got another set of data to do our validation. It an opportunity to do a level of validation with these samples.
What it means,

You are getting 37% data points to validate your model
But OOB is not done with the fully grown Ensemble. It is done using all the trees in the random forest ensemble for which the particular data point was omitted during training Read here
It's not equivalent to K-Fold or Train/test on fully built RF but it gives a decent idea about the validation error about to come.


Immediate often occurs simply to say that there are perhaps many supersets, e.g. when we have a tree like structure. There is also for example, the converse, an immediate subset.

Example

Set A has a superset B, and that superset has its own superset C. In this case, A must also be contained within C, but the set B lies within that realm and so is the immedaite superset of A.

Perhaps it is best visualised with a Venn diagram, where B is the immediate superset of A and equally the immediate subset of C:

There is some discussion here on Cross-Validated too, where the selected answer shows some further usage of the term (albeit in a slightly different context). Have a look at this source for some usage of the term, which leads you to my understanding.


How to do time-series prediction in the browser

I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.

The first thing comes to mind is sequence models, I think this will be useful for you, see Real-time Performance RNN in the Browser


In general identifying similarities is done with clustering, but in this case what you're looking for is a potential pattern in the data leading to a missing value in a specific column, right? So I would try to train a decision tree using all the columns except the target one as features, and using a binary class indicating whether the target column has a missing value or not. Visualizing the decision tree obtained after training  should show the pattern if there is one.


Thinking of a sum as an average, this model sounds reminiscent of continuous bag of words (CBOW) word embeddings (i.e. word2vec). In that context, the words in a sentence are used to predict a missing word using the average of embedded vectors (see this question). This method works pretty well for words, so it might make sense that you could extend it to other types of embeddings like user/item compatibility.

The average makes some intuitive sense when you think of comparisons. To place a user/item in embedding space, you can average the feature embeddings. If a user/item has different but similar features their average should put the user/item in a similar region of embedding space. A sum should work the same if the feature number is fixed.

An issue with a sum would arise if feature number isn't fixed. Users/items could appear different because their embedding vector is a different length, even if it has the same direction. Normalizing by feature number would fix this.

Another possible issue with using a sum rather than an average is that the user/item embedded vectors aren't comparable to the feature embedded vectors. This doesn't seem to matter for the author, because they don't try to compare the user/item vectors to the feature vectors directly.

TL;DR Yes, a sum might make sense to represent a set of features, but an average might be better.


How to represent a document in test data with the Document-Term Matrix created from the training set?

The easiest way is to treat all out-of-vocabulary terms as a specific term in your matrix (i.e. "OOV").

So for instance, if my training data contains 3 words: "I", "like", "cake", my document-term matrix would contain 4 items, "I", "like", "cake", and "OOV".


why do we have to calculate the entropy of parent node in Information Gain?

It's essential; you're computing gain from the parent to the same data split in the children! Not comparing children. A good split takes a high-entropy data set (lots of several different labels) and turns it into lower-entropy data sets (some labels on one child mostly, other labels on the other).


Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other.

Yes, but this is the worst case scenario of the curse of dimensionality:

This happens only if the data is extremely sparse. In real cases it's not really "all data", because some data points are less sparse than others, and it's not really "the same distance", because there can be small but meaningful differences in the distance between points.
There are known mitigation measures which can prevent this worst case scenario to happen, in particular various feature selection/extraction methods. So in real cases it's simply a mistake to work with some very high dimensionality data left "untreated".

Why not to encode nominal categorical variables of high cardinality with randomly distributed points in some high-dimensional space but such that the number of dimensions is much less than we would get with the one-hot encoding?

This would correspond to a random feature extraction method: instead of trying to group similar features together when reducing the dimensions, features are just randomly projected into a low-dimension space. The disadvantage is clear: instead of simplifying the data while preserving the patterns, the patterns in the data are just ignored. It's likely that the data would become pretty much useless after that.
The goal is not just to reduce dimensionality for the sake of it, it's to reduce dimensionality in a way which leaves the most important characteristics of the data as intact as possible, or even emphasize them by removing the noise. This method would be equivalent to randomly selecting features instead of selecting the most informative ones.
For example, in the case of text data there is often high dimensionality due to the high number of distinct words. A basic but very effective method consists in discarding words which appear rarely: this reduces the dimensions a lot (due to the Zipf distribution of words in a text), prevents noise due to words which happen by chance, and preserves most of the patterns which happen with moderately frequent words.


Short answer is: as much as possible.

The more data you have, the more likely you can make correct inference. In the worse case, you could simply discard anything that you don't like. It's like money, the more the better - you always have an option to spend or use less.

Statistics could be doggy without sufficient sample size, your statistical power would be affected and any model you have could be biased.

Having said that, you don't want everything from your user. Use your common sense, and ask yourself what exactly you want to do.

Based only on the data in your question, I'd say: (I could be wrong because I don't understand where you got the data)

How many total users you have that didn't drop out? If your total user base is like 10K, 28 users dropping out is not a problem. But if you only have 30 users...
Your sample size is small. Is that one-day installation? In any case, 28 users for a mobile app is a tiny data set. You may want to focus more on marketing.

EDIT:

How many respondent you need is hard to tell. Apple probably needs better data quality than you. If you'd like to do it statistically, you may calculate the minimum sample size and it's related to statistical power.


Will this MAPE implementation work for multidimensional output?

You can either directly use the mean absolute percentage error (MAPE) function in Keras API for TensorFlow or use that function to test your custom function.

tf.keras.losses.MAPE code is avaible in the TensorFlow GitHub repo:

def mean_absolute_percentage_error(y_true, y_pred):
  y_pred = ops.convert_to_tensor(y_pred)
  y_true = math_ops.cast(y_true, y_pred.dtype)
  diff = math_ops.abs(
      (y_true - y_pred) / K.maximum(math_ops.abs(y_true), K.epsilon()))
  return 100. * K.mean(diff, axis=-1)


MemoryError when concatenating a large data-frame

You should use sparse matrices

When you are categorizing a column, you are creating many columns that mostly hold zeroes, and since dense data structures keep track of all the zeros, the memory usage explodes. Sparse matrices on the other hand only keep track of non-zero values and their indexes. There are many version of sparse matrices optimized for different use cases.


Data cleaning: Relationships between columns

I see several issues in your data.

First of all, if there is a one-to-one relationship between X1 and X2, you should remove one of the two columns, because they are redundant. Redundant data may have a negative impact on your classification performance.

Secondly, the fields X3 and X4 also seem to be totally redundant, since the value of the class label Y only depends on X1/X2. So unless the columns X3 and X4 may be interesting on their own, I don't see the point of including them into the data.

Having dealt with these issues, and in order to obtain Y from X1/X2, there are two possibilities. If file1 contains the value of Y for any possible value of X1 in your domain, you don't need any machine learning technique. You have a perfect mapping. Otherwise, you will need to apply machine learning to find a function that "fills the gaps". Depending on the nature of the Y variable, you will need to use a regression (if Y is a real number) or classification (if Y is a discrete variable).


How to mantain the nested structure of a tf.dataset after applying map?

I have managed to solve this problem by "flattening" (eliminating the parenthesis) the return of the random_crop_flip, and applying another map on top of them, where I specified the shapes and returned my desired structure (x ,y), z:
def _set_shapes(img_bl, img_fu, mask):
    img_bl.set_shape([128, 128, 128, 1])
    img_fu.set_shape([128, 128, 128, 1])
    mask.set_shape([128, 128, 128, 1])

    return (img_bl, img_fu), mask

Then my code looks like this:
x, y = get_filenames(train_data_path, img_type='FLAIR')
z = get_filenames(train_data_path, img_type='mask')

path_dataset = tf.data.Dataset.from_tensor_slices((x, y))
mask_dataset = tf.data.Dataset.from_tensor_slices(z)
dataset = tf.data.Dataset.zip((path_dataset, mask_dataset)).shuffle(50).repeat(10)

ds = dataset. \
    map(lambda xx, zz: ((tf.py_function(load, [xx], [tf.float32, tf.float32])),
                        tf.py_function(load_mask, [zz], [tf.int32])),
        num_parallel_calls=tf.data.AUTOTUNE)

ds = ds.map(lambda xx, zz: (tf.py_function(random_crop_flip, [xx, zz],
                                           [tf.float32, tf.float32, tf.int32])),
            num_parallel_calls=tf.data.AUTOTUNE)
ds = ds.map(_set_shapes)
ds = ds.batch(2)
ds = ds.prefetch(tf.data.AUTOTUNE)


Is there an alternative to Google Maps to find GPS coordinates of a place in Germany?

There are a number of APIs for OpenStreetmap, e.g. via html request:

https://nominatim.openstreetmap.org/search?q=pleikartsförsterstrasse,%20heidelberg&format=json&polygon=1&addressdetails=1

A json is returned. You can easily access the type of information you need.


how can autoencoder reduce dimensionality?

Autoencoders are trained using both encoder and decoder section, but after training then only the encoder is used, and the decoder is trashed.

So, if you want to obtain the dimensionality reduction you have to set the layer between encoder and decoder of a dimension lower than the input's one. Then trash the decoder, and use that middle layer as output layer.


Unequal length observation sequences when training Hidden Markov model

No worries, length of observation would help in training of model. longer the length, training for Forward-backward Algo/baum Welch algo would be better. But length of observation is nothing to do with forecasting. read my blog below-

http://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html

transition probabilities would give next hidden state. and from next hidden state you can get next observation.So number of observation would not matter as long as u have trained model( emission and transition probabilities). Also if are intrested to know in detail that how next state would be calculated, follow below article.

http://machinelearningstories.blogspot.com/2017_03_01_archive.html


how can LSTM output ever be negative with 0 initialization?

Cannot find the equations in your reference, so I take them from wikipedia. At first time step the following equation holds: $$h_{i} = \sigma(A)\tanh((\sigma(B)(\tanh(C))$$
where $A, B$ and $C$ are values calculated from linear functions. So it can be negative if $C$ is so.


I have already posted it here:

https://stackoverflow.com/questions/43366561/use-sklearns-gridsearchcv-with-a-pipeline-preprocessing-just-once/55401454#55401454

Suppose you have this pipeline:

classifier = Pipeline([
    ('vectorizer', CountVectorizer(max_features=100000, ngram_range=(1, 3))),
    ('clf', RandomForestClassifier(n_estimators=10, random_state=SEED, n_jobs=-1))])

Then, when specifying parameters you need to include this 'clf_' name that you used for your estimator. So the parameters grid is going to be:

params={'clf__max_features':[0.3, 0.5, 0.7],
        'clf__min_samples_leaf':[1, 2, 3],
        'clf__max_depth':[None]
        }


You can use Deep convolution GAN network of Deep learning. It would be a generative model.

you can get detailed help from :

https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b


How to get same accuracy with identical models in Keras and Tensorflow?

I created a rule to achieve reproducibility:

Works for python 3.6, not 3.7
First install Keras 2.2.4
After install tensorflow 1.9

And finally in the code:

import numpy as np
import random as rn
import tensorflow as tf
import keras
from keras import backend as K

#-----------------------------Keras reproducible------------------#
SEED = 1234

tf.set_random_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
np.random.seed(SEED)
rn.seed(SEED)

session_conf = tf.ConfigProto(
    intra_op_parallelism_threads=1,
    inter_op_parallelism_threads=1
)
sess = tf.Session(
    graph=tf.get_default_graph(),
    config=session_conf
)
K.set_session(sess)
#-----------------------------------------------------------------#


Isn't graph embedding a step back from non-euclidean space?

Building embeddings is the first step in graph processing.
We mainly use to apply mathematics on something like plane or hyperplane. Almost all mathematical methods can work with linear space, especially linear algebra which we use in neural networks and computer data processing.
So, in fact, embedding is just one of the first steps we need to do in order to apply some math on graph data. We can embed node features to a lower dimension, we can also embed an entire graph to some space we know how to work with. Then apply functions, transformations, whatever we want.
When you are building an embedding, your task is to represent a graph in linear space so that it will keep all the characteristics it has in non-linear "graph" space.
In this article you can read more about different approaches to graph embedding. I will just cite one phrase from there.

Machine learning algorithms are tuned for continuous data, hence why embedding is always to a continuous vector space.


How to filter out numbers from text documents

Yes, orange has regex abilities as seen here.  If you're unfamiliar with regex here's a link regarding regex with numbers.  The idea is pretty straight forward.  Use regular expression to find all numbers, then replace the numbers with something like a blanks string a placeholder word like "NUMBER".


How to add date column in python pandas dataframe

You can use the built-in date_range function from pandas library to generate dates and then add them to your dataframe. You can use it in the following way:
In [9]: import pandas as pd

In [10]: df = pd.DataFrame({'column1':[34,54,32,23,26]})

In [11]: df
Out[11]:
   column1
0       34
1       54
2       32
3       23
4       26

In [12]: df['date'] = pd.date_range(start='1/1/1979', periods=len(df), freq='D')

In [13]: df
Out[13]:
   column1       date
0       34 1979-01-01
1       54 1979-01-02
2       32 1979-01-03
3       23 1979-01-04
4       26 1979-01-05

Here I have incremented the dates day-wise per row. If you want to generate dates in some other way, check out the official documentation here.


matplotlib subplots_adjust - meaning of parameters

These values represent the distance of your subplot from the boundary of the figure. Its value is between 0 & 1. For example, if you set top=1 the upper boundary of your subplot will coincide with the upper boundary of your figure and if you set top=0 the upper boundary of your subplot will coincide with the lower boundary of your figure and the plot will be entirely squashed down.

It is the percent of figure height and figure width.

A better way to see how this works is to plot a graph and the use the option Configure Subplots, in the window that pops up, to adjust these values. Try this and you will able to appreciate the use of these values in a much better way.


How to identify clusters after multiple runs?

Use the famous Hungarian algorithm.

It computers the best match permutation.

You can find more details on Wikipedia:

https://en.wikipedia.org/wiki/Hungarian_algorithm


when my RF log_loss socre is different then submit score?

In a Kaggle-like competition, the data your submission is evaluated on is not part of what is given to you. It may always produce a different result than what you estimate in your training process.

But, if it's very different, then you have likely over-fit and your model is high variance, so, your test error is probably higher than the training error here. In particular, you don't seem to be using early stopping. Your submission error is also then going to be higher than training error.

You mean your test error is lower than submission error, right? they may be quite different than your test error due to the high model variance.

The submission error on average might be higher than your test error, even when done properly, because one tends to 'overfit' the hyperparameter tuning in any validation process. (But here I don't see hyperparam tuning even?)

A more correct process would be, at least, to try to find the best tuning params with some cross-validation process, then refit the model on all the data you have using those params.


'y_train' and 'y_val' should be whatever it is you are trying to predict. They can be values, classes, or they can be a sequence. The form of what you are trying to predict will influence how you structure a RNN in Keras: Many to one and many to many LSTM examples in Keras.
'data_dim' is the number of features in the dataset. If you are predicting the weather, the features may be 'pressure', 'temperature, 'precipitation', and so 'data_dim' = 3.
'num_classes' is the number of classes that the outcome variable can assume. If the outcome variable is 'type of animal', and the classes the outcome variable can assume are 'dog', 'cat', 'bird', then 'num_classes' = 3.


You don't want to follow those rules.
You have your matrix of observations and you multiply by a matrix of weights, in which, initially al the weights are assigned randomly.
A matrix of weights is composed in this way: each column represents a neuron and the rows the weight to which you multiply your features.
If you have k layers you'll have k matrices of weights.
At each iteration, what the DL framework allows you to do (You don't want to it manually), compares the real target to your predicted and computes a cost function(How far is the predicted from real?). Derivative is then computed respect to the weights, it is demonstrated that the direction of the steepest descent is obtained by updating the weights by a value of $W_i = W_i + \Delta W_i$ where: $\Delta W_i = n(d_i - y_i)$. It goes in this fashion by a number of iterations you defined.
I hope I gave you a hint to understand how it works.


Those symbols you are talking about are hex values. They could actually be useful to keep in the dataset, as the tcpdump tool outputs them if the right options are set.

Despite that, you can replace occurrences of them using pandas and a simple regular expression. A regular expression that matches hex values is: 0[xX][0-9a-fA-F]+ - see a live demo of that here.

I downloaded one of the CSV files and did it like this:

import pandas as pd

filename = "./UNSW-NB15_1.csv"
df = pd.read_csv(filename, nrows=5000)     # read first 5000 lines
df.to_html("with_hex.html")                # open in a browser to inspect (e.g. line 211)

Now use the DataFrame replace and set the regex flag, and insert whatever you want to replace the matches with (here "NaN").

clean = df.replace("0[xX][0-9a-fA-F]+", value="NaN", regex=True)

I used this to have a look at the cleaned versions of the data quickly in a browser (notice line 211):

clean.to_html("clean.html")

You can use the same trick to replace almost anything, so your fields with '-' can be replaced with:

# "inplace" means it overwrites the values, no new dataframe is returned
clean.replace("-", "NaN", regex=False, inplace=True)