Pattern Recognition, Bishop - MLE and Least squares section 3.1

We have

$$\frac12 [(1+2)+(3+5)]=\frac12(1+3)+\frac12(2+5)$$

We can swap the order of summation of the LHS since addition is commutative and associative.

After we decided how to pair them up, we just distribute $\frac12$ into them using distributive law.

\begin{align}
\frac1N \sum_{n=1}^N \sum_{j=1}^{M-1}w_j \phi_j (x_n) &=\frac1N \sum_{j=1}^{M-1}\sum_{n=1}^N w_j \phi_j (x_n) \\
&= \sum_{j=1}^{M-1}\frac1N\sum_{n=1}^N w_j \phi_j (x_n) \\
\end{align}

where the first equality is due to we are dealing with finite sum, and hence we can swap the order of additions, that is summing up the row first or column first doesn't matter.

After which, we use distributive law to bring $\frac1N$ inside the first summation.


Develop classification model, given the k nearest neighbors

If I understand you correctly, based on the data and the given nearest neighbors, you need to decide what distance calculation was used to determine the nearest neighbors.  It seems unlikely that you are being asked to create an altogether new distance measure, rather explore the different accepted distance measures.

If this is correct, it sounds like all you need to do is run the nearest neighbors recursively with different distance calculations and compare the results to the provided nearest neighbors to determine which measure created the 5 nearest neighbors you were given.


When k=3 it is a simple case, the 3 nearest examples share the same distance to the new sample. Two of them are from class 1 and a single example is from class 2, so the new sample will be classified as class 1.

When k=2 it is more tricky. Assuming that the 2 features (x,y) share similar importance to the classification (otherwise distances should be calculated with some predetermined weights), you can:

Choose randomly 2 out of the 3 candidates (you are basically acknowledging that for this case, your model doesn't have enough information for classification based on 2-NN).
take all candidates into account (i.e. for cases like this you go to 3-NN).

When k=1, same options as with the k=2 case (you will still need to go up to 3-NN).

In this specific case, because your 3 nearest neighbors are all of the same distance, I would choose to use K-NN. However for a similar case with 4 samples at the same distance, 2 of each class, I would choose randomly (not going to 5-NN).


TensorFlow: number of channels of conv1d filter

In TensorFlow there are different convolution layers. Conv1d, Conv2d and Conv3d. the first one is used for one dimensional signals like sounds, the second one is used for images, gray-scale or RGB images and both cases are considered to be two dimensional signals. The last one is used for three dimensional signals like video frames, images as two dimensional signals vary during time. In your case Conv1d is used as one dimensional signal and you can specify the number of filters in the arguments of the method. You can take a look at here and here.


Can a trained recognition model be used to generate a sample?

Generative models can. Among others, you can use GAN, auto-encoders or VAE, ... As far as I know, discriminative approaches can not generate a sample.
Hope it helps.


How to select features for Text classification problem

Since I am not sure where you are at the moment in your solution, let me give you a comprehensive, yet brief, view of what you should be looking to do.

Preparing the training data: You would be required to collect data with the correct classifications. Once you have this, you should incorporate text mining algorithms (bag of words for instance) and end up with a document-term matrix (or term-document matrix, whichever is appropriate). Think of this as a list of all the different words (or word collections) possible in a complaint. These would in turn serve as your features
Training the data: This is where the algorithms you have stated above (RandomForest for instance) come into picture. I would also suggest you to try other boosting algorithms here subject to data availability
Testing on a holdout sample: Test the model accuracy on a holdout sample and finalize on the models you would like to choose. Note that you might also have to tune the text mining algorithm in case you do not get appropriate features for your classification algorithm
Classifying new complaints: Apply the text mining algorithm to the newer data to generate the features as before and run it by the classifier

Hope this gives you a good starting point for building your text classification solution.


Data Science Conferences w/ Poster Sessions?

KDNuggets has a really nice list of conferences on their page:

http://www.kdnuggets.com/meetings/index.html


At first the way you explained it, it sounded to me it is a multilabel classification (like e.g. CelebA) and approaches I have used for there was coming to my mind. Also I thought it is hierarchical classification because you have subcategories in your dependent variable. But at the end you mention that you want to

  Predicting the accidents using the violation levels.

Aren't then your dependent variables (violation level , and accident) already correlated/related to each other? It seems to me that you do not have two independent dependent variable. I would try to reduce the problem to a simple multiclass classification, in your case 6 as follows (assuming you have these explicit labels for each data point):

Severe-Yes
Severe-No
Medium-Yes
Medium-No
Light-Yes
Light-No

Then you have 6 classes/labels to build a model e.g. Gradient Boosting Tree or Neural Network.


Although you have not made it clear in your code snippet but the activation output could be inside or outside a function:

Outside Function : Just run

a = sess.run(act_layer_1)
print(a)

Inside Function: You have to return the value of this activation bunched with other values you need to print and then again use sess.run() to get all the values and print the index which contains your value.

A longer but better in debugging would be to use tf.InteractiveSession(). More details can be found here:

How to print the value of a Tensor object in TensorFlow?


Using python and machine learning to extract information from an invoice? Inital dataset?

I'm more of a beginner as well, but wanted to possibly help guide you towards next steps based on some of my experiences.

First, what you've illustrated here sounds a little along the lines of how a traditional EDI system does in Supply Chain & Procurement processing, where it will receive a document and enter the information into transaction-based operational systems to create records as data. I'm not entirely sure how those work, and if they can only extract standardized formats from documents but that's something that you could possibly look into with a few searches.

Now as far as creating a CSV data set, that is a great idea for testing the accuracy of your algorithm on a set of invoices to train your model. Training a model is pretty self-explanatory, but essentially you'd be using a supervised machine learning strategy, where the system actively uses a training data set where the correct answers are known. By comparing the models results to the data set, you could then know if the algorithm is appropriately retrieving the information. I'd recommend that you include the information the application should be getting from each invoice, and compare that to what the machine did retrieve. That way you effectively known the error of the model. (x,y) coordinates would work well for preliminary attempts to get a better feel for location-based assessment of the invoices, so that's a pretty good idea in my opinion as well.

Regarding algorithm types, this one will almost always be "it depends". Many times, I'll use a couple and compare the error that returns for each model to find the one that works the best for what I'm predicting. That said, Clustering can be useful for continuous, numerical data and "grouping" items based on the location of the coordinates. I think clustering could be useful if you're breaking down the coordinates of the information on an image, but wouldn't be of much use to gather and extract the text from the invoice. And to not go too deep, K-Means would probably not be great at all because the locations for the cluster centroids is entirely randomized and you must also declare a number of n clusters, which would be impracticable for the purpose at hand.

I'd recommend looking into the different types of Neural Networks(NN's). These are used extensively to take images of some kind, and break them down to best estimate the contents of an unknown input. It's the type of algorithm Google uses to do image searches and things along those lines. It's also the type of software that tells an autonomous vehicle that the sign ahead is a stop sign. There are different types of NN's out there, and it can get pretty overwhelming pretty quickly, so I recommend looking into basic videos to explain it. If you do decide to go this route, Google Cloud has a pretty solid platform with there google.vision and other machine-learning-in-a-box libraries that are really easy to import and play with.

as far as inputs, define the information that you want extracted, and record that data for use of testing and training as I mentioned above. Then play around with a few methods in sciket-learn and TensorFlow!

Unfortunately, I can't give you a straight-forward answer on the many trade-offs with different algorithms or inputs, but I hope that this gave you a few ideas about next steps or provided you with a better understanding of how you can begin the project. Good luck!


Handling a feature containing multiple values

You need to rethink your approach. Rather than "what can I code to make things work", you need to ask "what is the right thing to do, and how can I implement this".

Clustering is hard (easy to run, hard to get good results).

Clustering non-continuous data is even harder.

The reason why people create a lot of false results here is because you have infinitely many way to weight different values and attributes, and essentially you can get almost any result you want just by playing around with parameters. Don't let yourself be lured by the common hack of "one hot encoding" everything; it just shows that people don't (want to) know what they are doing.

To get a reliable result, you need to be very clear on your assumptions, such as "I assume all actors are equally representative, and the overlap as measured by Jaccard is a good indicator of similarity" (I would disagree here, actors should be weighted).
Then you need to do the same thing with genres. Here it is even more questionable hos to compute similarity of genres.
And after that, you need to combine all these different similarities into one. That is probably the hardest step, and will involve even more weighting parameters.

All in all, I'd say: whatever you do, the clustering is not statistically sound. It's too many parameters chosen without even a good reason (yet a proof) that this way is better than another way of choosing them.
In particular, you will never know if there is an even better clustering with different parameters.


How to specify version for dependencies so that each one is compatible and stays within a size limit?

Till now, what I have learned is that it is a trial and error process.
And the best solution is to look for projects that have already done this using a working combination of libraries.
I used this requirements.txt file, which works fine:
gunicorn
flask
numpy
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
torchvision==0.2.2
fastai==1.0.52
jupyter

The web app works perfectly and the permitted size is not exceeded.
Although Heroku's size limit is 500 MB, they have a soft limit of 300 MB. These libraries, their dependencies, and the model file together exceed that resulting in a slightly longer response time.


I tried different classifiers using a combination of SMOTE and class_weight, the results are almost the same as using only the SMOTE approach, and this new config made almost no difference (which could be expected, following the logic behind the class_weigh approach).

PS: I have a pretty large dataset with multiple classes. This might result in different performance in different contexts.


Finding the duplicate values between all columns and sort in new column with Pandas?

Given your input data is saved in a variable df, I count the values which occur in all 4 unique columns as follows:
import pandas as pd
import numpy as np

output = (
    df
    .melt()
    .drop_duplicates()
    .groupby("value")
    .agg(count=("value", "count"))
    .reset_index()
)
output["SIM"] = np.where(output["count"] == 4, "SIM", "NON-SIM")
output = output.pivot(columns="similarity", values="value")

print(output)

similarity  NON-SIM SIM
0           NaN     a
1           b       NaN
2           c       NaN
3           NaN     d
4           dc      NaN
5           dx      NaN
6           f       NaN
7           g       NaN
8           NaN     s
9           t       NaN
10          w       NaN
11          x       NaN
12          y       NaN


Train-Test split for a recommender system

My two cents,
Evaluating results of a recommendation engine where test-set is unseen users only, will allow you doing exactly that.
Evaluating results on unseen users only.
If this is indeed the motivation(product/business wise) behind your recommendation engine I would suggest trying to tackle this problem directly.
If evaluating performance on 'unseen' users is an additional metric and not the only one I would keep with current scope of train/test splitting(which should create some unseen users in the test set).

Regarding methods on how to best recommend to unseen users, your intuition(K-nn) is one common solution.
 See this for example


Dense layers inherently work with 1d-data and lose any positional "importance" of your input data (e.g. the location of a person in an image, how digits come together to form a number). Theoretically, you'll achieve the same training output if you decided to randomly shuffle the positions of your input features. If you want to stick with a Dense model as you've provided, I believe that normalizing on a 2d array would not benefit your training.
But it sounds like the position of each feature in the 6x25 matrix is important for your predictions. In that case, I would suggest switching from a Dense model to a convolutional model, which incorporates the position of each feature during training. Here's an example. In this case, here's a useful list of normalization techniques for 2D or 3D data inputs.
Please let me know if I can clarify anything. Good luck!


Oversampling/Undersampling only train set only or both train and validation set

Oversample the train data and NOT the validation data since if train data is unbalanced, your test data will most likely show the same trait and be unbalanced.
If you don't know if test data will be balanced or not, oversample only train data.
But there is one recommendation which I will personally give you since I suffered from same problem not so long ago.
Oversample the data (train)
Test accuracy on validation data (which is not oversampled)
Test this accuracy with accuracy obtained from not doing oversampling (or undersampling whichever you performed)
If the results vary only marginally, train the model on non oversampled data. The reason being since the oversampling technique will introduce data points near current data points belonging to same class which may not accurately depict your test data.
In such a case there are two things you can do

Tweak the model parameters if available

For example : if using random forest it has a parameter in model called "class_weight" which if kept at "balanced" will give equal weightage to every output variable which would be inversely proportional to class frequencies in input
Similar parameter is available on other models as well (logistic regression etc.)

When splitting data using train_test_split set parameter stratify

Example : train_test_split(train_data, df['target_column'], stratify = df['target_column'])
Stratify will make sure your train and validation data are split based on output label frequencies based on train data. Like if the data was like 90 to class 'A' and 10 to class 'B'. After split both train and validation will have 90:10 ratio of classes


Regressor to estimate probability from samples

Assume that X are features and y - labels

#Put algorithm as you wish
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(X_train, y_train )

clf.predict_proba(X_test) # there you will get probabilities of the class


Machine learning algorithm to classify blog posts

Question 1: Category prediction

To predict the category of a new blog post, you could do the following:

Build a MLP (multilayer Perceptron, a very simple neural network). Each category gets an output node, each tag is a binary input node. However, this will only work if the number of tags is very little. As soon as you add new tags, you will have to retrain the network.
Build a MLP with "important words" as features.
If you have internal links, you might want to have a look at "On Node Classification in Dynamic Content-based Networks". In case you're German, you might also like Über die Klassifizierung von Knoten in dynamischen Netzwerken mit Inhalt
You could take all words you currently have, see those as a vector space. Fix that vocabulary (and probably remove some meaningless words like "with", "a", "an" - this is commonly called a "stopword"). For each text, you count the different words you have in your vocabulary. A new blog post is point in this space. Use $k$ nearest neighbor for classification.
Use combinations of different predictiors by letting each predictor give a vote for a classification.

See also

Yiming Yang, Jan O. Pedersen: A Comparative Study on Feature Selection in Text Categorization, 1997.
Scikit-learn: Working With Text Data

Question 2: Tagging texts

This can be treated the same way like question 1.

Question 3: Finding locations

Download a database of countries / cities (e.g. maxmind) and just search for a match.


How to preprocess tensorflow imdb_review dataset

For TypeErrors it is always good to check the exact type of the variable causing issues, especially when troubleshooting TensorFlow and Keras:
print(type(df['text']))

fit_on_texts expects a list of string or similar, but you are providing a dictionary, so you'll want to convert accordingly. Example:
from typing import List, Dict

foo_dict: Dict[str, str] = {"one": "first sentence", "two": "second sentence"}
foo_list: List[str] = list(foo_dict.values())

print(foo_dict)
print(type(foo_dict))

print(foo_list)
print(type(foo_list))

Output:
{'one': 'first sentence', 'two': 'second sentence'}
<class 'dict'>
['first sentence', 'second sentence']
<class 'list'>

API docs will help here:
https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#fit_on_texts
I'm using type annotations in the example above which I find helps troubleshooting and avoiding bugs.


Convnet training error does not decrease

In your convnet code, you compute the cross entropy manually:

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-5).minimize(cross_entropy)

If you do that, you may run into numerical stability issues. Instead, you should use

tf.nn.softmax_cross_entropy_with_logits()

If using tensorflow version 1.8 or later, use

tf.nn.softmax_cross_entropy_with_logits_v2()

and weep at the state of tensorflow's API.

See this issue for more information: https://github.com/tensorflow/tensorflow/issues/2462


What's the best way to train a NER model?

I would start by training some very strong Named Entity classifier on available datasets for NER. One is the Annotated Corpus for Named Entity Recognition available on Kaggle.

Additionally, you can find a good list of datasets here. I know they have nothing to do with cybersecurity, but I think it's important to incorporate very different sources in a big, final dataset, in order to make a model that is good at generalizing on texts it has never seen before.

Another source of data for NER tasks is the annotated corpora available from nltk library, such as the free part of the Penn Treebank dataset, and Brown corpus.

Please beware that different datasets might use different categories for classification (i.e. the set of Named Entities can be different from dataset to dataset). Make sure you make all your data compatible to your classifier before training

After that, I suggest you to go with seq2seq models. Every state-of-the-art RNN is some form of seq2seq. Once you trained a classifier, you could try to annotate few articles manually, and check the performance of your model on those. It's time consuming, but I personally like these "qualitative" checks, I think they can tell you a lot.


Right now Tensorflow 2.1 doesn't support CUDA 11.0. Downgrade CUDA to version 10.1 and try again.
Reference: https://www.tensorflow.org/install/gpu


The way you type formulas is a bit confusing, but here's a go at interpeting it.

Euclidean distance is defined by:

Manhattan distance is defined by:

Handling missing terms is an independent issue. The way it is handled in the example is by taking the average of the present features (dividing by their number) and multiplying by the total number of features as a way to bring the data to a scale comparable to datapoints without missing features.

Now let's have a closer look at the example. The total number of features is 4. To calculate a distance between datapoints you can only use features that are present in both:

The first (Euclidean) formula finds 3 features present in both x1 and x2. Thus your coefficient is 4/3.
The second (Manhattan) formula finds two features present in both x3 and x4. Thus the coefficient is 4/2.


How to handle performance degradation due to domain transfers in data

There are multiple ways to detect covariate shift, for example Kullback–Leibler divergence but my favorite is adverserial validation. For images I would also look at the shap explainer, if its activating on different random places in the test set, you can suspect something fishy in concept.

Remedy concept shift with NN- try batch normalization if you havent. Why? Weights of the deeper layer, learned by the neural net, will be relatively less dependent on the weights learned on the shallower layer, hence avoiding shift.


This depends at least a little on your model type and encoding strategy.  It sounds like you'll be one-hot encoding, with possible level/dummy for "missing" and/or for "new".  Given you say you won't impute, I guess you'll be using tree-based models?

If you have a dedicated level for unseen categories, then the trained model will have no idea what to do with it.  Results will vary among errors in model build, zero input of the dummy feature, and random contributions from the dummy feature.

Assigning new categories to the missing category is a little better, but will depend on how you treat missing values in your model training (but you've said you don't want to impute).  At least it gives a consistent result.  This may even be the best choice, depending on your data.

In your elaboration you mention bank names, which suggests an alternative (maybe several).
Perhaps these unseen banks will have some commonality with banks in the training set (they're small and local maybe?); if so, you could try to coarsen the categories in the train set so that these new ones will fall nicely into those categories (and then encode all future unseen banks into that new category; make a note to reexamine this process when you retrain!).
If your training data has lots of small levels, it can also make sense to lump the rare categories together (to help avoid overfitting); in that case you can just drop the new levels into that one.  Again depending on your data this might work well (small and local would also be rare, so this accidentally recovers the last idea in this case).


Convolutional neural network overfitting. Dropout not helping

Ok, so after a lot of experimentation I have managed to get some results/insights.
In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance of the network, as a negative side, the training process is muuuuuch slower.
Second point, data is important, nothing new here but as I learned while fighting this problem, more data always seems to help a bit.
Third point, dropout is useful in large networks with lots of data and lots of iterations, in my network I applied dropout on the final fully connected layers only, convolution layers did not get dropout applied.
Fourth point (and this is something I am learning over and over): neural networds take A LOT to train, even on good GPUs (I trained this network on  floydhub, which uses quite expensive NVIDIA cards), so PATIENCE is key.
Final conclusion: Batch sizes are more important that one might think, apparently it is easier to hit a local minimum when batches are larger.
The code I wrote is available as a python notebook, I think it is decently documented.


Ummm... SVM is a linear model, so if you average the predictions of several models with different regularization, what you get is the same result of another model which uses a different regularization.
Regarding the intent behind your question: there's more methods than SVM and SVDD. You can check the PyOD package for ensemble models for outlier detection.


Is it a Problem if Training Data and Evaluation Data are Very Similar?

In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's called overfitting.

In your case, could be happened more, if some rows are the same in train and data set, then we could say that it's data leakage.


In DQN, why not use target network to predict current state Q values?

Bellman equation for deterministic environments is given as follows
$$
V(s) = max_aR(s, a) + \gamma V(s')
$$
Where $V$ is a value function, $R$ is a reward function, $s$ is current state, $s'$ is next state, $a$ is an action. In DQN when optimizing $V$ it is assumed that $V(s')$ has on average lower error than $V(s)$. This is intuitively because $s'$ is closer to the end state for which $V$ is known and so there are fewer steps in which error can accumulate. This way we can progressively lower errors of $V$ for all states starting from the end states up to the starting state. I don't know how exactly you use current state to update weights but assuming it looks like this
$$
\delta V(s) := \alpha(max_aR(s, a) + \gamma V(s) - V(s))
$$
instead of
$$
\delta V(s) := \alpha(max_aR(s, a) + \gamma V(s') - V(s))
$$
i.e. you don't use $V(s')$ in the weights update what happens is the same error propagates over and over again because information about rewards in the later states is not propagated to earlier states.
As a concrete example imagine an environment with 3 states $s_1$, $s_2$ and $s_3$ where $s_1$ is starting state $s_3$ is the end state, the action space is only one move: go from state $s_i$ to state $s_{i+1}$ and reward function is 1 for moving to $s_3$ and 0 otherwise. What happens then is
$$
\delta V(s_3) = 0 \\
\delta V(s_2) = \alpha (max_a(R(s_2, a) + \gamma V(s_2) - V(s_2)) = \alpha (1 + \gamma V(s_2) - V(s_2)) \\
\delta V(s_1) = \alpha (max_a(R(s_1, a) + \gamma V(s_1) - V(s_1)) = \alpha (0 + \gamma V(s_1) - V(s_1))
$$
Because starting value for $V(s_1)$ is 0 it's gradient will always be 0 and the network will never learn the value of this state. That's why it must take a loook at values of its next state.


Did you also validate the data set with a separate test set so you can see how it performs with known data?

This answer has a good run down on how to do that:
https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

It also is a best practice to ensure that you haven't overfit the data. The easiest way to do that is check the accuracy against the data that you used to train the data.  If its not substantially better than the test data (this data should never be used to train a model) then you have not over fit.

After that, if you eventually find out the correct classification in your system, I would compare the accuracy there to the accuracy you experienced in the training and test environments to determine if its performing as intended.


How to calculate which word fits the best given a context and possible words?

There are two options :

CBOW . Modify Word2Vec CBOW code to save the whole trained model (current implementations only persist embedding layer)

  CBOW Model: This method takes the context of each word as the input and tries to predict the word corresponding to the context.

Intro : https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa
Example : https://www.tensorflow.org/tutorials/representation/word2vec

Train an LSTM / GRU to predict next word (given previous N words)

Karpathy's article is probably the best introduction to text generation with RNN (this works at character level, you will have to modify it to work at word level [Word-Vector level])

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Example :

https://medium.com/phrasee/neural-text-generation-generating-text-using-conditional-language-models-a37b69c7cd4b


Which type of clustering algorithm to use to identify the "same" item in different data sets?

A: ABC Notebook, Large, Released: 2018-02-20, 150mm x 100mm.
B: Notebook (ABC), L, Date: 18/02/2018, 150mm x 100mm

It doesn't seem like a clustering problem. Rather, it's more like a text mining problem. E.g., you extract "ABC" and "Notebook" from the names of both A and B, and then decide they are the same. Besides that, there are also some format detection needed, like mapping date string to datetime format, L to Large.
After you extracted these features and constructed a feature matrix, you can simply determine if two entries are different items/people by their commonly existing features in the matrix (since there will be NAs in your matrix). You don't really need a machine learning clustering algorithm.

A: Doe, Jane, DOB 1970-06-23, 123 ML Ave, F
B: John Doe, DOB 1971-04-33, 123 ML Avenue, M
C: J. Doe, Born '71

E.g., A and B are not the same since there are mismatches in DOB and sex. B and C can be treated as same, as they both have 2 features - name and DOB - show up in your data set, and both 2 features matched.
You might need a bit more complex algorithm, since there might be a D: John Doe, DOB 1971-03-12, F, with which you should leave B, C, D alone. But that's all it need, I guess.


All release notes for new versions of the Orange Data Mining software can be found by looking at the release page on its GitHub page here. Then just navigate to the most recent release and click on the latest branch. This will display a window similar to the following where you can view all of the changes that were made (both enhancements and bug fixes) in the most recent update.

Snapshot for Orange 3.21.0 below:


You do not have to normalise features used in logistic regression, but sometimes it can help.

You should normalise features used in logistic regression, if you are using a gradient-based optimiser (e.g. SGD) to find the optimum weights. That is because the optimiser will perform better when partial derivatives of the cost function are of similar magnitude in each direction. When the derivatives vary too much, you will need a lower learning rate to compensate (making learning slower, and more likely to get stuck) or the optimiser will not converge - it may oscillate or start to diverge instead.


Starting my career as Data Scientist, is Software Engineering experience required?

1) I think that there's no need to question whether your background is adequate for a career in data science. CS degree IMHO is more than enough for data scientist from software engineering point of view. Having said that, theoretical knowledge is not very helpful without matching practical experience, so I would definitely try to enrich my experience through participating in additional school projects, internships or open source projects (maybe ones, focused on data science / machine learning / artificial intelligence).

2) I believe your concern about focusing on data science too early is unfounded, as long as you will be practicing software engineering either as a part of your data science job, or additionally in your spare time.

3) I find the following definition of a data scientist rather accurate and hope it will be helpful in your future career success:

  A data scientist is someone who is better at statistics than any
  software engineer and better at software engineering than any
  statistician.

P.S. Today's enormous number of various resources on data science topics is mind-blowing, but this open source curriculum for learning data science might fill some gaps between your BSc/MSc respective curricula and reality of the data science career (or, at least, provide some direction for further research and maybe answer some of your concerns): http://datasciencemasters.org, or on GitHub: https://github.com/datasciencemasters/go.


How to change the input of conv2d from gray scale to coloured image

Change $w1=[9,9,1,64]$ to $w1=[9,9,3,64]$. The first two numbers specify the size of the filter, the third specifies the number of input channels and the last one specifies the number of output filters.

  Is it possible to use conv2d function for RGB images in Tensorflow?

Yes, conv2d is used for one channel, grey level, images and three channel images, coloured images.


Policy Gradient Methods - ScoreFunction & Log(policy)

You have to see this approach more general and not so much from a neural network point of view. The reason one uses the re-arranged version is really just because in the end the gradient of your objective function ends up being an expectation again. This is also pointed out by David Silver at min 44:06.

For an example of the benefits of using $\log$, see UC Berkeley's DeepRL lecture by S. Levine here. The name of the video is "CS294-112 9/6/17"

At some later point, S. Levine also goes into more details regarding the math of the policy gradient and draws comparisons to the maximum likelihood approach with neural networks.


There exists somewhere in the world a distribution $D$ from which you can draw some samples $x$. The notation $x \sim D$ simply states that the sample $x$ came from the specific distribution that was noted as $D$ (e.g. Normal or Poisson distributions, but also the possible pixel values of images of beaches).

Say you have some ground truth function, mark it as $c$, that given a sample $x$ gives you its true label (say the value 1). Furthermore, you have some function of your own, $h$ that given some input, it outputs some label.

Now given that, the risk definition is quite intuitive: it simply "counts" the number of times that $c$ and $h$ didn't agree on the label. In order to do that, you (ideally) will

go over every sample $x$ in your distribution (i.e. $x \sim D$).
run it through $c$ (i.e. $c(x)$) and obtain some label $y$.
run it through $h$ (i.e. $h(x)$) and obtain some label $\hat{y}$.
check if $y \neq \hat{y}$. If so, you add 1 to your count (i.e. $1_{h(x) \neq c(x)}$ - that notes the indicator function)

Now last thing to note is that I wrote above "count", but we don't really care if the number is 500 or 100, we care for the relative number of mistakes (like 40% or 5% of the samples that were checked were classified differently). That is why it is noted as expectancy ($\mathbb{E}$).

Let me know if that was clear enough :-)


R packages: How to access csv files in data subfolder?

data-raw is for storing data alongside a short R script that will do the conversion to R data for the user, and the user will just use the data() function.[source] Alternatively, if you want the raw CSV to be user-accessible, I think you need to use the extdata folder, as documented here. Then the user can get the actual path to the file on their system, after package installation, with system.file("extdata", ..., package = "mypackage"). Then, finally, they can feed that path to read.csv() with whatever options they like.


Can VAEs be used to generate multivariate data?

The CTGAN paper (github) use a VAE to generate multivariate synthetic data from tabular data. See section 4.5.


Can BRAT be installed on Microsoft Windows?

The installation documentation shows that the easiest way to run a brat server on Windows is in a virtual machine running a UNIX-like operating system such as Ubuntu. Also, there's an issue (still opening) about running brat server on windows. I think you're on your own right now.


For a regression model, can you transform all your features to linear to make a better prediction?

Your idea is good, but you are not the first with this idea.

You can use Generalized Additive Models (GAM) with regression splines to check and/or add non-linearity in a linear regression setup. There is a clear advantage over looking at just descriptive figures one-by-one (with manual feature generation), since you can estimate a whole model with extreme flexibility.

Alternatively, you can simply do a linear regression with a lasso penalty. Add polynomials to your $X$, and let the lasso „shrink“ irrelevant features to zero.

The book „Introduction to Statistical Learning“ covers these topics in Sections 6,7.

BTW: even with polynomials, your model can retain interpretability if you care for this. Polys are relatively easy to interpret. However, under a GAM approach, interpretation is a little more difficult.

Maybe as a note: you want to make features linear (which can be a problem since you can only apply linear transformations to the data). The approaches proposed above aim at making your (linear) model more flexible to cope with non-linearity in data.


How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

If you assume variables are dependent, in a Gaussian distribution this means the covariance matrix is not diagonal.


Are you constrained to use only R or Python? If you have the option, try using Alteryx. It's a GUI based tool which has greatly helped me explore and analyze large volumes of data. The tool is built on R so plugging in R functions is a breeze too.


Combining different features as input to Neural Network

You should have all of your features in a comparable range. You can leave lexicon values in the [0, 1] range but normalize embedding to have approximately the same range (with using, for example, MinMaxScaler).


Which is better for Data Science, a double major in Math & CS or Physics & CS?

My background is Physics undergrad, Physics grad, now in industry.  Some advice off the top of my head:

If you can push top 10%, do it (always do your best).
Don't be surprised when jobs in industry don't care about you ranking/gpa (some do, some don't.). Rank/gpa is  important for grad school however, and grad school is important for careers in data science
Major in what makes you happy.
If you stay in physics, learn to code and take some stats.
Physics will always impress people and there are many machine learning/ data science opportunities in physics.
Physics will teach you how and where to apply machine learning rather than being given perfect datasets to learn with.


Predict multi-steps for Multi-Time series output with Keras

LSTM inputs should be a tensor of size (#samples, time_steps, #features). Considering your dataset; you will have #samples = 2000-1 (for the training); time_steps = 1 (this is the lag-time you choose), and #features = 14. Then for the testing/validation #samples = 1000-1. But here, I suggest you take due consideration to the autocorrelation of the dataset; it seems that you choose the lag-time = 1 (just for the sake of creating the response variables); however, check out the temporal correlation + temporal structure of the data. Similar, to other Deep Neural networks, LSTM requires large dataset to train and test; checkout if you can increase the lag-time and get more predictor data. Have a look at Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras this similar question for more information.


Does small batch size improve the model?

In general smaller or larger batch size doesn't guarantee better convergence. Batch size is more or less treated as a hyperparameter to tune keeping in the memory constraints you have.

There is a tradeoff for bigger and smaller batch size which have their own disadvantage,  making it a hyperparameter to tune in some sense.

Theory says that, bigger the batch size, lesser is the noise in the gradients and so better is the gradient estimate. This allows the model to take a better step towards a minima. However, the challenge is that bigger batch size needs more memory and each step is time consuming.

Even if somehow we can avoid the time and space constraints, bigger batch size still wouldn't give better solution in practice as compared to smaller batch size. This is because the surface of the neural networks objective is generally non-convex, which means that there might be local optimums. Just having an accurate gradient estimate doesn't guarantee us reaching the global optimum (which we seek). It could lead us to a local optimum accurately! Keeping the batch size small makes the gradient estimate noisy which might allow us to bypass a local optimum during convergence. But having very small batch size would be too noisy for the model to convergence anywhere.

So, the optimum batch size depends on the network you are training, data you are training on and the objective function you are trying to optimize.


Of course you can reproduce that graph with some ggplot!

Here is the code to create it:

library(ggplot2)
library(scales)

df <- read.csv2('Graphs_SK.csv')

p <- ggplot(data = df, aes(x = Governorate)) +
  geom_segment(aes(y = Mid.May.2015, yend = Dec.16, xend = Governorate),
               color = 'grey', size = 1) +
  geom_point(aes(y = Mid.May.2015, color = 'start'), size = 2) +
  geom_point(aes(y = Dec.16, color = 'end'), size = 2) +
  scale_colour_manual(name = '(Population in millions)',
                      labels = c('May 2015', 'Dec 2016'),
                      values = c('start' = 'steelblue4', 'end' = 'steelblue')) +
  scale_y_continuous(labels = comma, name = '', position = 'top') +
  coord_flip() +
  ggtitle('1 Year of something of Governorate') +
  guides(colour = guide_legend(title.position = 'right')) +
  theme(panel.background = element_rect(fill = 'white'),
        legend.position = c(-.15, 1.03),
        legend.direction = 'horizontal',
        legend.justification = c(0, 0),
        legend.key = element_rect(fill = NA),
        legend.title = element_text(size = 10),
        axis.text.y = element_text(hjust = 0),
        axis.ticks.y = element_line(size = NA),
        plot.title = element_text(hjust = .5))


Complete-linkage clustering tends to find compact clusters with equal diameters. It is define the inter-group distances as the largest distance of objects between two clusters. Beside, it is may produce clusters that has the same number of objects and the distance between objects is contiguous.

So, I can say that your conjecture is correct. I cannot wait for any experts to explain this in detail.


Is there any good alternative for Keras library?

I can't say which is more efficient or easier but MXNet exists, has interfaces to most languages (python, scala, R, C++, perl, Julia), and the code looks fairly similar to keras.

https://mxnet.incubator.apache.org/

Another one that I've come across is caffe, mostly because people seem to share a lot of pre-trained nets in their model zoo.

http://caffe.berkeleyvision.org/

Update: I didn't see caffe 2 but I think this is quite separate to the original caffe.

Finally I've not fully had time to check out H2O's offering in deep water which is an alternative to Keras but still sits on top of TF/MXNet/Caffe


Clustering unique visitors by useragent, ip, session_id

One possibility here (and this is really an extension of what Sean Owen posted) is to define a "stable user."

For the given info you have you can imagine making a user_id that is a hash of ip and some user agent info (pseudo code):

uid = MD5Hash(ip + UA.device + UA.model)

Then you flag these ids with "stable" or "unstable" based on usage heuristics you observe for your users.  This can be a threshold of # of visits in a given time window, length of time their cookies persist, some end action on your site (I realize this wasn't stated in your original log), etc...

The idea here is to separate the users that don't drop cookies from those that do.

From here you can attribute session_ids to stable uids from your logs.  You will then have "left over" session_ids for unstable users that you are relatively unsure about.  You may be over or under counting sessions, attributing behavior to multiple people when there is only one, etc...  But this is at least limited to the users you are now "less certain" about.

You then perform analytics on your stable group and project that to the unstable group.  Take a user count for example, you know the total # of sessions, but you are unsure of how many users generated those sessions.  You can find the # sessions / unique stable user and use this to project the "estimated" number of unique users in the unstable group since you know the number of sessions attributed to that group.

projected_num_unstable_users = num_sess_unstable / num_sess_per_stable_uid

This doesn't help with per user level investigation on unstable users but you can at least get some mileage out of a cohort of stable users that persist for some time.  You can, by various methods, project behavior and counts into the unstable group.  The above is a simple example of something you might want to know.  The general idea is again to define a set of users you are confident persist, measure what you want to measure, and use certain ground truths (num searches, visits, clicks, etc...) to project into the unknown user space and estimate counts for them.

This is a longstanding problem in unique user counting, logging, etc... for services that don't require log in.


It is definitely possible to use them in the Encoder (Discriminator) part of a GAN, since according to Hinton's original paper it's used for classification which means it fits in Discriminators as well even though the proper loss function is a subject to research.

I chose a topic of Capsule Nets in Discriminators of GANs for my Master thesis and I'm working on the model right now. However, I simplified the routing algorithm by not vectorizing the DigitCaps values (otherwise Keras code gets complicated, although I'm not sure if what I've done is 100% correct).


Random seed in Machine learning model comparison

I agree that sometimes the authors are not that clear about some details. If a dataset is already splitted into train/test/validation, then there is not much to do (supposing the dataset is well made). Given a dataset not already splitted then you have (at least) two ways to test your model:

fix a random seed, check that the split is not biased and run several trainings. Even with the same split, the same model can converge to different local minima, because of the stochastic descent;
don't fix a random seed and run several trainings (but store the seed where you keep your experiment data!).

If your findings are not coherent with the literature, and you are sure there aren't bugs in the code, then you should ask specific questions or write to the authors. Depending on the model, you might try some standard validation techniques.


BERT: it is possible to use it for topic modeling?

No. Most folks use https://radimrehurek.com/gensim/models/ldamodel.html for topic modeling and https://github.com/bmabey/pyLDAvis for visualization.


Simply put, your data shows that for a given value of $x$, you can have a wide range of $y$ values and that the ranges depend on the value of $x$.

This usually happens when you have a partial view of the global picture, meaning that there are other variables in your data that could help explain the relationship.

To leverage your plots, you can consider changing the color of the markers depending on an additional variable. You could also change the size of the marker depending on another one. There are a couple of options there, depending on what you look for.

EDA is usually not the final step per se, you may rather ask yourself some questions related to the problem you are trying to solve, and see if the data accounts for it.

Does this help?


Why might trees work so much better than boosting classifiers?

As a disclaimer, I would like to point out that the performance in this specific case might still be due to the specific dataset used.

A likely explanation lies in the essence of what trees and boosting algorithms do.

As @akvall pointed out in the comments, Boosting algorithms can often overfit since this is what they are designed to do! As a reminder, regardless of how fancy a boosting algorithm works, it follows the following logic:

train on the training set
evaluate to see which mistakes were made
retrain and focus more on the previous mistakes
repeat until satisfactory results

Trees do not work the same way and are therefore less prone to overfitting. A Random Forest will simply compute independent trees and use majority voting to make a prediction.

Every boosting algorithm will "boost" for a certain amount of iterations, it is probably wise to look at how many times your algorithms did "boost".


Yes, theoretically that's what tf.function does. This uses a tensorflow module called AutoGraph to basically convert your python/numpy operations to tensorflow ops.
However not all python operations can be converted. Things like printing, appending to lists and mutating global variables won't work in graph mode!
I suggest you take a look at the official guide to see how it is used and if it is applicable in your situation.


Not able to connect to GPU on Google Colab

In Google Colab you just need to specify the use of GPUs in the menu above. Click:

Edit > Notebook settings >

and then select Hardware accelerator to GPU.

At that point, if you type in a cell:

import tensorflow as tf
tf.test.is_gpu_available()

It should return True.


Can I save prediction value in same csv file as a another column using panda python

if you want this column in the same dataframe just do

data['pred'] = pred1
data.to_csv('data1.csv')

The first line automatically adds a column called 'pred' to the dataframe with values coming from pred1.

Hope it helps. Good luck!


what is darknet and why is it needed for YOLO object detection?

https://pjreddie.com/darknet/ is their website...

I cite :

"Darknet: Open Source Neural Networks in C

Darknet is an open source neural network framework written in C and CUDA. It is fast, easy to install, and supports CPU and GPU computation."

As to why they used that, well it's open source and in C, which are good points and seems to be performant (see the graphs in your link and associated paper). But the main point seems to be about history. The darknet project seems to have started in 2014. Were tensorflow / keras available and perfromant at this time in the first place ? Even if they were why should they use keras / tensorflow ?


Text vectorizer that capture feature offset in the text?

One approach is to create another matrix that stores this information. Scikit-learn stores text features in a  document-by-token matrix. The cells of this matrix would be the token index in a document. This matrix then could be used as features during modeling.

It would require writing a custom vectorizer which would be similar to scikit-learn's CountVector implementation.


Eliminating rows in a dataframe based on specific conditions

This snippet should do the work:

city = ['PHX', 'JFK', 'NTU']
colomn_to_exclude = df.apply(lambda row: (row['origin'] not in city) and (row['destination'] not in city))
new_df = df[~colomn_to_exclude]

the second line checks the lines where your exclusion condition is verified and the line after subset the dataframe accordingly.

You could also write something like

colomn_to_exclude = df.apply(lambda row: (row['origin'] not in city) and   (row['destination'] not in city))
df.drop(~colomn_to_exclude, axis=1, inplace=True)

and you won't have to make a copy of your dataframe


Auto-Encoder to condense (pre-process) large one-hot input vectors?

It might be useful to think of this in terms of orthogonality. You state that "categories are not correlating in any way", which effectively means each category should be completely orthogonal to all the others -- otherwise one would be representing some degree of correlation (or anti-correlation) between at least two categories (to the degree that those categories are not purely orthogonal). Now, the only way to get 300 orthogonal vectors is in 300 dimensional space. There is no way to compress that without losing information.

Alternatively one can think in terms of using simpler dimension reduction techniques than autoencoders. Let's consider using PCA (which is, after all, just a kind of linear autoencoder with identity activation function). What does the input data look like? Presumably we have one observation for each creature, so 300 observations. We have 300 features. The result is a 300x300 identity matrix. The singular values of that are all identically 1, so PCA to anything less than 300 dimensions is just going to fail. Adding non-linear activations and extra layers is not going to fix that, it will just make it noisier via stochastic optimization so you may get a result -- it just isn't a meaningful one.

What you need if you want to compress things is some correlation structure amongst your creatures. If you can come up with a set of properties/features that creatures might have and then score all your creatures accordingly you may be able to do some dimension reduction on that -- but dimension reduction on completely orthogonal vectors just isn't a meaningful thing.


It is very unlikely for such huge dataset(450k ) to overfit after just one epoch.

Try to run the code for 20-30 epochs and see if you see any decrease in the validation set error (increase in validation accuracy.)

Try to change your learning rate if (unless you are using adaptive learning parameter)


Class Imbalance and Cost-Sensitive Learning XGBoost

XGBoost has several parameters to tune for imbalance datasets. You wouldn't mess with the objective function from my knowledge. You can find them below:

scale_pos_weight
max_delta_step
min_child_weight

Another thing to consider is to resample the dataset. We talk about Undersampling, Oversampling and Ensemble sampling. I think I was using the imbalanced-learn Python library for that. Things can get even more creative there. For instance, create many XGBoost trees on the same undersampled class instances but each XGBoost tree holds another copy (or randomly sampled copy) of your oversampled class instances. Then you can average your results.

In the end, you will have to try and see for yourself.


You could create a "flag" variable for particular actors. Each actor would have their own column in your data and the column would have a 1 if the actor was in that film and a 0 if the actor was not in that film.

Of course, this creates lots of columns and can be tedious to code, but binary indicator variables can be really useful if you want to build a predictive model - worth it in my opinion.

Example:

       actor1  actor2 actor3
movie1   1       0      0
movie2   0       1      1
movie3   0       1      0

Edit Re: Director/Actor

It is probably unrealistic to create a variable for every permutation of director/actor combinations. If you want to see how the variables interact you can code interactions into your model when you build it. If you were to do that in R it would look like this:

movie.predictor <- lm(suggestion ~ actor1 + actor2 + director1 + director2 +
                      actor1:director1 + actor2:director1 + actor1:director2 +
                      actor2:director2,
                      data = movies.data)

The ":" in R asks the lm(...) function to consider the interaction of those two variables in the regression problem, assuming you are doing regression. Don't worry the same ":" technique works across several model functions. Just read the documentation for your preferred language and package and look for how to code "variable interactions"

Of course, now you are typing out all of the permutations of variable interactions. You might want to also consider a clustering algorithm to group actors and directors together. I am not a movie expert but it seems like directors have favorite actors (and vice-a-versa) so I would expect there to be clusters of directors and actors with a little overlap here and there.


How to normalize just one feature by scikit-learn?

As mentioned, the easiest way is to apply the StandardScaler to only the subset of features that need to be scaled, and then concatenate the result with the remaining features.

Alternatively, scikit-learn also offers (a still experimental, i.e. subject to change) ColumnTransformer API. It works similar to a pipline:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import  StandarScaler
column_trans = ColumnTransformer(
    [('scaler', StandardScaler(),2],
    remainder='passthrough')
column_trans.fit_transform(X)

The third entry of the tuple (here : 2) can be a single or a list of column indices (or column names when working with a pandas DataFrame).


how can I generate a Bernoulli block mixture model in matlab?

First, $x_{ij}$ is not a bernoulli random variable. In the text, $x_{ij}$ is described as real valued. Also, the dimensions of $z$ and $w$ are incorrect.

The text indicates that we're drawing a matrix of data call it $X (n\times d)$ such that $X_{i,j}=x_{ij}$.

A particular entry has distribution $x_{ij} \sim f(\alpha_{z_i, w_j})$. Where $\alpha$ is some given matrix of parameters. The maximum row index of $\alpha$ is $g$. The maximum column index of $\alpha$ is $m$. So $\alpha$ is $g \times m$.

All that remains is to pin down the indices $z$ and $w$. These are vectors rather than matrices since they have a categorical distribution, for example $z_i$ should be an integer between $1$ and $g$.

Putting this together we need to:

Draw $z$, a vector of length $n$, by $z_i \sim \text{Cat}(\pi), i=1,...,n$
Draw $w$, a vector of length $d$, by $w_j \sim \text{Cat}(\rho), i=1,...,d$
Draw $X$, an $(n\times d)$ matrix, by $x_{ij} \sim f(\alpha_{z_i, w_j})$


Dealing with Skewed data using Boxcox transformation by adding 1

Boxcox you need to tune the lambda parameter, in order to transform the data to normal one (and than its implicitly non-skewed) however, I would suggest to rather use log transform, its faster.


Convolutional neural networks for non-image applications?

They can be employed wherever you can find meaningful adjacent patterns in the input. As an example, you can take a look at here for its use in NLP:

You can also read the well-known paper Convolutional Neural Networks for Sentence Classification which is about sentence classification.


Cluster analysis as an associative model?

I think the question you're wrestling with is essentially this: Is there a way to use information that may be present in the data without A as part of your strategy for predicting A?  There is actually a name for the set of methods that do exactly that: semi-supervised learning.

While there are multiple techniques, a method analogous to what you suggest in your question would be something like the following.  Here I refer to the set of data with A as labeled and the rest as unlabeled.

Apply an unsupervised learning technique (such as clustering) to the full data set (both labelled and unlabelled).
Transform the labelled data based on the result of your unsupervised learning technique.
Apply a supervised learning technique with the transformed set (possibly combined with the original set) of labelled data.

Generally you would want to apply a cross-validation strategy as well to detect overfitting.


How to use the $\chi^2$ test to select the features, that can be String or categorical?

You can only compute chi2 between two numerical arrays. You are getting that error because  you are comparing a string. Also I am not sure if it works for multiclassification also.

df = df.apply(LabelEncoder().fit_transform)

This will solve the problem for you. But there are a thousand ways to encode features and for sure other will work better for you.


Classifier that optimizes performance on only a subset of the data?

I want to clarify that the fact that you can pick only 100 boxes from the test dataset is NOT a limitation (like you describe it). You just need to see it from this perspective:

  How to pick the 100 boxes (out of 500K) that MOST LIKELY contain gold?

A very suitable algorithm would be a Decision Tree classifier. By using the Information Gain (based on Entropy Reduction), it can learn which features can split the dataset more accurately into two classes, i.e. the one that contains gold and the one that doesn't, and neglect (prune) the weakest attributes. After that, you will not only have a trained classifier but also gain insight on which features are the most important to make the decision (very important information).


Calculating Feature Importance of Time Series Data

For time series data,

Sensitivity analysis can help with overall Importance of a feature. For example, is  "Day of the week" a good feature for stock price forecasting. LIME is one approach that can help. Details :  https://arxiv.org/abs/1606.05386 . One simple way is to mask each feature and check the impact on model's performance.
Auto-corelation and Seasonality removal  (Details in tutorial at end of the answer)
SHAP : (SHapley Additive exPlanations) is good at identifying features that impact output with lag (https://medium.com/datadriveninvestor/time-step-wise-feature-importance-in-deep-learning-using-shap-e1c46a655455)

End to end example :
https://machinelearningmastery.com/feature-selection-time-series-forecasting-python/


How are precision and recall better metrics than accuracy for classification in my example?

Your calculation is correct, but you forgot to ask yourself one question: why should we consider "red" as the positive class? Precision and recall can be calculated for every class (i.e. considering the current class as positive), as opposed to accuracy.
So if we take "blue" as positive we get:

precision = NaN (because there's 0 predicted positive)
recall = 0 / 2 = 0

Usually after that one would calculate micro- and/or macro-precision/recall:

macro-precision = NaN because the precision is NaN for blue
micro-precision = totalTP / totalTP+totalFP = 8/10
macro-recall = ( 1 + 0 ) / 2 = 0.5
micro-recall = totalTP / totalTP+totalFN = 8/10

By definition, micro-performance favours the majority class so it's still high. However macro-performance would clearly show the problem that accuracy cannot show.


Try using the number of sample instead of None in the x_ placeholder. Also, try making these changes.

n_samples = len( XOR_X )
x_ = tf.placeholder(dtype=tf.float32 , shape=[n_samples, num_input], name='X')


Pandas - how can I calculate a field to count the number of occurences that values in the ID field appear in sequential order

Welcome to Data Science! Here I create your data frame and show one way to create the column you need. I use numpy in addition to pandas:

import pandas as pd
import numpy as np

# Create df
ID = ['bike', 'bike', 'car', 'car', 'car']
Color = ['red', 'black', 'green', 'orange', 'blue']
df = pd.DataFrame(data={'ID': ID, 'Color': Color})

Add a new column called 'Order', with counts based on each group - these are mini dataframes that contain only one of the ID values only. This makes use of the groupby method on a Pandas dataframe.

df['Order'] = df.groupby('ID').transform(lambda x: 1 + np.arange(len(x)))

The transform method takes a function and applies it to each group. I use an anonymous function (just a function with no name), also called lambda functions, using the keyword lambda.

The result:

    Color    ID  Order
0     red  bike      1
1   black  bike      2
2   green   car      1
3  orange   car      2
4    blue   car      3

As anymous.asker mentioned, it would be helpful in the future if you post the code that creates your dataframe!


What pandas function is most similar to the transpose procedure in SAS?

The pivot operator can be used to achieve the desired result. As per your example:
Define original data:
import pandas as pd
data = {'Txn_ID': [1, 1, 2, 2, 2, 3],
        'Item_Desc': ['Apples', 'Milk ', 'Eggs', 'Flour', 'Salt', 'Eggs'],
        'Amt': [0.96, 2.10, 7.00, 2.20, 4.75, 3.50]
}

Create dataframe
df = pd.DataFrame.from_dict(data)

Transpose result using the .pivot operator
df_new = df.pivot(index = 'Txn_ID', columns = 'Item_Desc')

| Txn_ID | Apples | Eggs | Flour | Milk | Salt |
|--------|--------|------|-------|------|------|
| 1      | 0.96   | NaN  | NaN   | 2.10 | NaN  |
| 2      | NaN    | 7.00 | 2.20  | NaN  | 4.75 |
| 3      | NaN    | 3.50 | NaN   | NaN  | NaN  |


You may use a sampling based method, in which you get an approximation of the actual result by combining the partial results for samples extracted from your data. There are several methods based on sampling, but you could take a look to the following link as an example, so you can understand what I am talking about:

A Fast Approximation to Multidimensional Scaling


Is there any method to determine which clustering algorithm to use on a particular dataset?

No.

Clustering is an explorative technique, it is subjective what is good, and the best clusters are those that are "interpretable but unexpected", a property that you cannot quantity with statistics. So it is a trial-and-error task.

Furthermore, data preparation is much more important than the choice of clustering algorithm. On badly prepared data, none will work.

Last but not least, categoricial data is a huge problem. It lacks detail for most clustering approaches - treating this as binary variables is much too coarse and tends to produce bad solutions (such as tiny "clusters" and trivial splits on a single variable). This is likely a problem of the data, not the algorithm. Similar issues can be seen with integer attributes or any other attribute that has only very few discrete levels (including Likert-like-scale questionnaires). Methods such as k-modes exist for categoricial data, but often don't produce better results either...


Why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?

I agree with the comment you received from Cross Validated – data leakage is something that fits this problem setting as it's known to cause too optimistic CV score when compared to test score. We could confirm that it's actually a data leakage problem if you provided information about the data pre-processing steps that you've taken.

Data leakage means that information is leaking from the test data to the training data, causing the trained model to over-estimate expected performance and to generalize poorly to unseen data, because the leaked information allows the model to learn something that it otherwise wouldn't learn.

As an example, let's say that you had only a small amount of training examples and you end up synthesizing more training examples that are somehow similar to the other training examples. If you implement the synthesis before CV loop and not inside it (within each fold), your CV accuracy will skyrocket but test accuracy will be low. In addition to data synthesis, data leakage can also happen with other pre-processing steps, such as scaling the data, inputting missing values or performing feature selection.


Decision tree and SVM for text classification - theory

Similarly to NB or kNN, the DT and SVM algorithms work with the features which are provided as input. So whenever ML is applied to text it's important to understand how the unstructured text is transformed into structured data, i.e. how text instances are represented with features.
There are many options, but traditionally a document is represented as as a vector over the full vocabulary. A very simple version of this is a boolean vector: a cell $v_i$ contains 1 if the word $w_i$ occurs in the document and 0 otherwise. The DT training will generate the tree the usual way, so in this case the conditions at the nodes will be v_i == 1, representing whether the word $w_i$ is present or not. If the values in the vector are say TFIDF weights, the conditions might look like v_i > 3.5 for instance. Similarly for SVM: the algorithm will find the optimal way to separate the instances in a multi-dimensional space: each dimension actually represents a single word, but the algorithm itself doesn't know (and doesn't care) about that.


Multilabel classifcation in sklearn with soft (fuzzy) labels

For example, for a 3-class classification, we want to train with a label like $A$, which is one-hot encoded as $(1, 0, 0)$, and also with a fuzzy label like $(0.8, 0.2, 0)$. In that case, kNN and SVM of sklearn does not support fuzzy labels.

However, we can use sklearn's MultiOutputRegressor that extends a one-output Regressor such as Support Vector Regression (SVR) to multiple outputs. It is worth noting that neural networks are a natural fit for this type of label since they readily work with numerical vectors as labels.

Here is a code that goes through different types of labels for kNN, SVC (multi-class SVM), and MultiRegression SVR:

import sklearn
import pandas as pd
from sklearn.svm import SVC, SVR
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multioutput import MultiOutputRegressor
import numpy as np

N = 1000
split = int(0.8 * N)
folds = 5
seed = 1234

# Data
np.random.seed(seed)
feature_1 = np.random.normal(0, 2, N)
feature_2 = np.random.normal(5, 6, N)
X = np.vstack([feature_1, feature_2]).T

Y_label = np.random.choice(['A', 'B', 'C'], N)

Y_one_hot = pd.get_dummies(Y_label).values

smooth_filter = np.array([0.01, 0.98, 0.01])
Y_fuzzy = np.apply_along_axis(
    lambda m: np.convolve(m, smooth_filter, mode='same'), axis=1, arr=Y_one_hot
)

kfold = KFold(n_splits=folds, random_state=seed)

kNN = KNeighborsClassifier(n_neighbors=3)
svc = SVC()
svr = SVR()
multi_svr = MultiOutputRegressor(estimator=SVR())

knn_label = np.average(cross_val_score(kNN, X, Y_label, cv=kfold))
knn_one_hot = np.average(cross_val_score(kNN, X, Y_one_hot, cv=kfold))
try:
    knn_fuzzy = np.average(cross_val_score(kNN, X, Y_fuzzy, cv=kfold))
except ValueError:
    print('kNN: fuzzy classes are not supported')
svc_label = np.average(cross_val_score(svc, X, Y_label, cv=kfold))
try:
    svc_one_hot = np.average(cross_val_score(svc, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVC: vector is not supported')
try:
    svr_one_hot = np.average(cross_val_score(svr, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVR: vector is not supported')
multi_svr_one_hot = np.average(cross_val_score(multi_svr, X, Y_one_hot, cv=kfold, scoring='neg_mean_absolute_error'))
multi_svr_fuzzy = np.average(cross_val_score(multi_svr, X, Y_fuzzy, cv=kfold, scoring='neg_mean_absolute_error'))

print('sklearn version', sklearn.__version__)
print('Y example: ',
      "label: ", Y_label[0],
      ", one hot: ", Y_one_hot[0, :],
      ", fuzzy: ", Y_fuzzy[0, :])
print('kNN label: ', knn_label)
print('kNN one hot: ', knn_one_hot)
print('SVC label: ', svc_label)
print('MultiSVR one hot: ', multi_svr_one_hot)
print('MultiSVR fuzzy: ', multi_svr_fuzzy)

Output:

kNN: fuzzy classes are not supported
SVC: vector is not supported
SVR: vector is not supported
sklearn version 0.19.1
Y example:  label:  B , one hot:  [0 1 0] , fuzzy:  [0.01 0.98 0.01]
kNN label:  0.321
kNN one hot:  0.254
SVC label:  0.332
MultiSVR one hot:  -0.4066160996805417
MultiSVR fuzzy:  -0.3970780923514713

Although kNN does not throw an exception for one-hot encoded labels, accuracy 0.254 shows that it does not work correctly with the vector.

Also, Negative Mean Absolute Error is reported for MultiSVR since the task is understood as regression. Score accuracy can only be used after changing the fuzzy labels and predictions back to a label.


What recommendation engine for a situation where users can only see a fraction of all items?

I feel that you need to address two things separately.

First, You need to have access control for the users in your system. You can have access tokens attached to each user and file. Filter the files database before you process.

Second, Ranking the documents I would suggest have some weight for a document weight and user weight which relative to the current browsing user.

For example I can think of document weight and user weight as follows but they can be much more complex as per your system-

DocumentWeight = Number of Views/ Number of Users can Access
UserWeight = ## Relative to browsing user- Users in similar project will have higher weights

DocumentScore = Sum over all viewed users{DocumentWeight x UserWeight}

You can rank the documents, this will statistically pull up the documents you need. I hope this will be some help.


Transform an Autoencoder to a Variational Autoencoder?

Yes. Two changes are required to convert an AE to VAE, which shed light on their differences too. Note that if an already-trained AE is converted to VAE, it requires re-training, because of the following changes in the structure and loss function.

Network of AE can be represented as $$x \overbrace{\rightarrow .. \rightarrow  y \overset{f}{\rightarrow}}^{\mbox{encoder}} z \overbrace{\rightarrow .. \rightarrow}^{\mbox{decoder}}\hat{x},$$

where

$x$ denotes the input (vector, matrix, etc.) to the network, $\hat{x}$ denotes the output (reconstruction of $x$),
$z$ denotes the latent output that is calculated from its previous layer $y$ as $z=f(y)$.
And $f$, $g$, and $h$ denote non-linear functions such as $f(y) = \mbox{sigmoid}(Wy+B)$, $\mbox{ReLU}$, $\mbox{tanh}$, etc.

These two changes are:

Structure: we need to add a layer between $y$ and $z$. This new layer represents mean $\mu=g(y)$ and standard deviation $\sigma=h(y)$ of Gaussian distributions. Both $\mu$ and $\sigma$ must have the same dimension as $z$. Every dimension $d$ of these vectors corresponds to a Gaussian distribution $N(\mu_d, \sigma_d^2)$, from which $z_d$  is sampled. That is, for each input $x$ to the network, we take the corresponding $\mu$ and $\sigma$, then pick a random $\epsilon_d$ from $N(0, 1)$ for every dimension $d$, and finally compute $z=\mu+\sigma \odot   \epsilon$, where $\odot$ is element-wise product. As a comparison, $z$ in AE was computed deterministically as $z=f(y)$, now it is computed probabilistically as $z=g(y)+h(y)\odot \epsilon$, i.e. $z$ would be different if $x$ is tried again. The rest of network remains unchanged. Network of VAE can be represented as $$x \overbrace{\rightarrow .. \rightarrow  y \overset{g,h}{\rightarrow}(\mu, \sigma) \overset{\mu+\sigma\odot \epsilon}{\rightarrow} }^{\mbox{encoder}} z \overbrace{\rightarrow .. \rightarrow}^{\mbox{decoder}}\hat{x},$$
Objective function: we want to enforce our assumption (prior) that the distribution of factor $z_d$ is centered around $0$ and has a constant variance (this assumption is equivalent to parameter regularization). To this end, we add a penalty per dimension $d$ that punishes any deviation of latent distribution $q(z_d|x) = N(\mu_d, \sigma_d^2)$$= N(g_d(y), h_d(y)^2)$ from unit Gaussian $p(z_d)=N(0, 1)$. In practice, KL-divergence is used for this penalty. At the end, the loss function of VAE becomes: $$L_{VAE}(x,\hat{x},\mu,\sigma) = L_{AE}(x, \hat{x}) + \overbrace{\frac{1}{2} \sum_{d=1}^{D}(\mu_d^2 + \sigma_d^2 - 2\mbox{log}\sigma_d - 1)}^{KL(q \parallel p)}$$
where $D$ is the dimension of $z$.

Side notes

In practice, since $\sigma_d$ can get very close to $0$, $\mbox{log}\sigma_d$ in objective function can explode to large values, so we let the network generate $\sigma'_d = \mbox{log}\sigma_d  = h_d(y)$ instead, and then use $\sigma_d = exp(h_d(y))$. This way, both $\sigma_d=exp(h_d(y))$ and $\mbox{log}\sigma_d=h_d(y)$ would be numerically stable.
The name "variational" comes from the fact that we assumed (1) each latent factor $z_d$ is independent of other factors, i.e. we ignore other $(\mu_{d'}, \sigma_{d'})_{d' \neq d}$ when we sample $z_d$, and (2) $z_d$ follows a Gaussian distribution. In other words, $q(z|x)$ is a simplified variation to the true (and probably a more complex) distribution $p(z|x)$.


Identifying important interactions between features using machine learning

I think that it is important, in this situation, to ask yourself why you are using machine learning to detect the interactions. It feels a bit like data dredging. Using domain knowledge to think about what interactions would be likely or feasible given the phenomenon you are studying may well serve you well.

That said, you can fit a tree model using, for instance, R. This will allow you to detect complex and high order interactions. You can see this as an example on page 30 here. Note that I would probably use partykit for fitting the tree model because it allows you to use non-normal distributions. This method, though, works best if you have a very large data set and you may struggle to fit them in a linear model once you have found them (because they can only occur on one side of the tree).

An alternative method that addresses some of the limitations above is to use a technique called Additive Groves. These work on the principle of observing the relative performance of differently restricted tree models. Because interaction effects are not additive, this technique can allow for the identification of interactions.


Why doesn't loss go down during Neural Net training?

Your loss does go down, but not significantly. This is because your target values are very large ~10e7 and the default learning rate is scaled for smaller values. The easiest way to fix this is to normalize Y.

If your intention is for your code scale Y, then the problem is that Pipeline does not apply StandardScaler (or any transformations) to Y. You have to use sklearn.compose.TransformedTargetRegressor, or apply the transforms to Y outside of Pipeline.

Pick 1:

Outside the pipeline:
Y = dataframe['revenue'].values
Y = StandardScaler().fit_transform(dataframe['revenue'].values.reshape(-1,1))

Inside the pipeline:
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))

from sklearn.compose import TransformedTargetRegressor
estimators.append(('mlp',TransformedTargetRegressor(
regressor=KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1),transformer=StandardScaler())))


Model selection and assessment using leave-one-out cross validation

What you just said is dividing the data into three parts: training, validating and testing.
This is a very common practice in machine learning.
We use validation to help in selecting hyper parameters.
Going with this option vs. Leave one out vs. cross validation depends mainly on how many number of samples you have. If you have a lot of samples, then going with the option of splitting the data into three parts can be more efficient.
I am not sure about your statement of unbiased. You need to be careful when you make judgment


Naive Bayes for Multi label text classification

Since generally, Naive bayes alogorithms implemented by most libraries don't support multilabel classification. Although You can devise your own algorithm taking inspirations from naive bayes approach.

For a particular class c, and document d (with 3 words w1,w2,w3)

'/' stands for 'given' p(a/b) = probability(a given b)

As per bayes theorem.

p(c/d) = p(c,d) / p(d)

p(c/d) = (p(c)*p(d/c)) / p(d)

where:

p(d/c) = p(w1/c)*p(w2/c)*p(w3/c)

Since words are assumed to be independent of each other.

And p(w1/c) can be obtained using your code, calculating count of w1  inside class c documents divided by overall count of w1 in all documents or you can use your own logic

But If want to avoid writing detailed code

you can restructure your input data to achieve multilabel classification. such that a given document d with n labels/classes (eg d labeled with c1,c2,c3) get expanded into data of n samples (3 here) of same document d with different label each time ((d,c1),(d,c2),(d,c3)). Now you can pass this data into any R/python library supporting multinomial naive bayes. Same needs to be done in your dataset as well. Currently, you are passing traindata$topics which is y variable (training labels) as it is, without modification.

Even after you train your data using this approach. You need to use a probability threshold e.g 0.2, so that class labels with probability above 0.2 will be assigned to that test document.

More better Approach which requires restructuring your input

If you have 'n' class labels then you can train 'n' different binary naive bayes classifier for each of the class. For example, for training a classifier for class 'c1', you train a naive bayes classifier with dependent  variable Y denoting the presence of class c1 on that document as '1' and absence as '0'.

After you train 'n' binary naive bayes classifier using this approach. You will now use the output of these n classifiers. e.g if out of these, n classifiers, if a particular classifier, which corresponds to class 'c1' has output probability above 0.5, then class label 'c1' will be assigned to this test document.


Regex remove 1-2 character sequences: hyphens literal, not word boundaries


You look at loss at every batch. You should average your loss over all batches. When you look at different batches your loss may increase simply because one batch is harder to predict than the other one. That's why it's not really interpretable. So start with that. If the problem persists it's probably exploding gradients. In that case lower your learning rate to 1e-3 or 1e-4 or even less if it continues.


Transfer learning between Language Model and classification

Ok, so it seems that the whole idea of transfer learning in NLP is to use more than just the word-embeddings which is considered "low level", but rather higher level representations. This is akin to what goes on in Computer Vision where the final (or almost final) layer embedding in nets trained on ImageNet are then used in transfer learning in other tasks, or as the authors dubbed it: "a counterpart of ImageNet for NLP". This is similar to what GPT is doing.

You can find more info also in the accompanying article one of the authors wrote.

Specifically, from what I gather after reading the paper, is that they use a 3-layer LSTM with dropout for the Language Model part. Then they fine tune it with the relevant text (using a bunch of tricks specified in the paper/lecture). Then they use the same architecture weights, but augment the model with 2 additional linear blocks. The input for the final blocks are: the final input, the final hidden state, maximum and averaged hidden state of all previous hidden states of the regular. This model is also fine-tuned on the supervised task (classification) using "gradual unfreezing".


You can indeed use the ability of recurrent network like LSTM to handle the varying length problem.
But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do :

Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the Input layer : the 0s will not be taken into account for training the model. Bellow an example using the Keras framework (not complete, not tested) :

from keras.models import Model
from keras.layers import Input, LSTM, Dense
input = Input(shape=[None, nb_of_features_per_timestep])
layer = Masking(mask_value=0)(input)
layer = LSTM(50, return_sequences=False)(layer)
output = Dense(nb_classe)(layer)
model = Model(input,output)

You can also use the bucketing trick : you group your dataset into "buckets" of same size (approximately). You can then create batches of same size. You don't need the Masking layer anymore.
You can also use fully convolutional network. Those networks can also take Input with different size. I let you search if you want to use that.

Concerning encoder/decoder : those model can maybe help you but I think that you will have to use one of the tricks that I mentioned.
If you don't understand how RNNs work, I recommend you to dig a little into the subject before.


How to store strings in CSV with new line characters?

I assume that you want to keep the newlines in the strings for some reason after you have loaded the csv files from disk. Also that this is done again in Python. My solution will require Python 3, although the principle could be applied to Python 2.

The main trick

This is to replace the \n characters before writing with a weird character that otherwise wouldn't be included, then to swap that weird character back for \n after reading the file back from disk.

For my weird character, I will use the Icelandic thorn: Þ, but you can choose anythin that should otherwise not appear in your text variables.
Its name, as defined in the standardised Unicode specification is: LATIN SMALL LETTER THORN. You can use it in Python 3 a couple of ways:

weird_name = '\N{LATIN SMALL LETTER THORN}'
weird_char = '\xfe'             # hex representation
weird_name == weird_char        # True

That \N is pretty cool (and works in python 3.6 inside formatted strings too)... it basically allows you to pass the Name of a character, as per Unicode's specification.

Replacing \n

Now we use this weird character to replace '\n'. Here are the two ways that pop into my mind for achieving this:

using a list comprehension on your list of lists: data:

new_data = [[sample[0].replace('\n', weird_char) + weird_char, sample[1]]
             for sample in data]

putting the data into a dataframe, and using replace on the whole text column in one go

df1 = pd.DataFrame(data, columns=['text', 'category'])
df1.text = df.text.str.replace('\n', weird_char)

The resulting dataframe looks like this, with newlines replaced:

               text              category
0         some text in one line      1
1  text withþnew line character      0
2    another newþline character      1

Writing the results to disk

Now we write either of those identical dataframes to disk. I set index=False as you said you don't want row numbers to be in the CSV:

FILE = '~/path/to/test_file.csv'
df.to_csv(FILE, index=False)

What does it look like on disk?

  text,category

  some text in one line,1

  text withþnew line character,0

  another newþline character,1

Getting the original data back from disk

Read the data back from file:

new_df = pd.read_csv(FILE)

And we can replace the Þ characters back to \n:

new_df.text = new_df.text.str.replace(weird_char, '\n')

And the final DataFrame:

new_df
               text               category
0          some text in one line      1
1  text with\nnew line character      0
2    another new\nline character      1

If you want things back into your list of lists, then you can do this:

original_lists = [[text, category] for index, text, category in old_df_again.itertuples()]

Which looks like this:

[['some text in one line', 1],
 ['text with\nnew line character', 0],
 ['another new\nline character', 1]]


Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset

...Because the Keras documentation does not specify the keys for the class_weights..

You may get an idea with these two parameters,

labels: Either "inferred" (labels are generated from the directory structure), or a list/tuple of integer labels of the same size as the number of image files found in the directory. Labels should be sorted according to the alphanumeric order of the image file paths (obtained via os.walk(directory) in Python).

class_names: Only valid if "labels" is "inferred". This is the explicit list of class names (must match names of subdirectories). Used to control the order of the classes (otherwise alphanumerical order is used).

So, it will be mapped to alphabetical order if not specified explicitly.
traindata = image_dataset_from_directory("/content/Data") # Default
traindata = image_dataset_from_directory("/content/Data",class_names=['train_data','test_data']) #as per the values passed in class_names

Does anyone have a solution that may work in combination with a dataset loaded by image_dataset_from_directory?

You may try a wrapper of a custom Generator over the initial Keras generator. Below is a very basic code sample
def cust_gen():
    for image, label in traindata:
        yield image, label, np.ones((2,1)) # 3rd parm is the sample_weight

history = model.fit(cust_gen(), epochs=epochs)

You will have to write a custom code to create the sample_weight for each batch.


Measure of correlation for term frequency

If you did a simple linear regression for each of the 5 outputs, your r-squared would be a good measure of the "aggregate correlation." If you wanted to compare to the other results, it would be helpful to do the same for each of your single features as well.


How can a GRU perform as well as an LSTM?

GRU and LSTM are two popular RNN variants out of many possible similar architectures motivated by similar theoretical ideas of having a "pass through" channel where gradients do not degrade as much, and a system of sigmoid-based control gates to manage signals passing between time steps.

Even with LSTM, there are variations which may or may not get used, such as adding "peephole" connections between previous cell state and the gates.

LSTM and GRU are the two architectures explored so far that do well across a wide range of problems, as verified by experiment. I suspect, but cannot show conclusively, that there is no strong theory that explains this rough equivalence. Instead we are left with more intuition-based theories or conjectures:

GRU has less parameters per "cell", allowing it in theory to generalise better from less examples, at the cost of less flexibility.
LSTM has a more sophisticated memory in the form of separating internal cell state from cell output, allowing it to output features useful for a task without needing to memorise those features. This comes at the cost of needing to learn extra gates which help map between state and features.

When considering performance of these architectures in general, you have to allow that some problems will make use of these strengths better, or it may be a wash. For instance, in a problem where forwarding the layer output between time steps is already a good state representation and feature representation, then there is little need for the additional internal state of the LSTM.

In effect the choice between LSTM and GRU is yet another hyperparameter to consider when searching for a good solution, and like most other hyperparameters, there is no strong theory to guide an a priori selection.


You can use SVD to build a recommendation engine, but I don't think it's the best way to get intuition around what's going on under the hood. Regardless, here's a presentation with more details, I'd recommend reviewing slide 9.

And to answer your questions:

A_k represents an embedding dimension (i.e. the low-rank approximation) that is used to predict the user-rating matrix.
The cosine similarity is just the dot product for user $i$ and item $j$, which maps to the predicted rating for user $i$ and item $j$. The dot product is what defines the users and items as being similar.
Yes, you should use the MAE on A and A_k. You may prefer to use MSE instead. This measures the quality of your predictions for user $i$ and item $j$. Note, this is obviously the MSE of a matrix, which is the Frobenius Norm.

I think an easier way to understand SVD is to see it applied to image compression for different components. See this presentation here.


Match between objective function and evaluation metric

The evaluation metric for model validation has to be the same throughout the the hyperparameter search process in order fairly compare different models.
The objective function for model fitting can be different throughout the hyperparameter search process. During the hyperparameter search process, you can compare different algorithms and each of those algorithms can have different objective functions.
The objective function and evaluation metric should be thought of as complete separate concepts. There can only one objective function per algorithm. However, there can be many evaluation metrics. Objective functions are chosen for computers so they can efficiently and effectively fit the training data. Evaluation metrics are chosen for human stakeholders so they can better understand the impact of the model.
The confusion between the two concepts often comes from just measuring a machine learning system on minimizing error. If the measurement of a machine learning system broadens to include other requirements, then the difference between objective function and evaluation metric is more clear. The objective function is typically setup only to minimize error over the training dataset and work well with the chosen optimization technique. Evaluation metrics can also assess model error but can also include prediction speed, model size, model fairness, and many other requirements.


Changing a list of tensors to a tensor of lists!

In order to stack list of tf tensors you could use the tf function stack hence the name
for ur case,

tf.stack([some_function(x) for x in something],axis=0)

or you could stack them in numpy and then convert the array to a tensor, to do so using numpy np.stack inputting a list and axis


How to deal with name strings in large data sets for ML?

You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace names by the mean on target variable, however, this is highly prone to overfitting and you should take care.

The following two videos will give an excellent explanation:

https://www.coursera.org/learn/competitive-data-science/lecture/b5Gxv/concept-of-mean-encoding
https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization

However, I would also consider taking out sensitive information such as name depending on your application, always think about if the features makes sense.

I hope this helps, any question let a comment.


I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data

You can use flatten numpy for this task. Simply convert your data to a numpy array and use the function as:

In [1]: import numpy as np

In [2]: x = [[[1,2], [3,4]]]

In [3]: x
Out[3]: [[[1, 2], [3, 4]]]

In [4]: np.array(x).flatten()
Out[4]: array([1, 2, 3, 4])


How to create visualisation on medical data?

If you have good working knowledge in Python than you are good for it. D3.js actually has a Python comparable called Bokeh. You question implies a lot of ramifications and does explain fully what you are trying to achieve so let's go by parts.

Some 2D Scientific Plotting Python Libraries:

Matplotlib
ggplot
Bokeh
Chaco
pyQtGraph (This one has some interesting features in volume slicing if it suits you)

Some UI development Python libraries:

PyQt5
PyQt4 (there are significant changes to PyQt5, thus the suggestion)
PySide (also a Qt port)
wxPython

Some Python numerical libraries:

Pandas
Numpy
Scipy
Scikit-learn (You can do some serious data analysis with this)

Using combinations of the above you can build some very powerful software solutions. Any of the UI libraries suggested has spreadsheet widgets. There are libraries to read csv and excel, among others (JSON, HDF5, etc.).

The data problem can be resolved by either building your own randomized data or by using resources on the net like this or this.


Parking Prediction based on Mobile application

If this data is coming in real-time then you don't need a model -- simply check how many spots have an ExpiryDate greater than now (i.e. when you need to provide a prediction) and subtract this from the total capacity of spots.
If the data is not coming in real-time, then you could use time of day and day of week as predictors. You might even want to make them into interaction terms. You would also need to decide how often you want to call your model and group your data so that each row represents how many tickets were active during that timeframe; this will define your target variable (what you're trying to predict).
By the way, I think you're referring to a survival model. I would recommend gradient boosting instead; it's much more powerful. Gradient boosting models (GBM) are part of caret in R and scikit-learn in Python by the way.


Slight confusion on the learning process

Well, you make a very good point, this is the most painful part of machine learning, we have to label data to make them usable to train models.By labelling the data, we mean to go through the dataset and say what the output of each input is. It means that to each input we attribute a true output.
The input of the learning function in sk-learn for example are always (almost) model.train(X, Y). with X the inputs and Y the corresponding expected outputs.
Labelling is a tedious and long task that must be hand made most of the time. Some website even pay people to label data (especially for image classification or segmentation), such as amazon mechanical turk.


Plotly and Lightning are [supposedly] able to visualize extremely large data sets.


is a 4 output regression equivalent to 4 single output regression superposition?

"My question is : can I consider that all of my outputs are unlinked"

No, in general you can't, unless you have prior knowledge indicating so. There is a high chance that your outputs are correlated. To explore linear correlations, perform a correlation analysis (calculate correlation matrix). To investigate nonlinear relations, calculate the distance correlation.

  and if yes, is it true to imagine that I can perform 4 regressions in
  parallel (one for each output) to make an equivalent 4 ouput model ?

If your analysis proves that there is low correlation between the outputs, only then you can implement 4 independent regression models in parallel with meaningful results.

My suggestion would be to implement a MIMO machine learning model (multiple inputs / multiple outputs), such as a multivariate Recurrent Neural Network.

Hope it helps :)