How word2vec understands the relationship between numbers?

Attempting an answer from the understanding I have of word2vec and deep learning through my personal experience and some live presentations of Yoshua Bengio and Yann Le Cun that I had the chance to attend.

As mentioned in the question, I think too that numbers are grouped together as they are have similar functions in the sentences where they are found.

For the reason of how they get ordered, as said in my comment:
Intuitively I would say that it will be true for small numbers and less true for big numbers (the way 7-9-8 are ordered seems to go that way). When we speak we often say things like: "I ate 1 or 2 apples" or "I will buy 2 or 3 things" but rarely "I will buy 3 or 2 things". I think this is how it ends up ordering the numbers. Bigger numbers will appear less in a corpus so they will most probably not be ordered as precisely.

When looking at tutorials about word2vec, an example that is often given is that, if we see the relationship between 2 words as a vector, the final representation of the words by the model will give that the vector linking "man" to "woman" is approximately the same as the one linking "king" to "queen". Or that the vector linking "Paris" to "France" is approximately equal to the one linking "Rome" to "Italy". This even allows to do operations such as: France + Italy - Paris = Rome! The clear mechanics of how these relationships are so accurate (compared to other approaches) are, as of today, still obscur to the scientific community.

If these kind of operations works for countries and main cities, it seems logical that the vectors "3-2" and "2-1" point in the same direction, i.e. that 1-2-3 are ordered (given that, as said previously, consecutive numbers are more likely to be used together in a sentence). Of course, to achieve this, the numbers have to frequently appear in the training corpus.

I am very happy that someone showed this result with numbers because the way word2vec has been trained in most of the models that are available online did not includ digits. They were replaced by some symbols i.e., 12.34 becoming ##.##, and such observations cannot be made.

Any suggestion to improve this answer is welcome!


What type of classification algorithm for recognizing events in a set of data?

You will need your data to look something like this:

Blue Red Green Label
20   12  13    _
18   11  13    _
18   12  13    _
19   13  14    _
24  12   13    _
28  14   19    B
30  19   21    B
29  18   20    B
25  14   16    B
21  12   13    _
19  11   12    _
18  11   12    _

That is, every point in the data will need to be annotated with a label: _ for nothing, B for blue, Y for yellow.

A sequence labeling model can be trained to predict the label for each point in the input sequence. Note that it's common to use a "Begin Inside Outside" (BIO) scheme to help the model understand when an "event" starts and ends:

Blue Red Green Label
20   12  13    Outside
18   11  13    Outside
18   12  13    Outside
19   13  14    Outside
24  12   13    Outside
28  14   19    B_Begin
30  19   21    B_Inside
29  18   20    B_Inside
25  14   16    B_Inside
21  12   13    Outside
19  11   12    Outside
18  11   12    Outside

(there are also variants of this scheme)

As far as I know Conditional Random Fields would be the traditional approach for this kind of task, but there might more recent NN approaches that I'm not aware of.


How to set a class_weight Dictionary for Random Forest?

If you're just doing multiclass classification, you should specify the weights as a single dictionary, e.g. {0: 1.0, 1: 1.5, 2: 3.2} for a three-class problem. (Or use the convenience modes "balanced" or "balanced_subsample").
The list of dictionaries is used for multilabel classification (where each row can have multiple true labels).  In that case, each dictionary is for one of the outputs, the keys being the classes for that output and values the weights.


You could use the dplyr package and its case_when function. Note the spacing is just for readability:

library(dplyr)

country %>%
  mutate(
    type = case_when(
      surfacearea <  3415080                           ~ "LOW",
      surfacearea >  3415081 & surfacearea <=  6830160 ~ "MEDIUM-LOW",
      surfacearea >  6830160 & surfacearea <= 10245240 ~ "MEDIUM",
      surfacearea > 10245240 & surfacearea <= 13660320 ~ "MEDIUM-HIGH",
      TRUE                                             ~ "HIGH"
    )
  )

The last line is the catch-all case. For more see ?case_when. To complete your pipeline, you could do:

country %>%
  group_by(type) %>%
  mutate(
    type = case_when(
      surfacearea <  3415080                           ~ "LOW",
      surfacearea >  3415081 & surfacearea <=  6830160 ~ "MEDIUM-LOW",
      surfacearea >  6830160 & surfacearea <= 10245240 ~ "MEDIUM",
      surfacearea > 10245240 & surfacearea <= 13660320 ~ "MEDIUM-HIGH",
      TRUE                                             ~ "HIGH"
    )
  ) %>%
  arrange(type)

There is also the between function which could further help the readability, but that function is a shortcut for x >= left & x <= right so it might not work since your first bucket is strictly less-than.


What does the classification report interpret? Class 1 indicates abnormal data

Classification report gives a perspective of your model performance. The 1st row shows the scores for class 0. The column 'support' displays how many object of class 0 were in the test set. The 2nd row provides info on the model performance for class 1.

Recall for each class is calculated as follows:
True Positives/(True Positives + False Negatives)
E.g., recall for class 1: 15/(15+27) = 0.56

Precision for each class is calculated as follows:
True Positives/(True Positives + False Positives)
E.g., precision for class 1: 15/(15+22) = 0.41

This particular classification report shows that the performance of the model is poor. Accuracy as a metric may be misleading. Although the model shows 99% accuracy, it hardly detects the object of class 1 (I suppose these are the object of interest, the abnormal data you want to detect).

One of the reasons for that may be the imbalance of the data, i.e. there are many objects of class 0 and few samples of class 1. As a result, the classifier is overfitting to class 0 to minimize error during training. Consider reading about some techniques to overcome the imbalance problem.


One of articles which helped me a lot is : A Step by Step Backpropagation Example by Matt Mazur. It covers forward and backward pass of MLP. I hope that helps.

Another great source is http://www.deeplearningbook.org/


Voice recognition with fourier transformation with audio input in python

The vanilla version of Fourier Transform (fft) is not the best feature extractor for audio or speech signals. This is primarily due to that FT is a global transformation, meaning that you lose all information along the time axis after the transformation.

You need to be familiar with the concept of short-time Fourier transform (STFT). Basically STFT tells you what frequency components exist in your signal at each timestamp. The result of STFT (its squared magnitude, to be precise) is called the spectrogram, which is what people usually visualize. An example of spectrogram from the link above:

You may refer to matplotlib.pyplot.specgram or scipy.signal.stft regarding how to plot a spectrogram in Python.

The most commonly used speech feature (as input for neural networks) is the Mel-Frequency Cepstral Coefficients, or MFCC, which carry the similar semantic meaning as the spectrogram. Other commonly used features include PLP, LPCC, etc which you can google for more details. But directly feeding the result of FT or STFT into a neural network is not the best practice.


Replacing the feature variables with encoded variables

I don't know what is wrong with your code but this worked fine for me-

cat_var.replace(to_replace=cat_var[noms].tolist(),value=encode,inplace=True)

Instead of converting encoded values to series, convert series to list using tolist().


In a binary classification, should the test dataset be balanced?

The answer to your first question:

  should the test dataset be balanced as well?

is, like many answers in data science, "it depends."

And really, it depends on the audience for, and interpretability of the model metrics, which is the thrust of your second question:

  What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Personally, if the metrics will just be used by you to evaluate the model, I would use the sensitivity and specificity within each class to evaluate the model, in which case, I care less about the balance of the classes in the test data as long as I have enough of both to be representative. I can account for the prior probabilities of the classes to evaluate the performance of the model.

On the other hand, if the metrics will be used to describe predictive power to a non-technical audience, say upper management, I would want to be able to discuss the overall accuracy, for which, I would want a reasonably balanced test set.

That said, it sounds like your test set is drawn independently of the training data.  If you are going to balance the training data set, why not draw one balanced data set from the raw data and then split the training and test data?  This will give you very similar class populations in both data sets without necessarily having to do any extra work.


Your question doesn't describe exactly what you did so it's impossible to tell you what happens, but anyway relying on R saving the workspace is not the best option to save your model. It would be much safer to actually save the model to a file, as described in this question for instance:

> set.seed(345)
> df <- data.frame(x = rnorm(20))
> df <- transform(df, y = 5 + (2.3 * x) + rnorm(20))
> ## model
> m1 <- lm(y ~ x, data = df)
> ## save this model
> save(m1, file = "my_model1.rda")
>
> ## a month later, new observations are available:
> newdf <- data.frame(x = rnorm(20))
> ## load the model
> load("my_model1.rda")
> ## predict for the new `x`s in `newdf`
> predict(m1, newdata = newdf)
        1         2         3         4         5         6
6.1370366 6.5631503 2.9808845 5.2464261 4.6651015 3.4475255
        7         8         9        10        11        12
6.7961764 5.3592901 3.3691800 9.2506653 4.7562096 3.9067537
       13        14        15        16        17        18
2.0423691 2.4764664 3.7308918 6.9999064 2.0081902 0.3256407
       19        20
5.4247548 2.6906722


Features selection/combination for random forest

The Random Forest model in sklearn has a feature_importances_ attribute to tell you which features are most important. Here is a helpful example.

There are a few other algorithms for selecting the best features that generalize to other models such as sequential backward selection and sequential forward selection. In the case of sequential forward selection, you begin by finding the single feature that provides you with the best accuracy. Then, you find the next feature in combination with the first that gives you the best accuracy. This pattern continues until you find $k$ features, where $k$ is the number of features you want to use. Sequential backward selection is just the opposite, where you start with all of the features and remove those which inhibit your accuracy the most. You can find more information on these algorithms here.


Oh, I think I've finally got it.  It's just an averaging problem: for each fold in your k-fold cross-validation, you get perfect auROC, but at the default threshold of 0.5 your hard classifiers (for each fold) sometimes have $FPR=0$ and $TPR<1$, but some other times $FPR>0$ and $TPR=1$.  Then averaging you are able to get both $\operatorname{mean}(FPR)>0$ and $\operatorname{mean}(TPR)<1$.
To check, have a look at the cv_results_ table, particularly at each test fold scores (split<i>_test_<xyz>), rather than just the mean_test_<xyz> scores.


If I understand correctly, your question is whether if you cannot compute the gradient of the function exactly, but only within some errors, then gradient descent method still works in that it gives you convergence to a minimum point?

The answer to this is Yes, if you use Backtracking gradient descent. (And not the standard version of gradient descent, that version works only if you have some strong assumptions on the cost function, such as being convex or having Lipschitz continuous gradients, together with some other assumptions such as the function has compact sub level sets. The point is that if you consider a regularisation technique, as mentioned in the answer by Victor, then even your original function has Lipschitz continuous gradients and has compact sub level sets, after you do regularisation, the new function won't satisfy these assumptions any more.) You can find more details in the paper I mention in my answer in the following link:

Does gradient descent always converge to an optimum?


How important is advanced SQL for data science?

What do you mean exactly with competitive as a data scientist? Unfortunately, many employers will have different expectations of someone they hire to be a Data Scientist, so there isn't a single answer!

In any case, I think it is a good idea to know three components to be effective with databases:

Managing a connection: how to create and connect to a database using a simply library in Python or R. This sort of workflow should look familiar to you:

import mysql
connection = mysql.connector.connect(
                 host="127.0.0.1",
                 user="mj_whales",
                 passwd="somethingsecret",
             )

query = "CREATE DATABASE my_table"
cursor = connection.cursor()
cursor.execute(query)

Getting data: understand basic syntax of a standard SQL-style query, for example:

FROM my_table SELECT my_feature WHERE my_other_feature > 0

Writing data: understand how to insert data efficiently. This will depend on the database type e.g. document database, SQL-style, InfluxDB for time-series data.

Here is a really good introduction to databases with Python that might help you understand your current strengths and weaknesses on the subject.


Since you have more than one target value for a specific value of the feature and because the Y values are have quite a large range I guess the nueral network finds it hard to converge. Try to normalize the target values so that it has zero mean and unit variance and then try to use any of the regression models like logistic regression, A CNN with a squared error to get a prediction. After that plot the ROC-AUC to determine the dominant regressor.


This problem can be implemented as a query that minimizes "loss" from the inputs. If "mean square error" loss is used, loss calculation will look like :

select col1, col2, (col1 - inp1)^2 + (col2 - inp2)^2 as mse_loss,
from table

Now, you can find the row with least value of mse_loss


General approach to work with text of phone calls (topics, promises, sentiments, etc.)

Step by step.

Since you say its seperated into C-customer and A-agent I am going to assume its labeled (some of the questions could be solved unsupervised also)

Who is the customer and who is the agent?
Simple binary classification. Find suitable numerical model representation of text and perform classification. Starting reference
Customer Name Named entity recognition - NER example how to use it:

  import nltk

from nltk.tag.stanford import NERTagger

st = NERTagger('stanford-ner/all.3class.distsim.crf.ser.gz', 'stanford-ner/stanford-ner.jar')
text = "blablabla"

for sent in nltk.sent_tokenize(text):
    tokens = nltk.tokenize.word_tokenize(sent)
    tags = st.tag(tokens)
    for tag in tags:
        if tag[1]=='PERSON': print tag`

The topic of conversation Start with these two approaches: LDA and LSA
Promises made by the operator to the customer (for example, "I call back tomorrow") General approach (its plausible, needs a bit of work by you) But lets say you use word embeddings, all of the sentences that have promises will be vectors that are close to each other in this high-dimensional space. You can limit your system to return all sentences that fall into this cluster in the space.
Negative Sentiment (if there is something in the conversation that the subscriber is not happy with) Simply just look at this introduction to Emotion AI Nutshell: You already have pre-trained models that can model positive-neutral-negative sentiment in the given text. Use the recources.


Is Stratification applicable to both Classification and Regression?

Yes that makes sense. Your training and testing datasets should both be similar distributions to the whole dataset in both classification and regression, and in the regression case, binning the target variable is one way to achieve that. You need to make sure that you choose good bin sizes -- you can completely skew how the distribution looks in a histogram based on the bin sizes. Too small and you only have one or two examples per bin, resulting in a very erratic histogram. Too large and you lose a lot of information about the shape of the distribution.

Binning is just one way to approximate the density function of the distribution. Depending on how your data looks, you might be able to fit a Gaussian (or any other distribution) curve to your target variable and use that instead to sample your train/test split.


After training and saving a model can we give more information as input?

Yes, by customizing the model with new information. But, you have to run atleast one training, with complete cycle and export output.
the general example from tensorflow is here


Improve NER label results on Non-English text

One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.

The idea is that you train a neural network to perform different NLP tasks. For example:

Translation
Part-of-speech tagging
Named entity recognition
Chunking

or any other task.

However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.

The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.

So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.

This should take care of your lack of NER data in Latin.


Forward propagation in complicated neural networks

You can do topological sorting. For example, here is some pythonic pseudocode

def order(current_neuron, level):
    current_neuron.level = max(level, current_neuron.level)
    for child in sorted(current_neuron.children, key=children level, reverse=True):
        order(child, level + 1)

for input_neuron in input_neurons:
    order(input_neuron, level=0)

(Take this pseudo code with a grain of salt. This is just what came directly to my mind; it is not tested.)

Then you can calculate the activations in order of node.level.


Scientific name for the problem given in the description. (In machine learning what kind of problem it is termed as)

Ok. I think I'm getting a clearer picture. Firstly any name isn't going to be different just because the data is large. Of course in the big data community anyone is free to coin a term and time will show if it gets adopted (same goes with what we might consider a scientific name for that matter too).

Another thing is input output. These terms can be confusing as what is input by a comp scientist into a training model is both the input and output of what a statistician would define.

I'm not aware of a name for the problems but names for models addressing such a problem include logistic regression and probit regression. Both deal with a(n) (set of) independent variable(s) - the feature name in your case - and a resultant dependent variable - the binary relevancy. The regression looks at fitting a linear function to a hidden variable; whose threshold gives the binary result. Fitting an "input" training set then allows the scoring based on this linear function, whether you want a binary score or a real valued one (from the hidden variable) that will be more indicative of the probability of your prediction.


City pollution data for data mining project

The second module of this free machine learning class revolves around working with geospatial pollution data for a river in the Netherlands:

https://www.edx.org/course/applied-machine-learning-microsoft-dat203-3x-1


If I understand your question correctly (please comment if I don't), you want to compare the output text from two different language models. Therefore, I don't think you should worry too much about the language models themselves.

If I was you, I'd probably do a simple TFIDF analysis so that you can gain a better understanding of what terms are more prevalent in certain news outlets.

Take a look at scattertext, it allows you to create visualizations like the following:


Decision tree, how to understand or calculate the probability/confidence of prediction result

What data mining package do you use?

In sklearn, the DecisionTreeClassifier can give you probabilities, but you have to use things like max_depth in order to truncate the tree. The probabilities that it returns is $P=n_A/(n_A+n_B)$, that is, the number of observations of class A that have been "captured" by that leaf over the entire number of observations captured by that leaf (during training). But again, you must prune or truncate your decision tree, because otherwise the decision tree grows until $n=1$ in each leaf and so $P=1$.

That being said, I think you want to use something like a random forest. In a random forest, multiple decision trees are trained, by using different resamples of your data. In the end, probabilities can be calculated by the proportion of decision trees which vote for each class. This I think is a much more robust approach to estimate probabilities than using individual decision trees.

But random forests are not interpretable, so if interpertability is a requirement, use the decision tree like I mentioned. You can use grid search to maximize the ROC AUC score by changing hyperparameters such as maximum depth to find whatever decision tree gives the most reliable probabilities.

EDIT: In case I wasn't clear enough, I think it's awful to use a single decision tree to predict probabilities. I have extended my answer in a blog post.


There is only one neural network to train in word2vec. CBOW, continuous-bag-of-words', and skip-gram are two different methods of constructing the training data for the neural network. You have to pick one of those two training methods.


Kernel based methods work by constructing a feature vector based on some distance metric of an input vector to one or more example vectors, which are determined during training.These are the prototypes in your case (also known as regressors in some contexts). They are essentially vectors in the space of the independent variables.

Radial Basis functions essentially "convert" the (Euclidean) distance between input vector and prototype to a similarity value, usually  [0,1]. Some of them, such as the Gaussian, have parameters such as the bandwidth.

In RBF Networks, there are a number of different options of determining the prototypes, for instance by Clustering at the space of independent variables, or by Orthogonal Least Squares (references at the bottom).

Like you correctly assumed, the first sentence is explaining that in many cases of training algorithms (the ones mentioned above included), the bandwidth of the RBF is constant and determined a-priori What is subject to determining is the actual location of the prototypes. In this context, the prototype would refer to the value composition in the input space, but not to the radius (which is fixed anyway).

I have not read the paper in question, but it seems that the authors are proposing some method to individually adjust the bandwidths of the RBFs. In this case, the radius may have been included as well into the notion of the prototype, but it is specific to the research in question.

References

J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.

S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.


Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow

To Improve the performance of the network, I added a Batch Normalization layer, and other variables that I had to remove when using Random Forest.  One of these variables is a categorical variable with 51 levels.  I made sure to add an embedding layer that will represent the values by a vector of size 10.
As the original dataset had 15 numerical variables and 9 categorical variables, I have decided to also perform dimension reduction using auto-encoding.


Can a recommendation system be built without any user ratings?

This is a common issue for recommender system refered to as cold start. You can look it up there are some solutions suggested out there but I have never tested them out. Have a look at this paper:

http://www.cs.toronto.edu/~mvolkovs/nips2017_deepcf.pdf


What you have is a classification problem. Given a user and their features, pick the best content for him.

There are many classifiers that are for this task. I would say that your main problem is that most of your features are categorical and not numerical.

Data preparation

With Sklearn for example, you can use LabelEncoder to transform categorical to numerical.

Below is an example of how to do this. You generate X, which is your features matrix. Each row is a user and the columns are the features.

Here are some dummy users I created. You data structure might be different, so you can try to make it like this, or do some more work on it.

users = [
    {
        'id': 1,
        'nationality': 'american',
        'company': 'xyz',
        'role': 'ceo',
        'content.id': 1001
    },
    {
        'id': 2,
        'nationality': 'american',
        'company': 'abc',
        'role': 'cto',
        'content.id': 1002
    },
    {
        'id': 3,
        'nationality': 'canadian',
        'company': 'fgh',
        'role': 'cto',
        'content.id': 1001
    }
]

Now, you can encode your categories. This means that for every word in your data, you assign a number to it. The LabelEncoder can do this for you. With this, you generate X, which is your feature matrix, where every row represents a user.

from sklearn.preprocessing import LabelEncoder

nationality_encoder = LabelEncoder().fit([user['nationality'] for user in users])
company_encoder = LabelEncoder().fit([user['company'] for user in users])
role_encoder = LabelEncoder().fit([user['role'] for user in users])

X = []
for user in users:
    X.append([
        nationality_encoder.transform([user['nationality']])[0],
        company_encoder.transform([user['company']])[0],
        role_encoder.transform([user['role']])[0]
    ])
print(X)
>> [[0, 2, 0],
>>  [0, 0, 1],
>>  [1, 1, 1]]

Then you can generate y, which are your labels. The content.id of each user. Here I assume you have the user-content pairs like you said you did. This vector has the same length as X. Each row of X is associate to the ordered values in y. So the user in X[2] has the content.id in y[2].

y = [user['content.id'] for user in users]
print(y)
>> [1001, 1002, 1001]

Classifier

Now you have your X and y, your features and labels.

Not all classifier will perform well with categorical data, but I would suggest you start with a DecisionTreeClassifier.

Here is an example:

from sklearn.tree import DecisionTreeClassifier

decision_tree = DecisionTreeClassifier()
decision_tree.fit(X, y)

New predictions

Now that you have a trained tree, you can predict the content.id for users that do not have it.

Suppose you have this new user:

new_user = {
        'id': 999,
        'nationality': 'american',
        'company': 'fgh',
        'role': 'ceo'
    }

You can apply the same transformation to get the X for him. However, now you do not know y, which is what you are trying to find out.

new_user_X = [[
    nationality_encoder.transform([new_user['nationality']])[0],
    company_encoder.transform([new_user['company']])[0],
    role_encoder.transform([new_user['role']])[0]
]]

Use the decision_tree you created to make a prediction. In this dummy example, the new_user should have content.id == 1.

new_user_content = decision_tree.predict(new_user_X)
print(new_user_content)
>> [1]


Using pandas, the nullity correlation matrix seems to be obtained by df.isnull().corr() (this is how it is done is missingno), and this makes sense.

missingno package also states, in the heatmap function documentation, that for large datasets the dendogram view is better. However, it does not tell me if "large" means many features or entries.


In CNN, why do we increase the number of filters in deeper Convolution layers for complex images?

For this you need to understand what filters actually do.
Every layer of filters is there to capture patterns. For example, the first layer of filters captures patterns like edges, corners, dots etc. Subsequent layers combine those patterns to make bigger patterns (like combining edges to make squares, circles, etc.).
Now as we move forward in the layers, the patterns get more complex; hence there are larger combinations of patterns to capture. That's why we increase the filter size in subsequent layers to capture as many combinations as possible.


How to update bias and bias's weight using backpropagation algorithm

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer.

There are usually two different approaches taken when implementing bias. You can do one or the other:

As a separate vector of bias weights for each layer, with different (slightly cut down) logic for calculating gradients.
As an additional column in the weights matrix, with a matching column of 1's added to input data (or previous layer outputs), so that the exact same code calculates bias weight gradients and updates as for connection weights.

In both cases, you only do backpropagation calculation from neuron activation deltas to the bias weight deltas, you don't need to calculate the "activation" delta for bias, because it is not something that can change, it is always 1.0. Also the bias does not contribute deltas back further to anything else.


Is it suitable to change a feature by itself to generate an another feature?

Sure, that's feature engineering.

If you're fitting a linear model, then you are looking for features that have a linear relationship with the predicted value. If you're predicting, say, the cost per hour of a device consuming current I, then clearly that's directly related to power not current, so $I^2$ is more likely to be useful.

What you want to be careful about is trying a bunch of transformations of the input blindly; it's possible for a small data set that one odd function of an input happens to be predictive in that sample, but doesn't generalize.


do autoencoders work well for non images?

Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are lots of parameters, and not every configuration will work well. Even the optimization algorithm is a hyperparameter.


Is it possible to create a neural network with two inputs, with sequential layers?

Sure!  There's all sorts of ways you can pass in two inputs.  Your idea of stacking the inputs like a 2-channel image is one idea.

Another possibly simpler idea would be something like this:

The intuition is that the network learns intermediate representations for the two sentences, then compares the representations.

Of course, you have an enormous amount of flexibility in how you configure the network connections. There are endless variations that allow you to have two (or more) separate inputs.  For another example, check out DeepMind's pysc2 paper.  On page 10 you can find diagrams for networks that deal with three different inputs.  There's a flat vector of non-spatial features along with two different spatial inputs.


Whenever you have skewed dataset, it means that you know some classes better than some others. In such cases it means that the data is your knowledge and there are learning algorithms for such occasions.

Consider an important fact here. Suppose that you have feature vectors of conditions of a nuclear company and they describe whether the company is in danger of nuclear radiation or not. In such case it is clear that it does not happen a lot that you have infected companies so most of your data have label of healthy condition. You have so much knowledge about the healthy class but you don't know much about the infected class because you don't have much data consequently you don't know its distribution and you can not estimate it well. Whenever your data is skewed, it means that e.g. you have 1 million feature vectors of negative class and 5 feature vectors of positive class. Now suppose that you change the feature vectors. In such cases you have imbalanced data-set or you just have the data samples of some classes without some other, you can use anomaly detection.

  In data mining, anomaly detection (also outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset. Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions.


Target encoding with KFold cross-validation - how to transform test set?

I have some doubts about the way how I should then encode test set. As there is no single mapping deduced from train set I think we should use the whole train set to fit the encodings and then use it on test set

Yep, that seems fine, they way that you do it there its a bit more complicated than using a pipeline. The idea of splitting into train and test is mimicking how the model will behave in production/unseen data. Doing target encoding with the test, is doing data leakage and getting a miss representation of how the model will behave in production. So you get the target values in train and then move to test.
If you do this, and then you have a category in test that is unseen, it will through an error. If you have a look at the target encoding library of category encoders, you can deal with this.:

handle_missing: str
options are ‘error’, ‘return_nan’ and ‘value’, defaults to ‘value’, which returns the target mean.

You can handle it in different ways, the best is depending in your problem. The default is returning the target mean.
They best practice to do is to create a pipeline where the target encoding is a step(transformer). This will allow you to do CV, evaluate your model on test and many other functionalities. (Here a tutorial on how to)
A code snippet:
import random
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from category_encoders.target_encoder import TargetEncoder
from category_encoders.m_estimate import MEstimateEncoder
from sklearn.linear_model import ElasticNet,LogisticRegression

random.seed(1234)
y = random.choices([1, 0], weights=[0.2, 0.8], k=100)
cat = random.choices(["A", "B", "C"], k=100)
df = pd.DataFrame.from_dict({"y": y, "cat": cat})

X_train, X_test, y_train, y_test = train_test_split(df[["cat"]], df["y"], train_size=0.8, random_state=42)
skf = StratifiedKFold(n_splits=5)

clf = LogisticRegression()
te = TargetEncoder()

pipe = Pipeline(
        [
         ("te", te),
          ("clf", clf),
        ]
    )

#Grid to serch for the hyper parameters
pipe_grid = {
    "te__smoothing": [0.0001],
    }

# Instantiate the grid
pipe_cv = GridSearchCV(
        pipe,
        param_grid=pipe_grid,
        n_jobs=-1,
        cv=skf,
    )

pipe_cv.fit(X_train, y_train)

# Add some unseen category to the test.
X_test['cat'] = 'UUUUU'

pipe_cv.predict(X_test)

Note that the code is not optimal but it should show you how to deal with this problem of doing target encoding with the train and test using a pipeline, and working with unseen data :)
Note that the category has been assigned randomly. So the model detects that the best is predicting the most frequent class. If you change for ElasticNet (a regressor) you will get the mean.
If you take out the unseen category assignation to test you will still get the same results


Sliding window Algorithm and its convolutional implementation

Consider the second row ie. the 16x16x3 example, if you do the sliding window approach you first take the 14x14x3 area marked in red and pass it as input to your trained CNN. What happens then, 5x5x3 filters then convolve the first 5x5x3 of the input ie. starting from top left and then it convolves on the next 5x5x3 section of the same input according to the stride, and so on, just as Convolution happens.

Now, consider the 14x14x3 section marked in green, (this will be the second window in Sliding Window Approach) and it like the first red area, goes into your trained CNN and hence is convolved by the same filters that convolved the red one, so far so good.

What will be the first part of this green input to be convolved? It would be the same as the second 5x5x3 part to be convolved on the first 14x14x3 green area, so what happens is that the same 5x5x3 area has undergone the same operation ie. same convolution in this case twice, that means you've basically done the same computation again on the same 5x5x3 input, once on the red input and then on the green input.

What's the use of doing the same calculation again, that's the reason we use the Convolutional Sliding Window Approach which will do every computation only once and since you aren't doing the same stuff again and again you will save that time and hence the Convolutional Sliding Window Approach will run faster and obviously does the same computation as the normal Sliding Window Approach.

Note: Modern implementations of CNN are very deep and involve lots of filters, so you are doing a lot of redundant work which slows down your implementation.

Hope it helps.


Predictions with arbitrairy sequence length for stateful RNN (LSTM/GRU) in Keras

I found a possible solution: You can save the weights, create a complete new model and load the weights again.

Original Stateful Model

modelinput = Input(batch_shape=(num_batches,None,num_features))
prediction = GRU(10,return_sequences=True,stateful=True)(inputs)
model = Model(inputs=modelinput,outputs=prediction)

Save Weights after Training

model.fit(...)
model.save_weights('weights.h5')

Create New Model (same structure, just not stateful and dynamic batch size)

modelinput_pred = Input(batch_shape=(None,None,num_features))
prediction_pred = GRU(10,return_sequences=True,stateful=False)(inputs_pred)
model_pred = Model(inputs=modelinput,outputs=prediction_pred)

Load Save Parameters and Predict

model_pred.load_weights('weights.h5')
model_pred.predict(...)

Nevertheless I hope for a better solution through sharing weights between a stateful GRU and a standard GRU. Is that possible?


One-hot & interaction one-hot on multiple categorical

Tree models will be able to figure out the Interaction whether Label Encoded or OHE.
If you separately create all the combinations of Category values as new Features, then there is nothing left for interaction (At least 1st level).
e.g. let's take the example Features
P -  {X,Y}
Q  - {Z,W}
Let's assume, P and Q have interaction on X and W.
 If we Label encode - We will observe multiple consecutive splits on P==X and Q==W.
 If you OHE - you will observe multiple consecutive splits on Is_P_X and Is_Q_W (since both are separate feature now).
In these cases, you will have to observe the splits to see the interaction or can use a partial dependence plot or similar approaches.
 If you Create features on all combinations -
In this case, you will observe very high Feature importance for feature X_W that is actually because of interaction but you will not observe it as consecutive splits on two feature but a split on single feature i.e X_W
In these cases, you can observe the interaction with Feature importance.
On a dummy data where I added the interaction effect on output for Sunday and Breezy weather.
Image 01 - Showing Interaction in the splits

Image 02 - Simply splitting on Sun_Breezy when every combination is a separate feature


How is a coincidence matrix constructed for computing Krippendorff's alpha?

The formula in the Wikipedia article seems to be limited to two coders. Look at the sum of values in u for m coders, i I like j, so each value in u is to be matched with one other value in u.  Actually, this sum would pair values of i with values of j and values in j with values i. So, o sub aa would be 2 not 1.  And o sub bb would be 4, not two.

However, I don’t see how Wikipedia’s formulation could yield coincidence matrices for more than two coders as this would require two sums which pair each of m valors with each of m-1 other values.


Can features negatively correlated with the target be used?

Negative correlation is not the same as low correlation.
If variables $x$ and $y$ have a correlation value of $c$, then $-x$ and $y$ will have a correlation of $-c$.
When people talk about "low correlation" they usually mean correlation that is close to 0.
In the context of machine learning, negative correlation is just as good as positive correlation; do your filtering based on the absolute value of the correlation.


Autoencoder doesn't learn to reduce dimesions

What kind of color are you trying to learn? If it's a random color, then yep, tough luck, you can't reduce the dimensionality of a 3D Euclidian manifold to less than 3D.

To be more precise, this is the case for an infinite manifold, for non-infinite one, you could get a 1D curve that goes through a 3D cube (in philosophy, it would be close to a Kohonen map, despite having just one node instead of a set of discrete nodes), but you would need a very big set of intermediate layers to make it happen.


How to calculate coverage in recommender systems?

Right, the capital letters denote the total available. In that blog post: U means all users, N means all items but in other places is usually written I, and L means all top-n recommendation lists. "Top-n" means that the recommender system outputs a ranked list of n items, so if you had 1000 users all getting a Top-10 list, you'd have L length of 1000*10.

I suggest you read Ge, Mouzhi, Carla Delgado-Battenfeld, and Dietmar Jannach. "Beyond accuracy: evaluating recommender systems by coverage and serendipity." Proceedings of the fourth ACM conference on Recommender systems. ACM, 2010, at http://ls13-www.cs.tu-dortmund.de/homepage/publications/jannach/Conference_RECSYS10b.pdf instead. It's a more clearly written definition of metrics and uses more traditional recommender system notation.


why should i do target encoding within cv loop?

From the performance point of view, you can one-hot encode the whole dataset before the model fitting and you will get identical results.
Why people do it in the CV loop?
Ideological perspective
In the CV, we set the part of the data aside and run the model on the given dataset from scratch. For example, you may want to normalize the database, but minimum, maximum, mean, and std are dependent on a particular fold, and, therefore, should be done inside the CV loop. Therefore, preprocessing is inside the CV loop, and so one hot encoding belongs here.
Sparse categorical variables
Suppose that the output of your model is a categorical random variable with tons of categories. For example, your output is a single word that described the image, and there are thousands of classes. Instead of one-hot encoding and getting tons of columns of almost all zeros, you may want to use sparse categorical variables, i.e., to map the target or a feature to integer values. For simplicity of the argument, suppose that a categorical feature is encoded as a sparse variable. Then, the values of the encoded column are 0,1,2...,n and for the linear regression, the order matters. In other words, 2 is closer to 0 than to 100. To make sure that your model performs well regardless of encoding order, you need to check cross-validation.


Pre-process data images before training OneClassSVM and decrease number of features

You should try converting them to Principle Components using PCA. Please refer this Analytics Vidya PCA, this should give give you a good understanding.

PCA converts n large vector into p Principle components.

  First principal component is a linear combination of original predictor variables which captures the maximum variance in the data set

Second PC is also a linear combination of original predictor variables which captures remaining variance. All the succeeding ones follow the same concept.

This way you can select top Principle components which explains good enough cumulative variance in your data


Does it make sense to randomly select features as a baseline?

Your supervisor is asking to use random feature selection as a baseline. Random performance is a common baseline in machine learning. For example in binary classification with equal numbers of samples from each group, a trained classification should perform better than 50% (random guessing).


Does logistic regression can only solve binary classification problem?

No, multiclass classification is also possible. Try reading up on 'One vs All' multiclass classification.

https://en.wikipedia.org/wiki/Multiclass_classification

Multiclass, One vs All classification in the case of logistic regression:

https://www.coursera.org/learn/machine-learning/lecture/68Pol/multiclass-classification-one-vs-all


Stanford CoreNLP is a very good implementation of CRF (In Natural Language Processing domain).

https://nlp.stanford.edu/software/CRF-NER.html  . CRF specific implementation is : https://github.com/stanfordnlp/CoreNLP/blob/master/src/edu/stanford/nlp/ie/crf/CRFClassifier.java

Few other resources :

https://www.analyticsvidhya.com/blog/2018/08/nlp-guide-conditional-random-fields-text-classification/

https://github.com/lancifollia/crf

https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463


Where can I find a trained neural network data to play with?

What about the super popular and classical MINIST? Tensorflow has a nice tutorial:

  https://www.tensorflow.org/get_started/mnist/beginners

Just follow the instructions!


Properties for building a Multilayer Perceptron Neural Network using Keras?

1) Activation is an architecture choice, which boils down to a hyperparameter choice. You can make a theoretical argument for using any function, but the best way to determine this is to try several and evaluate on a validation set. It's also important to remember you can mix and match activations of various layers.

2) In theory yes, many random initializations would be the same if your data was extremely well behaved and your network ideal. But in practice initializations seek to ensure the gradient starts off reasonable and the signal can be backpropagated correctly. Likely in this case any of those initializations would perform similarly, but the best approach is to try them out, switching if you get undesirable results.


Can Gaussian dist. use median/MAD instead of mean/SD?

I am not 100% sure if I got your question, but I will try my best.

By looking at the gaussian probability density function and inspecting the parameters you will see, that there is actually no mean at all. There is only the expected value $\mu$, which is not necessarily the mean of your data, and $\sigma$, which depends on $\mu$.

When you have data and want to fit it to a gaussian, you will see by applying maximum likelihood, that using the mean is quite the best thing you can do to fit a gaussian.

Ideally your mean and median should be close. If not, there are two options. First your data is simply not gaussian. Second you have outliers or errors in your data. Then you can think about cleaning your data first or as a lazy and fast solution take the median.
Otherwise I see no justification to take the median.


Fewer observations & larger documents vs More observations & smaller documents

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

The first thing you need to think about in any ML problem is: what is an instance for the problem? In other words, what is going to be the input for which you want a prediction at the end of the process?

Imagine you train your model with 3 big documents, one of each label. Then the input for such a model is a big set of documents with the same label. So it can only predict a label for a set of documents sharing the same label. This means that somehow you need to have the labels before applying your model... difficult isn't it? :)

This is why in this case an instance must be a single document. It's the job of the learning algorithm to learn to discover the label based on the instances, and for that it needs many instances of each possible label.

  On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

This is where there is a confusion: the TF-IDF weights are not supposed to encode the label in any way, they represent the importance of a particular word in a document. The learning algorithm will use this information for all the words, that is it's going to learn the difference between when the word delicious has a high TF-IDF and when the word disgusting has a high TF-IDF (for instance).

  Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

That would be true as well, but the main issue is the one I mentioned above: you won't be able to provide the same kind of input when you apply your model on your unlabeled data.


Size Issue when Plotting the Predicted Vaue

You are using the full X dataset and want to plot it with the y_predict values, this is not possible since the size of both arrays is not the same. y_predict are the predicted values on X_test while X contains all inputs (i.e. X_train and X_test). You should therefore use X_test instead of X.


Delete rows of a column using another column statistics in R

You can accomplish this using a logic index as follows:

us_contagious_diseases<-us_contagious_diseases[!(us_contagious_diseases$weeks_reporting<"10"),]


Creating a new Dataframe with specific row numbers from another

So, your question is to instantiate a new data frame df2 from another data frame df1, by simply selecting rows.
You can do this by indexing. What is great by pandas DataFrames is that you can index a DataFrame using a list of indices.
df2 = df1.iloc[[list of indices],:]


How to use External Data Sets in test set

The general idea is that you should use the same data on your train, test and prediction sets. For economical data this can be tricky :

In your train set, you need to get what data was available at a given time. This can be tricky because most of the time such economical data can be revised a long time after being first published. For example in my country, I think GDP can be revised up to 3 years after initial publication. Sor for exemple, if you have an instance in 2015 in your train / test set, you need to get what data was available in 2015, not revised numbers that could happen up to 2018.
In your test set, and for general prediction, the actual value can be an estimation of what is currently happening (nowcasting), so you have to make sure that you data source are consistent. Nowcasting techniques are sometimes used to get the current values (for exemple to get an idea of the activity of the hotel sector you can look at booking.com # of reservation ... etc.). Those techniques are entirely different from what is generaly used to get final numbers (agglomeration of fiscal declarations). Ideally you have to make sure the methods are consistent between, train, test and prediction sets. Using other methods to get data for prediction should not be intirely excluded, but should come with very strong warnings.


Cosine Similarity but with weighting for vector indexes

Cosine similarity won't work very well because it's only based on whether the rank at position $i$ is the same in vector 1 and vector 2.
For instance the vectors [3,2,4,1,5] vs [2,3,5,1,4] will have very low similarity because 4 positions out of 5 are different, even though there are only two swaps between (2,3) and (4,5).
A much better way to measure the similarity between two rankings is Spearman Rank Correlation.
Note that (if I'm not mistaken) in this case you could also directly use Pearson correlation, since the numeric values are already ranks. Spearman just ranks the values before applying Pearson correlation. So normally the two will give you the same result.
Note also that this method won't give more weight to the top of the ranking than the bottom. I'm not aware of a measure which does this, other than defining a custom weighted measure.


There are many places:

this one is very good: https://relational.fit.cvut.cz/

there are also:

https://datahub.io/search

https://www.reddit.com/r/bigquery/wiki/datasets


xgboost and linear regression new feature analysis

There is nothing that needs to be preconsidered when a new feature is presented to LightBGM or XGBoost.

Even though the two mentioned methods work are nonlinear methods, there is no restriction on use them, the method itself will select or not the new features.


Does anyone know how google play music's recommendation engine works?

Deep neural networks, probably with tens of internal layers, distributed across many many machines. Each user probably receives scores from many deep networks for various applications, and each network is probably automatically scored to determine when its particular model is most helpful. This is the pattern used across the industry. Music companies have the additional complexity of trying to parse sound data. Spotify does this with their Release Radar recommender.

As far as targets go, they probably build models to predict all forms of engagement. You probably get scores for listens, scores for time listening, scores for thumbs up or down, scores for add-to-list, and so on. Ensemble methods work well in practice and Google probably gives you on the order of dozens to hundreds of different machine-learned predictions.


Improve the speed of t-sne implementation in python for huge data

You must look at this Multicore implementation of t-SNE.

I actually tried it and can vouch for its superior performance.


Reinforcement learning: easily learnable state representation

Does state representation generally affect how difficult a problem is?

Yes, due to it being easier or harder for a neural network to learn the relationships between input features and the target policy or value function.

  Is this a poor state representation?

No, it should be fine, provided you are interested to see a RL learn to convert from cartesian coordinate space to solving this problem, you should be able to use the state directly as ML features.

It could be made into better features for a NN in two ways:

Scaling to fit in ranges with zero mean and limited max absolute values.
It could be engineered so that features include domain knowledge of the problem to be solved.

The first part is optional for you, but I think that the bearing values benefit a little from being scaled and centred. In a more general case, even if the state description was complete, you may still want a step that re-scales it.

The second part is tricky - yes you could find better features, but part of the challenge is to create an agent that learns to do this itself. In this case, you may want to gain experience training agents where it is less easy to engineer "golden" features, and focus on RL methods.

  Are there rules of thumb for how to design states?

Conceptually, the problem of state representation can break down into three parts:

Observations. Raw observations are not always direct candidates for a state representation. In a toy problem, often this is ignored and you feed data from a simulation that looks useful as a state. In the real world you can be limited by what is detectable.
State description. Unless you want to explore POMDPs then you usually want the state description to possess the Markov property. This might already mean processing observations into something else - e.g. using history of last 3 observations, keeping running totals or calculating differences.
Feature vector. Once you have decided to use function approximation for calculating action values or a policy function, then your inputs need to conform to how the function approximation works. For most function approximators, this means numerical values. For neural networks, it means scaling inputs to fit relatively small ranges. There is also the question of feature engineering using domain knowledge of the problem.

There is some overlap between these design steps, the distinctions are somewhat artificial. When you are designing a test problem like yours, you may find it simple to combine all three steps into a single observation = state = feature. However, in real-world problems, each of the steps may require some consideration.

You should also consider the likely nature/shape of the function that you will be approximating. Action-value methods like Q learning need to approximate their value functions, whilst policy gradient methods like REINFORCE and PPO need to approximate policies. Sometimes the map between input features and the target function is simple, and if you are lucky some intuition can lead you to figuring that out. This is also a big driver when choosing between DQN or PPO for example - what seems easier to figure out, the correct action, or the value of a state/action pair?

  Would it help to reformulating the state, eg. to [distance_from_agent_to_rabbit, angle_between_agent_and_rabbit]?

Maybe. The angle feature that would seem to most help is the difference in angle between the wolf's bearing and the vector between the wolf and rabbit. Then the correct action would map very clearly from that to steer the wolf towards the rabbit - in most cases a negative value means steer right and a positive value means steer left.

However, if you do this, you will in some ways have changed the nature of the problem to be solved. You have to ask, are you interested in applying your domain knowledge of the problem to help the agent, or are you interested to see if the agent can figure out an internal representation that discovers this relationship? For a toy problem, you may want to deliberately make something harder to learn.

  I am having trouble solving this environment. Agents trained in it get scores only slightly better than a random agent even after long training sessions. I have tried Deep Q-learning (with experience replay, target network) REINFORCE (with and without baseline) and PPO.

As discussed in comments, the biggest factor towards this actually turned out to be a bug in your environment code, where the agent could end up with a bearing value out of range. The environment still worked, because you are using trig functions to calculate movement, but this causes even larger range of bearing values plus makes states that are identical appear different to the agent making things even harder to learn.

An experiment

I managed to solve this environment using a simple single-step DQN-based agent, and had time to experiment with some different input features to the NN to demonstrate my points above. In each case, I used exactly the same hyperparameters for RL and NN (expect in the last case I had to change the size of the NN's input layer).

I counted the number of training episodes required (including ~80 episodes of purely random behaviour to start experience replay) for 100 completed training runs, and tried some different state feature representations. I counted the environment as "solved" when 100 test runs using the agent acting greedily scored an average return of 20 or more. I did not count the test runs towards the number of training episodes.

I got these results:

Unaltered state 629.99 +-23.66 episodes, but failed in 28 out of 100
Scaled state 577.78 +-19.41 episodes, no failures
Engineered state 153.84 +-3.26 episodes, no failures

There is not quite a significant difference in 100 trials between number of episodes for unscaled and scaled state. However, the high number of failures (gave up after 1500 episodes of training) for unscaled features is a significant difference.

For the scaled state variation, The scaling for the cartesian coordinates was $f_i = 2*(s_i-0.5)$ and the scaling for the bearings was $f_i = 0.5 * (s_i-\pi)$

The engineered state used [distance between wolf and rabbit, angle between vector to rabbit and wolf's bearing, rabbit's bearing] scaled as above, and the performance was radically better.

Another consideration

As an aside, it is worth mentioning episode timeout and "done" flag.

You need to carefully consider what it means for the episode to time out. There is a difference between this being part of the challenge (to succeed within a time limit) and being for training convenience (to avoid wasting time learning from overlong or stuck episodes). Sometimes it is a big difference:

For training convenience. The "done" flag is an annoyance here, you want to avoid claiming the episode is really over to the agent, as it will falsely learn that some states, sometimes, end the episode - these states may then even be desirable to the agent as they seem like they stop the flow of negative reward. You don't want to store that white lie in the experience replay table - a simple work-around is to have your agent stop prematurely at least 1 step before the environment times out.
Part of the challenge. If the environment can really time out, and not randomly stop at any point, then in order to preserve the Markov property, you must include a representation of the time in the state - it can be time so far or remaining time. Otherwise you have turned the problem into a POMDP, and added complications for calculating value functions.


Pearson's correlation is the default correlation used with Pandas corr method.

Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It makes no sense to say if categorical_var1 is increased by one , categorical_var2 also increases by X ( X's value depends on the correlation between the 2 variables ).

That's why you only see numerical variables! There are other statistical tests you can apply to categorical variables to better understand them.

Note : some columns may appear as numerical at first glance, but a string may be there due to an input mistake, or simply when the formatting of the file was done, that column type was set to 'Object'. Make sure to test the values in your supposedly numerical columns and apply astype to set them back to int or float


My accuracy changes throughout every epoc but the val_acc at the end of each epoc stays the same

Your model complexity is high enough to learn features and give you results better than 20%. Although I would suggest to increase your learning rate to 1e-3 or 1e-4.

The accuracy in epoch refers to accuracy for the batch its training on so it might fluctuate. But is val accuracy and loss is same that means the model isn't learning distiguishable features. It could mean the problem is hard.


How to change a cell in Pandas dataframe with respective frequency of the cell in respective column

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randint(2,size=(10, 4)), columns=list('ABCD'))
df

    A   B   C   D
0   0   1   1   1
1   1   0   1   0
2   1   0   0   1
3   0   1   1   1
4   0   0   1   1
5   0   1   1   1
6   0   0   0   0
7   1   1   1   0
8   1   1   1   0
9   1   0   1   0

values = df.apply(pd.value_counts)
values

    A   B   C   D
0   5   5   2   5
1   5   5   8   5

new_df = pd.DataFrame()
for x in df.columns:
    new_df[x] = df[x].apply(lambda row: values[x][row])
new_df

    A   B   C   D
0   5   5   8   5
1   5   5   8   5
2   5   5   2   5
3   5   5   8   5
4   5   5   8   5
5   5   5   8   5
6   5   5   2   5
7   5   5   8   5
8   5   5   8   5
9   5   5   8   5

I created a df with random integers 0 and 1.

Then count them by column in values df.

Then loop through each column and replace each cell with their respective count. Being random it is close to 5/5 split, but you can see the C column has 8/2 split.


Algorithm for backpropagation through time

As far as I can make out, this is simply the error aggregated at layer_1, meaning it is the sum of the errors from the current hidden layer, plus the error from all future hidden layers (in this case, that is only one layer: layer_2) plus the error from the output (prediction) layer - sigmoid_out.

I think that line of code (#99) is performing the loss update with respect to a chosen layer and all layers that are ahead in time - so closer to the network's output layer, but as we are moving backwards during backpropagation. It corresponds then, I believe, to equation 3.24 and 3.25 in Ilya Sutskever's PhD Thesis (on page 35).

So you are summing up the gradients of all layers ahead of the current layer, as well as the gradient of the output layer (the sigmoid). Equation 3.25 from the link above looks like this:

$$
\frac{\delta L}{\delta W} = \sum_{t=1}^{T-1} \bigg( W'^T (r_{t+1} \odot (1 - r_{t+1}) \odot \frac{\delta L}{\delta r_{t+1}}) \bigg) v_t^T +
\sum_{t=1}^{T-1} \frac{\delta log \hat P (v_t | r_{t-1})}{\delta W}
$$

  ... where , where $v_t$ are the input variables and $r_t$ are the RNN’s hidden variables (all of which are deterministic). (from page 34)

The explanation given in the thesis:

  The first summation in eq. 3.25 corresponds to the use of $W$ for computing $r_t$, and the second summation arises from the use of $W$ as RBM parameters for $log \hat P(v_t |r_{t−1})$, so each $∂log \hat P(v_{t+1}|r_t)/∂W$ term is computed with CD (contrastive divergence). Computing $∂L/∂rt$ is done most conveniently with a single backward pass
  through the sequence. It is also seen that the gradient of the RTRBM ( Recurrent Temporal Restricted Boltzmann Machine) would be computed exactly if CD were replaced with the derivatives of the RBM’s log probability.

bold text added by me

The main leap here to relate to your example blog is that the it equates the log probabilities of a Restricted Boltzmann Machine to the loss of a generative RNN. If you are interested in more details, I would recommend reading either all of Chapter 3, or perhaps just sections 3.9 and 3.10.

I posted that equation because I could see a version that so closely matches the code in that blog in either of the original papers that propose backpropagation through time. Additionally, none seem to be free to read.
You can find them link on the relevant Wikipedia page.

Have a look at this related paper, written by one of the original authors of BPTT. Backpropagation Through Time: What It Does and How to Do It.


(Disclaimer: I am the core maintainer of Kur.)

Kur and Keras are both deep learning frameworks, but they serve different purposes:

Keras builds on Theano and TensorFlow to provide a high-level API to the underlying tensor libraries. This means that you use Keras as part of your Python program, and you programmatically create your model like this (this is Keras' functional API):

import keras.layers as L
import keras.models as M

my_input = L.Input(shape=(100,))
intermediate = L.Dense(10, activation='relu')(my_input)
my_output = L.Dense(1, activation='softmax')(intermediate)

model = M.Model(input=my_input, output=my_output)

This is great when you are developing deep learning networks that you want to "embed" in some larger Python program, or when you are developing a single model that you intend to use. Keras is also quite mature and has a great community supporting it. Keras currently offers the superior API for models that you will create and use entirely within Python.
Kur builds on top of Keras to provide a descriptive framework for designing and running deep learning models; you don't write programs when using Kur. Instead of programming, you write "Kurfiles" which describe your model:

model:
  - input: my_input
  - dense: 10
  - activation: relu
  - dense: 1
  - activation: softmax
  - output: my_output

That's the same model as I just wrote in Keras. Kur's descriptive language makes quickly prototyping and testing your models. The advantage of the Kurfile is that you can tweak large or small pieces of your model without having to touch your codebase. And when Kurhub is released, you'll have a Docker-like repository for sharing models.

Kur also takes care of a lot of details that you need to consider when writing deep learning systems. For example, Kur knows about several different datatypes (CSV, Python pickle, etc.), and it can load the data and infer shapes for you. Without Kur, you'd have to write your own data processing pipeline (which may be easy or hard, depending on your data). Once you write the Kurfile (in YAML or JSON), you can train with kur train Kurfile or use it for inference with kur evaluate Kurfile. This makes it very easy to use as a tool, rather than as a Python API.

Both Kur and Keras can use Theano or TensorFlow, both offer Python APIs, and both are easy to write models in. It really depends on how to plan to develop and deploy your system.


I'd recommend spending more time thinking about feature selection and representation for your SVM than worrying about the number of dimensions in your model. Generally speaking, SVM tends to be very robust to uninformative features (e.g., see Joachims, 1997, or Joachims, 1999 for a nice overview). In my experience, SVM doesn't often benefit as much from spending time on feature selection as do other algorithms, such as Naïve Bayes. The best gains I've seen with SVM tend to come from trying to encode your own expert knowledge about the classification domain in a way that is computationally accessible. Say for example that you're classifying publications on whether they contain information on protein-protein interactions. Something that is lost in the bag of words and tfidf vectorization approaches is the concept of proximity—two protein-related words occurring close to each other in a document are more likely to be found in documents dealing with protein-protein interaction. This can sometimes be achieved using $n$-gram modeling, but there are better alternatives that you'll only be able to use if you think about the characteristics of the types of documents you're trying to identify.

If you still want to try doing feature selection, I'd recommend $\chi^{2}$ (chi-squared) feature selection. To do this, you rank your features with respect to the objective

\begin{equation}
\chi^{2}(\textbf{D},t,c) = \sum_{e_{t}\in{0,1}}\sum_{e_{c}\in{0,1}}\frac{(N_{e_{t}e_{c}}-E_{e_{t}e_{c}})^{2}}{E_{e_{t}}e_{c}},
\end{equation}
where $N$ is the observed frequency of a term in $\textbf{D}$, $E$ is its expected frequency, and $t$ and $c$ denote term and class, respectively. You can easily compute this in sklearn, unless you want the educational experience of coding it yourself $\ddot\smile$


Well, I have a python/numpy PLS implementation that I could probably
  adapt into a python script block, but is this necessary? Please tell
  me that it's just a mouse-click/import away.

Welcome to the site! I wish I could tell you that what you want is a simple point-and-click but I don't think that it will be.

Generally speaking, most people use Orange as a tool for the visualization layer while you are looking for a tool for your modeling layer. Python has tons and tons of tools that you can use for modeling and should fit what you need. So, while that will be more steps for you, it will ultimately provide you more flexibility for other projects you may do in the future; just use Orange for the visualizations.


Which model is better, one just before overfitting with higher accuracy or one with no overfitting and lower accuracy?

Right now I would say model 1 is better since it has better validation results (If you take the model at epoch 5, not the overfitted one).
However, I would procceed by retraining model 2 for more epochs and check if the results improve.


Should we identify outliers from population prior to taking sample?

It depends on how you want to treat outliers. Outliers can either be generated by chance from the data's distribution or can be a result of human error (error in measurement, error in data entry, etc.).

If you consider the outlier to be from the first category, in my opinion you shouldn't discard it, in order to take your samples from the true distribution of the data.
Outliers in this category, might indicate a heavily-skewed distribution, or might simply appear due to the nature of the problem (e.g. King's effect).
If you consider it to be an error (e.g. a human height of 3.75m), you should discard it as it corrupts the distribution of the data.

In order to determine which category the outliers belongs to, either requires domain knowledge (e.g. knowing that human height can't exceed 3m) or requires you to make assumptions on the data.


real word dataset on company full name with commonly used short name

I think a better idea would be to use approximate string matching techniques: the general idea would be to compute a similarity score for each candidate term, then consider it a match if the score is higher than some threshold. This approach can be refined by  computing a set of similarity scores according to different similarity measures (see e.g. this list) and train a classifier using these scores as features. It's even possible to use more sophisticated approaches involving training a model taking into account the probability of specific edits, but it shouldn't be necessary in a simple case like this.


I think the aspect of NLP you're looking for is Named Entity Recognition (NER). It's already built into Python libraries like NLTK and SpaCy. Some videos that might be of interest:

How to train a new 'DRUGS' entity type: https://prodi.gy/docs/video-new-entity-type
Practical example: https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da

The great thing for you is that out-of-the-box models are probably going to be extremely accurate in detecting place names (either LOC or GPE entity tags). They may even be able to detect drugs as a 'PRODUCT' tag. If you need to be more specific, the first video shows you how you would go about training a model for a new tag. For instance, in spaCy:

import spacy
nlp = spacy.load("en_core_web_lg")
doc = nlp("""When I lived in Paris last year, France was experiencing arecession. The nightlife was too fun, I developed an addiction to Adderall and Ritalin.""")
print([(e.text, e.label_) for e in doc.ents])

will print something like this:
[('Paris', 'GPE'), ('last year', 'DATE'), ('France', 'GPE'), ('Adderall', 'PRODUCT'), ('Ritalin', 'PRODUCT')]


How to update the posterior belief when we are observing a stream of correlated data from a fixed but unknown data source

I think all of the options you give will yield results for the problem you described. It seems you have a y like a die that yields results which are rolls of the die all unpredictable but still linked from the y and you want to infer the y based on the x values. Like a
 Hidden Markov model.

The part about wanting to accurately make an estimate of y at each x and the x's being correlated make me believe that LSTM technology may be of benefit. If you want to use a NN.

Optimal stopping and Sequential Hypothesis Testing and Sequential probability ratio test and HDI+ROPE will all work for the abstract problem you describe. Until you give us more details about your problem like what specifically you will be working with it is hard to give you concise direction.

If you create a LSTM That takes in an X and makes a guess about which y and train it on x data you have then you would have the predictive model you seek.


How do reshape an image to fit my Mnist Convolutional model?

model.predict(img) requires a batch axis, i.e. img needs to have a shape of (BS, H, W, C). Try model.predict(np.expand_dims(img,0)).


The point of a "Data Lake" is to store unstructured data together, along with the related meta-data, all in one place. The data is usually stored in the format in which it was recorded i.e. the raw format. Data is then only extracted and processed for usage as required.

There is a nice article that compares data warehouses and data lakes in an unbiased and practical manner.

From your description of your data and the problems you want to solve, I don't personally think a data lake would be required.

Perhaps your colleague is planning to create a data lake for larger purposes, e.g. to house the data for many projects such as yours, which might also want to use your data? There are the issues of security, data ownership and data stewardship to think about in this case. Questions to answer would be:

who actually owns the data?
who is allowed to use what data?
who is allowed to view what data?
who is responsible for ensuring data is available at all times?


Python - Converting 3D numpy array to 2D

I suggest you to visit this link
also for this case would work np.reshape((-1,2))


How can a neural network get unstuck from a local minimum?

In various ways suchs as

momentum: think of momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. This means replacing gradients with a leaky average over past gradients.
sparse features and preconditioning (Adagrad): decrease the learning rate dynamically on a per-coordinate basis. This means, use the magnitude of the gradient as a means of adjusting how quickly progress is achieved - coordinates with large gradients are compensated with a smaller learning rate.
RMSProp: a combination of momentul and adagrad, combining leaky averages and coefficient-wise preconditioner
Adadelta: The learning rate cannot be parameterised but rather adapts itself given rate of change in the model parameters
Adam: a great algorithm that summarises features from all of the above
Scheduling: decreasing the learning rate during training

sources: https://distill.pub/2017/momentum/, https://d2l.ai/chapter_optimization


Basic DNN with highly imbalanced dataset -- network predicts same labels

A neural network is probably a perfectly reasonable approach here. But this is an extremely unbalanced dataset and you're going to have to handle that somehow.

The network is learning that the best way for it to reduce the loss is to always guess neutral - if you were taking a standardized test and you realized that you'd get 90% right if you always guessed 'C', you certainly wouldn't answer 'A' and 'B' in equal measure.

There are a few standard ways to deal with this, and probably depend on how big your dataset is (how many examples 6% and 4% come out to be):

Oversample the minority classes to even things out (so you'd show the net the same example of the positive / negative classes multiple times for every time it saw each neutral example)
Undersample the majority class (only show the net a random subset of the neutral samples during each epoch)
Set a class weight / sample weight - proportionally weight down the contribution to the loss function by examples of the majority class.

I'd probably try 3 first because it's super easy. Check out the documentation of the Cross Entropy Loss. It takes a tensor of weights as a parameter and weights the classes according to the entries in that tensor. So just say the positive (4%) class gets weight 1, negative(6%) gets 2/3, and 90% gets 2/45 (if I did the math right...).

That would even out the total contributions of each class in the dataset. Hopefully you'll see better results that way. You'll probably want to look at precision / recall to evaluate the results (because random guessing gets you 90% if you're using accuracy).


Machine-learning algorithm / library for outcome prediction

It is a basic classification model. Our focus is to train the model using training dataset and evaluate its performance with the test set.

In your case, assume you have 10 numbers as input and a binary output.
Consider random 8 entries as your training data and train it with any classification algorithm using R/Python. Then test it on the remaining entries which acts as a test data.

Since it is associated with classification we need to consider performance metrics like accuracy, precision and recall based on the labeled feature.

For more details on classification try here


Stuck on deconvolution in Theano and TensorFlow

Maybe have a look at this post. You can do a convolution which produces an output of similar size and then "unpool" those feature maps.


Can overfitting occur even with validation loss still dropping?

I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of what you sampled is used to validate your network , this means that you train and validate using balanced data set. In the testing you used imbalanced database. This means that your validation and testing sets are not equivalent.
In such case you may have high validation accuracy and low testing accuracy.
Please find this reference that talks mainly about data imbalance problem for DNN, you can check how they sample to do the training, validation and testing


what happens to the sample points on pruned leaves

Once you prune nodes, they get removed from the tree altogether, so you wouldn't be able to predict based on them.
Typically, hyperparaneter tuning is used to ensure that the right nodes are pruned. Obviously, there is always a trade-off.


Open source alternative to google's Auto ML

AutoSklearn, AutoViML, TPOT, H2O are good choices. They can do feature engineering, but I would suggest to build your own feature based on domain knowledge.


Including column of indices as predictor for model?

No, don't include indices. They don't provide any meaningful information about the problem. They are just an enumeration. Take them away.

Is there a reason why including this column of indices gets you to 100% F1 score or even accuracy?

You probably get that when evaluating in train set, in test you wont get that. If you do, then you have a problem with your data (it might be sorted).
What does the underlying algorithm do here? Something like this for a decision tree
if index == 1:
 predict =1
if index == 2:
 predict =0
.
.
.

As you can see there is no meaningful information about the problem here and the prediction in train will still be zero.
Conclusion: don't use indexes as a training feature.


Unsupervised Anomaly Detection on system metrics like memory, cpu, io, net, etc

Do you have sequence data about these metrics? E.g. do they evolve over time and you want to locate anomalies in these sequences?

If so, you can have a look at the tools described here:

  https://github.com/rob-med/awesome-TS-anomaly-detection

They are all unsupervised methods, so you don't need anomaly-free data for training.

If you are not dealing with time-series, there are also many unsupervised approaches you can take. This paper offers a good survey on the topic

  https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173


How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

Additionally, in case I upweight my classes with compute_class_weight(), I assume that no further class distribution should be taken into consideration downstream (so when I use RandomForestClassifier(), class_weight hyperparameter shouldn't be ='balanced', again, because this would further distort the weights proportionality that is already set before. Is this correct?

The util function is just a utility that will return you the dictionary for class weight.
The place where you will use this dictionary is the class_weight parm of the Model e.g. RandomForest.
So, you will have to pass the return of the Util to the model.
Better, just keep the parameter value = "balanced" Or any specific value e.g. [{1:1}, {2:5}, {3:1}, {4:1}]

OK, but this is only for rebalancing proportionalty, I should take misclassification cost into consideration as well. Let's say that this is 10 larger in case of false negatives than false positives, so I guess that I should still increase the weights in class_weights accordingly by upweighting positives further by 10, right?*

Just using the Util and getting the dict of weight will not have any effect. Action will happen only when the Model uses the class_weight parameter.
The Model will add corresponding weight to the misclassification cost of the minority class. Low-level design detail changes with Model e.g. Neural network, Decision tree.

But there is another point in the pipeline where I could take care of this, namely in the evaluation metrics with F-beta for example with upweighted recall (F2, for instance). Does it have the same effect? Should I pick one method (F-beta for evaluation OR upweighting classes) or both of them simultaneously?

Change in Scoring technique will be needed because whatever be the Class weights, the counts are still unbalanced e.g. 100:1. So, 90 correct of the majority will make the overall accuracy ~90 which can be deceiving.


Time Resolution Changes in Time Series Forecasting

I would do two separate analyses:

By day for six months
By month for the last 36 months (with the most recent 6 aggregated)

The 36 month analysis will be important for catching annual trends you wouldn't be able to see with the daily data, but you'll have a lot more fine-grain with the daily analysis to catch quarterly, monthly, and weekly cycles.


With only one class in your training set, this sounds more like an unsupervised learning problem.  Maybe look into anomaly/outlier detection algorithms rather than more traditional binary classifiers.


how to deal with two high correlations feature which both has a low correlation with target

If two features have strong correlation together that could mean one of these features is redundant, and probably advised to delete it from the features list in your predictive model.

add to that if the correlation with the target is low then you may need to delete both.

How to decide?

there are many ways, one of them is Feature Selection Filter Method, or an algorithm such as FCBF.

which you try the correlation of the features before implementing your predictive model

The other one is the Wrapper method or Embedded method which will use your predictive model to decide which features to choose.


Tensorflow/Keras, How to convert tf.feature_column into input tensors?

I think my answer over there https://stackoverflow.com/questions/56693937/tensorflow-how-to-convert-feature-column-to-vector might help you.

Look at

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)


h2o, different stopping metric leads to different optimal for hyperparameters

First of all, you are using different metrics to determine how well you are doing, that means it's not weird that different metrics find different hyperparameter settings that work better. Second of all, some hyperparameters might not matter for the problem you are solving, which means all the signal you are getting from those hyperparameters is noise. Third of all, most of the machine learning algorithms are stochastic, meaning there is randomness involved in training them and sometimes in evaluating them, this means even starting the same grid or random search could lead to different hyperparameters. That said the probability of that is only high if the real performances are close to each other.


I hope this is not a late answer but actually you can use category_encoders library, it follows sklearn's style.

Example:

import category_encoders as ce

#I'll pretend that you've already split your data into train/test

#your categorical features
cat_features = ['cat_feature1', 'cat_feature2']

#count encoder
count_encoder = ce.CountEncoder(cols=cat_features)
count_encoder.fit(train[cat_features])

train = train.join(count_encoder.transform(train[cols]).add_suffix('_count'))
test = test.join(count_encoder.transform(test[cols]).add_suffix('_count'))


If you're new to both machine learning and programming, try taking a look at this guy's tutorials: http://www.pyimagesearch.com/. I've been using them for a while, and I think they're great because of how visual, hands-on, and practical they are. For example, he has facial recognition and handwriting recognition lessons. I'm using what I learned in his tutorials to build a self driving remote control car (similar to this: https://zhengludwig.wordpress.com/projects/self-driving-rc-car/).


Does SQL Server support the Poisson distribution?

Yes, SQL Server does support it, as a part of Data Analysis Expressions (DAX).

The documentation for the function for getting Poisson distribution data is here.


XGBClassifier and RandomForestClassifier variable importance plots are very different

There are multiple reasons why this could happen. The first, and probably the most important one is that while both of the algorithms are tree based ensemble methods, they are very different from each other. Second of all, these feature importances are just some heuristic that do some calculations on where the features are used. I assume these heuristics are very different for both the algorithms

In this case however the differences might just be too big to only point to these two reasons. What could have happened is that your features are highly correlated. If two columns are highly correlated then the algorithms become kind of indifferent to which one to use, which means the two algorithms might both pick a different one each time they come across this choice. You could take a look if column 0 and column 7 are very correlated.

Another option is that some of your columns are categorical and that the implementations deal with this in a different way. Python doesn't work cleanly with categorical values in Scikit-learn. One-hot encoding is not a nice representation for undeep trees but if you believe this may be causing it you could preprocess them like that to see if the feature importances become more similar.


How should we use a sparse matrix in classification?

Your whole question makes sense, except for:

  because it is sparse in usual usage is useless

Therefore, I will try to answer this from different angles.

Implementation of sparse matrix

A sparse matrix is not useless because many packages contain algorithms that accept a sparse matrix as input.

You can look at Sklearn's SVMs as an example.

You need a special implementation to deal with sparse matrix because they are stored differently from a normal matrix. However, as mentioned in the comments by @Majid Mortazavi you can follow this answer to convert your sparse matrix to a normal one and use any other implementations out there. You just need to be careful with your RAM usage.

Theoretical application of sparse matrix

A sparse matrix is not useless because many fields in machine learning make use of sparse features.

You can look at NLP as an example. A common feature used in text categorization is TF-IDF which is a sparse representation of a document.

Since the data is sparse, it is often enough to use linear models for classification, since most of the features for a data point will be $0$. This can be considered a good thing because they are easier and faster to compute and no polynomial kernels are needed.

Solving your challenge

You also mentioned PCA for dimensionality reduction but I don't understand why you would need it at this points. Seems like you are creating a sparse matrix to encode a categorical feature.

Maybe, instead of reducing your sparse matrix, you can have a look at better ways to deal with categorical features.


Best practice - memory management when data wrangling

Programming best practices that are specifically usefull for ML / DS :

Know and handle your data types, with a dict for example, and pass this dict as and agrument to your read csv. You can get important memory usage reduction by using the right types of float / int and category instead of string. Float precision is rarely important in ML (and usually achieve better statistical performance trough faster calculations).

As a part of your model building process, it is usefull to do some variable selection (to reduce the number of columns) or sampling (to reduce the number of rows). Sometimes just for testing your pipeline, sometimes to get a baseline, sometimes as a long run solution.

You have to actively avoid multiplying you dataframes. I usually check for that in real time by seeing what memory is consumed. (Either trough command line interface like htop, or trough ad-hoc memory requests in the python environnement). This means you have to look into your functions which are duplicating your df. Then you have some work to do to find an alternative that is low memory (using vectorized calculation, using a 'in_place = True' option ... etc). During this step you also have to check that your operations are not upgrading the data types you specified in the first part.

Clean after yourself. You have to delete the objects you don't use (the raw data after you preprocessed them, the whole data you just splitted ... etc.). Basically using del() everywhere you can and the garbage collector gc.collect() often.


Stochastic gradient descent based on vector operations?

First of all, word "sample" is normally used to describe subset of population, so I will refer to the same thing as "example".

Your SGD implementation is slow because of this line:

for each training example i:

Here you explicitly use exactly one example for each update of model parameters. By definition, vectorization is a technique for converting operations on one element into operations on a vector of such elements. Thus, no, you cannot process examples one by one and still use vectorization.

You can, however, approximate true SGD by using mini-batches. Mini-batch is a small subset of original dataset (say, 100 examples). You calculate error and parameter updates based on mini-batches, but you still iterate over many of them without global optimization, making the process stochastic. So, to make your implementation much faster it's enough to change previous line to:

batches = split dataset into mini-batches
for batch in batches:

and calculate error from batch, not from a single example.

Though pretty obvious, I should also mention vectorization on per-example level. That is, instead of something like this:

theta = np.array([...])  # parameter vector
x = np.array([...])      # example
y = 0                    # predicted response
for i in range(len(example)):
    y += x[i] * theta[i]
error = (true_y - y) ** 2  # true_y - true value of response

you should definitely do something like this:

error = (true_y - sum(np.dot(x, theta))) ** 2

which, again, easy to generalize for mini-batches:

true_y = np.array([...])     # vector of response values
X = np.array([[...], [...]]) # mini-batch
errors = true_y - sum(np.dot(X, theta), 1)
error = sum(e ** 2 for e in errors)


You should be reading about text classification with supervised learning technique. You could chose a neural network such as CNN implementation and use it to your data. But you have to first prepare your data by cleaning it up and labelling it with respect to your desired category. For example,

You should collect the tweet data and label each tweet or set of words as job or news with respect to their original category.
Then you should use the prepared data and labels to train the chosen network.
Read about k fold cross validation and segregate data in such a way so that systematic iterative training and validation can be done and tweaks to the network could be made.
Finally once the training is completed you could use the model to test it on unseen test data.

These are common steps involved in text classification. It should be customized according to your need.  You could google profanity filter which is mostly developed with a NLP type classifier. That would help you a lot.


which metric is better for boosting methods

Depends.

The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric.

Which one? Depends on which underlying problem you are solving, if what you are doing is to determine which algorithm is better, your conclusion will only be applicable to your specific dataset

Accuracy: Is possible to measure the accuracy as the comparing metric, but it becomes trivial if your dataset is unbalanced, which means you have much more positives than negatives or viceversa. The accuracy is used when the dataset is balanced and if is equally bad to have a mistake on positives and negatives. Also, it has the problem of being too dependant on the threshold for defining positives/negatives.
Area Under the Curve: The AUC is one of the most robust metrics for measuring the ability of a model to separate positives and negatives, is unsensitive to threshold and immune to unbalanceness. I would use this.
RMSE: I only know RMSE for continuous regression, not for classification.
LogLoss: Its use is in multinomial classification