Incremental training and Auto Machine Learning for big datasets

First of all using auto-sklearn, you can use

automl.fit(X_train, y_train, dataset_name='X_train',
               feat_type=feature_types)

    print(automl.show_models())

so you can extract the instance of the best model from the first fitting. However in order to learn incrementally you have to (in case of sklearn models) have fit_partially method. Naive Bayes varaints and other algorithms here have this functionality. So you are out of luck if these are not in the output of show_models: In this case you ought to do your own automated ml targeted on fit_partial models.
Alternative is using spark it has some cool streaming (incremental learning algos)
StreamingKMeans, StreamingLinearRegressionWithSGD, StreamingLogisticRegressionWithSGD and generally StreamingLinearAlgorithm.
To conclude, I would not use auto-sklearn if these are your problems and choose some alternatives that do work parallel.


Around how much difference between two classes can be accounted as an imbalanced dataset?

In general there will be not a hard rule about this, but this dataset seems to be like balanced.
The point about disbalanced is that you have to keep in mind that the accuracy of your model will have a different starting point. For this python has the function to run a baseline with dummy classification. https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html. This gives you a good indication what is your baseline (if you choose the the strategy to most_frequent). In your case, any prediction model which always forecast class A, would have a accuracy of: 17/(10+17) = 0,6296... , so any binary classification with a score under this value would be really bad


I think your interpretation does not focus on the right places.

First, let's understand where the predecessor idea of Transformer XL in LSTM Language Models: Truncated Back-Propagation Through Time (TBPTT). In LSTM Language Models, the training data text is concatenated into a single very long sequence. From this sequence, we create training batches in a way that for batch $k$, the text sequence at position $i$ is the continuation of the sequence at the same position in batch $k-1$. When training the LSTM, the last hidden states generated at batch $k-1$ were used as initial hidden states when processing batch $k$. The gradients were not propagated through batches, only the values of the hidden states. This allowed the LSTM to be able to profit from unlimited-length context.

The same idea cannot be directly applied to Transformers, as there is not a single hidden state, but as many hidden states as the number of tokens in the sequence times the number of layers.

TransformerXL solves that by incorporating all those hidden states from the previous batch into the computation of the new batch, without propagating the gradients (see figure below from the original article).

This, however, poses a problem in how the positional information in managed in the Transformer, as the positional embeddings encode absolute positions. Therefore, the authors reformulated them to be relative.

Therefore, the main advantage of Transformer XL over vanilla Transformer LMs is that it can profit from unlimited-length context.

Note that Transformer XL is "only" a Language Model, it does not translate, just computes probabilities of the next token based on the previous ones.


Can I add categories to a dataset after and retrain an H5 file? Without retraining the previous ones?

To get around this issue, I would suggest doing training in batches. This solves the issue of not being able to load 100,000 images and also removes the need to figure out how to merge the models.


How to have Multiple labels in a single video?

Given that you have only 3 classes and that they closely depend on each other, I think it's worth trying a multiclass setting as WBM said. The idea is to label each video using the full combination of actions, since the maximum number of combinations is 2^3 = 8:

R-I-F
R-I
R-F
R
I-F
I
F
none

Probably some combinations of actions are impossible, so the number of classes is likely less than 8. Why this is a reasonable approach:

The setup is exactly the same, i.e. you can use the same labels and the predictions can be used the same way as in your multi-label approach
This is a "joint model", i.e. a model which learns everything together and therefore can exploit fine-grained distinction between classes (e.g. between R-I-F and R-I).

However note that this kind of method may require more data, in particular it needs to have enough instances for each class.


Can we apply community detection algorithms for word vector space?

"Improving Community Detection in in Wikipedia Articles using Semantic Features"
This paper talks about various methods of community detection and might be helpful.


How do I assign specific values to categorical variables

One possible way to map from string values to specific numerical values is by using a Python dictionary as a lookup table. The lookup table can be used for each value in the column with .apply(func) on the column.
import pandas as pd
l = [{'col1':'Increased'},{'col1':'Decreased'},{'col1':'Neutral'}]
df = pd.DataFrame(l)
print(df)

Output:
        col1
0  Increased
1  Decreased
2    Neutral

Create mapping and apply:
value_map_d = {'Increased':1,'Neutral':0,'Decreased':-1}
df['col1_numerical'] = df['col1'].apply(lambda x: value_map_d.get(x))
print(df)

Output:
        col1  col1_numerical
0  Increased               1
1  Decreased              -1
2    Neutral               0

In the example above, I read the values column col1 and write the numerical codes to a new column in the data frame called col1_numerical. The original column is dtype object and the output column is dtype int64 because all the values retrieved in the value map are integers.


What is the difference between these two methods of batching with stateful LSTM

Answering my own question: they are the same.


Small project ideas for Machine Learning

Gather million twitter user accounts. Then try to guess their gender based on their avatar, tweets and who they follow.


What am I supposed to see on tensorboard images tab?

This is what I see using data from Pascal VOC that was converted using this example script:
I am using Tensorflow 2. I think it's just the images preprocessed using the data_augmentation_options that were specified in your pipeline.config file.
I seem to recall seeing bounding boxes on the images in Tensorboard back when I was using the Object Detection API with Tensorflow 1. But I haven't seen any bounding boxes on the images using the API with Tensorflow 2. I have verified that the models subsequently worked. I just didn't see bounding boxes in Tensorboard. Note that I am just running training and then doing evaluation separately. If you specified eval_config or eval_input_reader in your pipeline.config, you might get different results.


What supervised machine learning model can be used to generate a scorecard-like result?

It depends what you mean by "can be used": any regression algorithm can be used, the question is how reliably it would perform. You can compare different algorithms experimentally (if you have a dataset).

[Updated after question edited]

In general the way to use ML with this kind of setting is to train a classification model based only on the categorical features. Depending on the type of algorithm, the combination of features might not always be a weighted sum, and the result label may or may not be based on a cutoff point. In order to have a cutoff point (thus a numerical prediction), the method must be a soft classification method. Alternatively a regression model could be trained for predicting the numerical value.

So that leaves you with many options:

soft classification: linear/logistic regression, Naive Bayes, ...
regression: linear/logistic regression, SVM, decision trees, ...

Note: technically the probability doesn't represent "how far from the cutoff point", it represents the probability of the instance being positive (p=1).


Multiclass Classification that includes a Geospatial Element

If this is a U.S. model you can try County (FIPS) codes, which are coarser-grained areas than ZIP codes (there can be many ZIP codes in a county).  If bearing and distance worked for you, this should also work.  Your classifier would need to handle categorical data.  In R, this is pretty easy to do if you already have a ZIP code:

library(noncensus)
data(zip_codes)
my_dataframe['fips']<-zip_codes$fips[match(my_dataframe$zip, zip_codes$zip)]

This assumes you already have a dataframe called my_dataframe with ZIP codes as a column named 'zip'.

As a bonus you can use the choroplethr package to create a nice visualization of some scalar value, for example, your item price:

library(choroplethr)
df_to_plot<-data.frame()
df_to_plot['region']<-my_dataframe$fips
df_to_plot['value']<-my_dataframe$price

county_choropleth(df_to_plot, title='price by location')


Why Liblinear performs drastically better than libsvm linear kernel?

Here is just a guess, but according to me, the linearSVC might perfoms better than SVM with linear kernel because of regularization.

Because linearSVC is based on liblinear rather than libsvm, it has more flexibility and it gives you the possibility to use regularization with your SVM (default is L2-Ridge regularization).

Because you have more features than observations, it exists multiple solutions to your classification model. Some of them are more "robust" than other ones. L2 regularization will help you reduce the ampltitude of your model coefficients and maybe lead to a more stable solution.

More on LinearSVC in the documentation :

http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html

More on Ridge regression benefits when working with a lot of features (and so multicollinearity) :

https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago


Why does vanilla transformer has fixed-length input?

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.

The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-attention. The maximum length is that which makes the model use up the whole memory of the GPU for at least one sentence (once the other elements of the model are also taken into account, like the embeddings which take a lot of memory).

Transformer-XL is certainly a way to take into account as much context as possible in language modeling (its role is analogous to truncated back-propagation through time in LSTM language models). However, the gradients are not propagated through the attention over the memory segment, only through the current segment.

There have been several architectural attempts to reduce the amount of memory needed by transformers, like using locality-constraints in the attention (Dynamic Convolutions model) or using locality-sensitive hashing (Reformer model).

There have been other implementation attempts, like gradient checkpointing(e.g. this), which is a general technique to run computations that don't fit at once in the GPU memory


Is there "Attention Is All You Need" implementation in Keras?


Training set, validation set, and test set with Orange

The answer is yes. If Test & Score is given only one data set, then all it can do is show results of cross-validation.

To test the models on a separate data set, use separate File widgets to load training and test data. Connect File widget with training data to Test & Score, and the connect File widget with Test data to Test & Score. The connect whatever learner to test (in the workflow on the figure I've used logistic regression and random forest). Make sure "Test on train data" is selected in the Test & Score.


Cosine is for continuous values. It's not the most appropriate thing here.

For binary values, look at

Simple matching distance
Hamming distance
Jaccard distance
Many many many more.

Don't expect clustering to just work. If you have binary data, you only have very few bits of information. The resulting distances will show discrete steps, and clustering will likely still fail to provide good results. Roughly, you have "identical", "one difference", "two differences", and then probably everything already is connected.
It's because of the data, and a not very well defined task (what would be a good cluster?).


Modelling of an environment that is stochastic in nature

Is it possible to model an environment that is stochastic in nature?

Yes, a model of a stochastic environment can be one of:

A distribution model, that outputs a probability distribution for next state and reward, given current state and action. If you are reading Sutton & Barto, or similar work which uses the function $p(s',r|s,a)$, then if you can implement this function for the whole environment, it means that you have access to a distribution model.
A sampling model, that outputs a single reward and next state, given current state and action, with the same probability of any outcome as the real environment. If you can implement an accurate simulation of an environment, then you have a sampling model.

If you want to use approaches such as Dynamic Programming, which work with expected values, then this is much easier with a distribution model, and in that case you need an accurate model from the start (otherwise Dynamic Programming may converge to a non-optimal policy).

  Is it because its difficult to model such environment we use model free methods?

Not really, there is no special difficulty in modelling stochastic environments. For instance, if your environment is a dice game, it is just a matter of implementing the rules and a random number generator for the dice to create a sampling model. A distribution model is usually straightforward for basic dice rules, such as rolling a die to see how many steps you can take.

However, independently of how random the environment is, complex environments can become hard to model. Distribution models may require a lot of maths to calculate all possibilities, so sampling models (simulations) are easier. For instance, a card game is relatively easy to implement in simulation when you track the deck contents. But a distibution model for it is more complex, as you have to track what has already been played or can be figured out from other observations.

Many environments are too complex to model. For example, they may involve real world physics but don't include enough measurements to establish the full state. For instance, when an agent flies a drone it will be affected by air turbulence but cannot directly observe it. Chaotic effects such as turbulence are very hard to model, and gaining real world experience is likely going to be more accurate than any model based on a physics engine, no matter how hard you try to code one. Similarly, visualising the real world, or navigating human social environments can be very hard to model accurately.

When deciding whether or not to use a model-free method, there needs to be a cost analysis. Even in complex environments you may prefer to use a model-based method:

The advantage of a model is that it allows you to safely explore without taking a real action, and also in some cases it may be a lot faster to query the model than to take an action and wait for a result in the real world. In the time it takes for a real action to resolve a computer may be able to check 10, 100, 1000 or more simulated actions from its model.
The disadvantages of a model are:

It might not be as accurate as you like, meaning that basing decisions off it makes the agent's policy too far from optimal
Using a model adds complexity to the agent.

In some cases, the real environment is fast, safe and reliable enough that there's not much to gain by using a model. These cases include a lot of the toy problems used to study learning, where the environment is actually simulated (which is a form of model) but kept separate from the agent (so despite this, the agent is still technically model-free).


BERT is running out of memory in forward pass for my dictionary

It is likely to be independent of dictionary. Loading BERT model and running a forward pass has its own memory requirements. How did you figure that it is memory leak?
Try visualizing memory footprints on each step in your code by using some break points. It will give you clear idea about the hardware requirements and memory leak if any.


What does it mean the term variation for an image dataset?

Assume, you have two different datasets of cat images:

Dataset 1

Images were taken under full daylight.
The cat covers always almost the whole image.
The cat is always in the center of the image.

Dataset 2

The images were taken under various lighting conditions: some under full daylight, some in rooms, some during the night.
The cats were differently far away from the camera, e.g. sometimes the cat covers only 20% of the image.
The cats are in different positions in the images, e.g. sometimes, the cat is in the center but other times, it is in the upper right corner.

In this case, we would say that Dataset 2 has a higher variation than Dataset 1.

Generally, we use data augmentation to simulate missing information in datasets. For example, you could decrease the brightness of some images in Dataset 1 to simulate how a cat looks at night. Or you could zoom out to simulate how a cat looks from farther away.
Note that a perfect dataset that contained all relevant information would not require any data augmentation at all. However, gathering data is expensive and data augmentation is cheap. Therefore, we almost always have to augment.


Im not sure if I understand your question. But, you do not need to create a graph to work out node degree. Node degree can be obtained by taking the sum of the row or column in the adjacency matrix. E.g. in this 4 node graph where each node has degree $2$, and the row and column sums are also 2.

  0|1|2|3|              1----0
0|0|1|0|1|              ¦    ¦
1|1|0|1|0|              2----3
2|0|1|0|1|
3|1|0|1|0|

And so if we were to update the matrix to add a diagonal edge between nodes 1 and 3. (apologies for the rubbish graphic). Adding values to the diagonal creates a self-loop edge.

  0|1|2|3|              1----0
0|0|1|0|1|              ¦ \  ¦
1|1|0|1|1|              2----3
2|0|1|0|1|
3|1|1|1|0|

In relation to node location, yes refer to node locations using the indices of the adjacency matrix $(i,j)$.

In python this is done by:

import numpy as np

a = np.matrix([[0,1,0,1],
               [1,0,1,1],
               [0,1,0,1],
               [1,1,1,0]])

a.sum(axis=0) # row sum. aka Node Degree
a.sum(axis=1) # column sum. aka Node Degree
# form a tuple with the result of above
# i.e. (a.sum(axis=1),a.sum(axis=1)) = (1,1)

a.sum(axis=0).argmax() # index of row with max sum (returns 1st index if multiple nodes have same degree)

d= 2 # return nodes with degree value d
np.where(a.sum(axis=0) > d)[1] # returns indeces of nodes with degree > d


There are two packages in R that should be able to use SMOTE to up sample the minority class:

unbalanced package, and
DMwR package

It doesn't seem like they provide the boosting that is described in the paper that you might be referring to. But it should be straightforward to sew in SMOTE with the idea of boosting using some base tree packages in R like rpart (for example).


How to use Kaggle Api in Google Colab for directly using dataset?

Step 1 - Refer the Kaggle doc to understand the API basics and to get your key
https://github.com/Kaggle/kaggle-api

Step 2 - Use these LoC -

    import os
    os.environ['KAGGLE_USERNAME'] = "jha01roshan"
    os.environ['KAGGLE_KEY'] = "xxxxxxxxxxxxxxxx"
    import kaggle

    !kaggle competitions download -c ashrae-energy-prediction

    import pandas as pd
    building_metadata = pd.read_csv("/content/building_metadata.csv")
    sample_submission = pd.read_csv("/content/sample_submission.csv.zip")


I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other resources, like data science books.
Together with the repo, I suggest the TidyTuesday videos by David Robinson, where he creates screencasts of complete data analyses in R for the TidyTuesday datasets.


For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind?

A few things going on here:

Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post.
You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful given perfect noise), all parameters are set to zero. This in turn also gives you an $R^2=0$. In this case the model consists only of an intercept (should be equal to 0.5), which is essentially the mean.

Change the matrix to have less columns than rows and you will get a low (but positive) $R^2$ for linear regression. Also look at the Lasso coefficients, they are all zero.

Have a look at the predicted values as well. You will see that the linear model does not work well. So the R-squared gives no indication about model fit in this case.

You can also try to change the random pattern to have some trend and you will find that Lasso does not shrink all coefficients to zero.

import pandas as pd
import numpy as np
from sklearn import linear_model

n = 1000
k = 5

X = pd.DataFrame(np.random.rand(n,k))
y = np.random.randint(2, size=n)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print(lin.coef_)
print(lin.intercept_)

print(lasso.coef_)
print(lasso.intercept_)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Output:

[-0.00446436 -0.04509905  0.07870413 -0.00165025 -0.03951546]
0.5010650787133537
[-0. -0.  0. -0. -0.]
0.495
Linear regression score 0.003094912043144493
Lasso score 0.0


Textbook definition of Data augmentation

How about the definition from "The Art of Data Augmentation" by David A van Dyk & Xiao-Li Meng:
The term data augmentation refers to methods for constructing iterative optimization or sampling algorithms via the introduction of unobserved data or latent variables.
Citation:
David A van Dyk & Xiao-Li Meng (2001) The Art of Data Augmentation, Journal of Computational and Graphical Statistics, 10:1, 1-50, DOI: 10.1198/10618600152418584


How to convert an array of numbers into probability values?

Any Survival Function (1 minus the CDF) will have the desired property. Exponential is a potentially good candidate here, as it sometimes can be used to describe distances, but it's hard to say without more information.

$$S(x) = \exp(-ax)$$

The parameter $a$ can be tuned or possibly estimated from the data.

For reference, if $a = 1$ then you get,

$$ [0.44, 0.58, 0.86]$$

$$[0.62, 0.4, 0.36]$$

for the first and second arrays respectively.


Real world performance evaluation on problem similar to malware detection

Can you measure a proxy measurement that will be measurable even if the system never goes down? For example, does the engineer find “malware” or whatever the analogue is in this situation?

If not, and if it is safe to do so, why not simply take no action in X% of cases and use the data gathered in those cases as evaluation data?


binning high cardinality categorical features

The obvious disadvantage is that you lose information.

Probably the best approach is to generate new features based on the city, using domain knowledge: size of the cities, but also relevant geographical/societal/economic features of those cities.  Without domain knowledge and additional data, maybe clustering on your data can help to group together similar cities (though now your model will see that same information in the city-groups as well as in the data you used to create the clusters).

Weight of evidence is sort of like clustering similar cities, but according only to your target variable.  If you have cities with very few samples, then the WoE scores will suffer from high variability, so you'll probably want to lump them together anyway.


LSTM sequence prediction: 3d input to 2d output

What are your features like?  Given that you have a Dense layer outputting a softmax of size 5, this implies that all you want to predict is 1 feature, a categorical feature with 5 options.

If this is not true, more information about the features is needed to help here.

Your Y-variable for each feature should be of size (num_samples, time_step_len, num_categories_of_feature).  You need to one-hot-encode each categorical feature separately, which gives the last dimension size, num_categories_of_feature.  As you have it currently, the Y_train size is (num_samples, features).  So, as you have the problem framed, the network has no way to learn the sequence patterns, as you only give it the end result.  You should create your Y_train data to be the true value for the next time-step, for every time-step.  Hence, (num_samples, time_step_len, num_categories_of_feature).  Side note: I've only worked with LSTMs/RNN's on one problem, and this is how I did it.  I cared about learning the sequences in it's entirety, because my inputs at prediction time are variable.  If you always have 11 time-steps and always just want the next time-step prediction, this might not apply.  I really don't know to be honest.

This is where I'm not totally sure if this is the only way to do this, but the way I think of this problem for wanting to predict 5 categorical variables, you need a way to output softmaxs for each variable.  A softmax activation of size "features", like you have it here, is estimating a probability distribution of size 5, which implies your Y variable is only 1 categorical feature that has 5 potential values. So, you will need to set up your network to have 5 outputs with independent softmax outputs the size equal to the number of categories for each variable.  A single softmax should only be used to estimate a distribution over a single class variable.  10 options for feat1?  Softmax of size 10.  etc.

losses = {"feat1_output": "categorical_crossentropy", "feat2_output": "categorical_crossentropy", "feat3_output": "categorical_crossentropy", "feat4_output": "categorical_crossentropy", "feat5_output": "categorical_crossentropy"}
lossWeights = {"feat1_output": 1.0, "feat2_output": 1.0, ... , ...}# if everything is equal, dont worry about specifying loss weights.
metrics = {"feat1_output": "categorical_accuracy", "feat2_output": "categorical_accuracy", "feat3_output": "categorical_accuracy", "feat4_output": "categorical_accuracy", "feat5_output": "categorical_accuracy"}
opt = Adam(lr=init_lr,decay=init_lr / num_epochs)
model.compile(loss = losses, loss_weights = lossWeights, optimizer=opt, metrics=metrics)

Now, you will be optimizing 5 loss functions at the same time, one for each categorical prediction.  You must now have 5 Y-variable datasets, each of size (num_samples, time_step_len, num_categories_of_feature).  You will then give 5 y datasets for the outputs in the fit function, as a list.  However, to properly name the output layers, you will need to specify the names for the output layers in the model definition.


Carlini and Wagner (2018), “Audio Adversarial Examples: Targeted Attacks on Speech-to-Text”


KNN scoring low compared to Logistic regression in MNIST challenge

If you are referring score as test accuracy, your knn is overfitting a lot because of 1 nearest neighbour. Please do it with high k values and also try to do cross validation to get best model.


You could do it simple and it should work like this:
def myfunc(df):
    return df.nlargest(3, 'values')[['values']].sum()

and then:
data_agg = df.groupby('date', as_index=False).apply(myfunc)

You decide if "data_agg" is the proper name then.
Good luck!


Does the length of the text data affects the score of clustering

Yes, the length is very likely to influence any kind of similarity score. In general:

String containing very few words will have their highest similarity scores with other short strings with which they share one or two words in common. However if the words they contain are not common at all, they will have a lot of zero similarity scores (possibly only zeros).
Long strings (i.e. with many words) usually have low similarity scores, simply because they are unlikely to have a high proportion of their words in common. However compared to short strings they rarely have no word in common so they are much less likely to have lots of zero similarity scores.

As a result strings of similar length may tend to cluster together, depending on the data.


How many coefficients does the Logistic regression model has as a function of the number of features?

You are correct that the number of parameters depends on the number of features and it of the number of observations. You are seeing this error because you and the computer do not agree on what your input data mean.

Sklearn assumes your data to be $n\times p$, where each row represents an observation, and each column represents a variable. I think you’re doing the transpose if that, $p \times n$, where each column represents an observation. As you have more and more observations, you have more and more columns, telling sklearn that there are additional features.

If you transpose your predictor data, the functions should behave as expected.

This isn’t unique to logistic regression. You would have this happen with any model. Say you trained a k-NN on 80 observations of 6 features. Then you test on 20 observations of those 6 features. Python should come back like, “You gave me 80 features for training and now only 20 for testing. What gives?”


The default setting for max_seq_len is 25 as seen here under heading Server API:  bert-as-service readme

There is an open issue regarding this on the Github repo here and the creator seems to be implementing a feature:  bert-as-service issues


Does hierarchical agglomerative clustering with centroid-linkage suffer from chain-effect?

The potential worst cases of centroid linkage are probably too crazy to explain as easy as the single-link effect... To see how it reacts to the classic Single-Link problem, why don't you just try yourself?

Roughly, because of the way centroid linkage works, it may end up using a virtual cluster center that is outside of the actual cluster. You then may see some very weird links to happen. That is also why it can have non-monotone linkage levels (so later merges may be cheaper than earlier merges).


Sliding window time series data with Python Pandas data frame

You can use the built-in Pandas functions to do it:

df["Time stamp"] = pd.to_datetime(df["Time stamp"]) # Convert column type to be datetime
indexed_df = df.set_index(["Time stamp"])           # Create a datetime index
indexed_df.rolling(100)                             # Create rolling windows
indexed_df.rolling(100).mean()                      # Then apply functions to rolling windows

This code is not 100% correct because the datetime conversion is not correct but it should help you get started.


Appropriate objective function and evaluation metric when I DO care about outliers?

These are several things you can try:

Use quartic error, $(y - \hat{y})^4$, instead of quadratic error. This is going to penalize a lot big errors, way more than MSE. The issue is that this is not implemented in xgboost, and you would need to develop a custom loss.
If your target is always positive, you can use the target as training weights. This will give more weights to the outliers. If it is not always positive, you can use the absolute value of the target as weights. If using the target values directly puts too much weight on the outliers, you might want to transform it (e.g. using the log or square root), and if you have samples whose target value is zero, you might want to add some epsilon to all the weights. Note that xgboost can be easily trained using weights.
Try to predict the quantile of the training distribution, then transform your predictions using the training cumulative probability function.


Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

If most of your data and machine learning pipeline is in Python, then Dask project for Python is a very good candidate. It allows you to scale (certain types of) data-frame operations to datasets more than memory. The good thing about Dask is that it is also easy to scale down if required. Most of the code stays in Python, you don't pay the serialization overhead (which is paid in PySpark for Python -> JVM).


How to classify named entities of the same type?

You might be interested in resources built around TimeML, I think there are some corpora and specific parsers specialized in extracting the time details of events. I don't remember any specifics but I tried to google "timeml extract time" and found a few related resources, that might give you at least some inspiration about how people have dealth with similar problems.

If you don't find anything that suits your need, in general the best approach would be to train a custom NER using your own annotated dataset with "start-time", "end-time", "neither" labels.


How to access the data of the column on which some groupby operation has been carried out?

As county is not a key when you make the new dataframe so you can't access it.
So you just need to reset index so that county can be accessed when required.
df = pd.DataFrame(california_venues.groupby('county').count().reset_index())


How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

You can use the site academic torrents. The imagenet data you are looking for is shared there.


Just use a subsample of your data.

There is little use in evaluating (nor clustering, usually) the entire data set. It's highly redundant. Trying to scale this to the entire data set is a waste of time.


Importance/intuition behind stacking RNNs

All neural networks can increase expressiveness and representational capacity by stacking layers. Each later layer can learn to non-linearly weigh the earlier layers. These non-linearities allow any function to be approximated. In the case of Recurrent Neural Network (RNN), it is functions over time. Stacked RNNs have increased abilities to learn functions over time.


How to present Market Basket Analysis Results?

The best way to understand multiple association rules is to visualize them. This makes it even easier to present. This paper covers multiple approaches for visualizing association rules. Go through its references. They also suggest their tool, but it is in R. If you want resources for python try searching for "association rules visualization python" and you'll find some resources.


Career advice for a data scientist - Am i in the right place?

Welcome to the site! The only thing I can really do is offer general advice:

1 - Realize that your situation is not unique. There are many, many data science departments that just contain 1 or 2 people at most and many of us work alone - it's what I call being a "one man band" in data science. I work for one of the largest corporations in the world and I work alone because (1) budgets are always hard and (2) good data scientists are very, very hard to find. Oh, you'll find resumes out there, but most people will fail even the most basic tech screenings and just be awful at data science. You should take pride in being the person that your company picked and realize that just because you leave your current company it doesn't mean that your "alone" situation will improve.

2 - So, how can you turn your situation into a positive? If they don't have the budget to get someone senior, they might still have the budget to train you better. Get them to budget you money to go to conferences, get all the books you can, sign up for online courses that your company is willing to pay for. Even if they can't give you help, they can still get you resources in other ways. That's the advantage of being alone, your "department" doesn't need a big training budget, just enough to fly you to a couple of conferences each year.

3 - The beauty of being a data scientist is that there are always skills you can work on. How's your software deployment skills? What about your skills around building data pipelines? Do you tend to over-rely on one model instead of learning news one (that happens to all of us). There's always new parts you can be adding to your toolkit. In that sense, adding some DB skills is a good idea; work on both DB design and how to create the pipelines from those DBs into your programs - that's a very valuable skill.

Good luck!


are OneHotEncoder and keras To_categorical same?

To answer your question:

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

The OneHotEncoder instance will create a dimension per unique word seen in the training sample. Here you are only showing it 9739 different words at training so it does not need more dimensions to perform one hot encoding.

One way to have it accommodate the entire vocabulary is

onehot_encoder = OneHotEncoder(sparse=False, categories=human_vocab)
onehot_encoder.fit_transform(X)


Total of 972 emoji is not really that big not to be able to label them manually, but I doubt that they will work as a good ground truth. Sources like Twitter are full of irony, sarcasm and other tricky settings where emotional symbols (such as emoji or emoticon) mean something different from normal interpretation. For example, someone may write "xxx cheated their clients, and now they are cheated themselves! ha ha ha! :D". This is definitely negative comment, but author is glad to see xxx company in trouble and thus adds positive emoticon. These cases are not that frequent, but definitely not suitable for ground truth.

Much more common approach is to use emoticon as a seed for collecting actual data set. For example, in this paper authors use emoticon and emotional hash tags to grab lexicon of words useful for further classification.


Binary (Unary) Recommendation System with Biased Views

The simplest answer with the cleanest solution may be one that combines your binary click data and view data into another model feature that you can optimize against. What this looks like depends on your knowledge of your end-to-end system.

For instance, is it "good" that a user clicks on something with a high amount of views, or better if a user clicks on something with a low amount of views? You might have very different formulas depending on this question alone:

normalization_function((1/views) * mean(clicks)) vs perhaps normalization_function((view) * mean(clicks))

Be sure to check the assumptions of your matrix factorization implementation, but this new features may just be able to be swapped with your click data.

As a side note, I don't know exactly what you are implementing, but clicks and views generally represent different things -- as a rough guide (clicks = engagement, view = eyeballs), so it might not mean much to combine these.


Intuition behind the RNN/LSTM hidden state?

I personally don't think they are comparable to the hidden state of a Markov model. One key difference is that, in a HMM you can explain what a given state means to someone, where in a RNN/LSTM you cannot interpret a given state.

The closest thing that you can compare the hidden state of an RNN/LSTM is to think of it as the output of an intermediate layer of a fully-connected neural network but for time-series data.

And the larger the hidden state the more memory it can retain of the past.


How to transform stock data for LSTM-based neural network

Yes, you should calculate the log return for each stock, then feed them to the neural network.

I did a similar project before, but with only the close price, not the volume. I follow the method in this paper: Deep learning with long short-term memory networks for financial market predictions

Here's the code: https://github.com/tqa236/LSTM_algo_trading

Please take it with a grain of salt, the code quality is not very high. Also, the result is not quite impressive, both in my project and the original paper for these several recent years. We reach the same conclusion that it's really difficult to use vanilla LSTM to predict the stock market nowadays.


Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

You could use pandas.describe.

import pandas as pd
import numpy as np
df  = pd.DataFrame({'first': [1,2,3,4,np.NaN], 'second': [100,200,np.NaN, np.NaN,4]})
df.describe(include='all')

Regarding the percentage of NAs, you could use just a small formula for that:

count_nan = len(df) - df.count()
count_nan / df.shape[1]


What is the theoretical differences of Multitask learning vs Fine tuning based transfer learning?

As per my understanding,, in the both scenerio above when learning the object type and color what you are doing is a multitask learning. That is you are teaching your model to do two task simultaneously, 1. Predicting Object type(Which fruit/shape) and 2. Predicing color. So it is not like TL rather Multi task learning.
1.
TL does have some source task.But all you do in TL is that you freeze the whole model except the last few layers and retrain them on your specific dataset.(or add fews layers at the end and train them). TL doesn't train your model from beginning fro new task. But the aboe scenerios are not TL. So it is ambiguous to name them source and target task.
2.As said above, both of them are multitask learning rather than TL.So t is ambiguous to name them source and target task.
3.Both of the scenerio mentioned is multitask learning. Although you can use any of them as your source task and do a TL on it to work on the other. For example, you can train your Fruits and Color model in scenerio 1 and do a transfer learning on it for the second scenerio(to learn shape and color). As they have some same properties underneath, so transfer learning is a valid option here.


Multicollinearity and impact of individual features

When you face multicollinearity, your regression coefficients will likely be biased, because under multicollinearity, the regression can not tell apart the different effects: https://datascience.stackexchange.com/a/57118/71442.

When you use only one variable at a time, you will face the omitted variable bias, because there are no other confounders, to which the regression could attribute relevant effects. https://en.wikipedia.org/wiki/Omitted-variable_bias

To my best knowledge, there is no easy-to-go way to mitigate one or both of the described effects. You should carefully check the correlation between your $x$ and decide if multicollinearity is a problem. If this is the case, and you believe (for theoretical reasons), that all the highly correlated $x$ are important, you could try to find other representations of (some) $x$ to mitigate multicollinearity, e.g. dummy/indicator representations.

I guess your chair example is generic, so I don‘t speculate on this. Maybe you can give some more background regarding the actual problem.

The discussion above refers to causal modeling. If you are only interested in making predictions (not statistical inference), you could look into shrinking coefficients using lasso.


Does label encoding an entire dataset cause data leakage?

The cleanest solution would be to apply scikit's OneHotEncoder with the handle_unknown parameter set to "ignore":

handle_unknown{‘error’, ‘ignore’}, default=’error’
Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

Other manual solution are described in this and this question on Stackoverflow, for example.


Computer science corpus for training a language model

Depends on the domain and language, but I'll share an adaptive example.

The wikipedia corpus's English version contains more than 1.9 billion words from 4.4 million articles.

You can create create virtual corpora from the full corpus to contain only topics of interest, such as biology, investments, Buddhism, psychology, cars, basketball, etc.


Having averaged trials which are less than the number of features

About dealing with the mean of trials, I should say you can not recover the original trials and therefore you have to trust the results of classifucation model trained on averaged trials.

About dealing with a dataset with higher number of features than number of samples, I would suggest the following steps:

1- reduce the dimension of your data from 70 to 2 using PCA or t-SNE and

2- visualize it using a scatter plot to see if your labels are linearly separable in 2D space

3- If the result of PCA linearly separates labels, it means that your labels are kind of linearly separable in the original 70D space. So, you would apply the results of PCA (or even your original features) to linear-SVM.

4- If the result of t-SNE linearly separates labels, it means that your labels are not linearly separable in the original 70D space because unlike PCA, t-SNE is not a linear transformer. In this case, you would apply kernel SVM to your data.

5- If because of lack of data none of aforementioned methods worked, you would try low-variance classifiers such as Random Forest.

6- If Random Forest didn’t work, you would try using Bayesian methods such as Bayesian logistic regression because these methods assume a prior for parameters’ distributions and update it to posterior distributions using training data which compensates the lack of enough training data.


How to deal with unbalanced data in pixelwise classification?

Looks like the problem lied just in poorly chosen parameters. In the example given above, simply increasing epoch count significantly improved the results. In my actual image processing problem, I had to decrease learning rate, since with the default one, ReLU's were saturating very quickly, hindering further progress.


Q: How many machine learning specialists does it take to change a light bulb?

A: Just one, but they require a million light bulbs to train properly.

Q: How many machine learning specialists does it take to change a fluorescent light bulb?

A: That wasn't in the training data!


How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

A solution is set a maximum length for the number of categories and fill the future categories in order to see them in your data. Hence, that part of the categories which are not seen in the sampled data would be set to zero.


Note: this is more frequently called canonicalisation than harmonisation.  URLify and URL-safe are also terms that can help you in your search.

whitespace
Combining the built-in functions string.split + string.join is a solid way to canonicalise intermediate and trailing whitespace, including tabs, carriage returns and so on.

non-ASCII chars
ASCIIification is actually somewhat contrary to canonicalisation.  That is, you should think hard about whether you really want to be converting 'München' to 'muenchen', rather than 'muenchen' to proper 'München'.

If you do want to go in that direction, you should give more details on the use case and the desired conversions.  Here are some examples of possible input:

  Where is Münsterstraße?
  ısparta'daki istanbul'dakı ve gülenciler!
  Isparta'daki İstanbul'dakı ve Gülenciler!
  ¿Cómo está?
  Это вообще фарс...
  Биће белајааа, чорба
  Biće belajaaa, čorba
  You can now resume your résumé.

Converting to ASCII will be lossy and there are some decisions to make, often dependent on the source and target language.

If you define the likely input, scope and desired behaviour then others can make recommendations.


Why use different variations of Softmax in training and validation for neural networks with Pytorch?

It's more of a pytorch implementation thing. log_softmax() outputs the raw logits and they are passed to NLL Loss in training. During inference you just need the probabilities so softmax will suffice.
You don't use different algorithms for training and testing, the question is quite misleading as currently stated. You are using different implementations of the same algorithm.
You can find more on the matter here:

Pytorch discuss
Pytorch discuss #2
Pytorch docs

P.S: I remember encountering the same question on a Udacity scholarship. If you 're reading code from one of their courses, there is probably an explanation on the solution jupyter.


What GPU specifications matter when training and using neural networks?

Besides the criteria listed look for

The number of cores
The ability to do low precision arithmetic

In practice you are probably best off just getting the latest generation NVIDIA card (currently 10xx).


I agree with @peter's detailed answer on points #1 through #5 and would like to supplement that with some more details:

Perform a PCA or MFA of the correlated variables and check how many predictors from this step explain all the correlation. For example, highly correlated variables might cause the first component of PCA to explain 95% of the variances in the data. Then, you can simply use this first component in the model.
Random forests can also be used for feature selection by looking at the feature importances of the variable. However, correlated variables can cause misleading feature importances. You can use permutation feature importance (https://scikit-learn.org/stable/modules/permutation_importance.html)
You might be overfitting when using all the features. Therefore, it is essential to use a validation set to check for overfitting.
You can also drop useless features by adding a feature with random values and dropping any feature that has lower feature importance than that feature.

For this topic, I highly recommend the book: Feature Engineering and Selection: A Practical Approach for Predictive Models; Open-sourced here: http://www.feat.engineering/ And also available on Amazon.


HOW TO: Deep Neural Network weight initialization

As far as I know the two formulas you gave are pretty much the standard initialization. I had done a literature review a while ago, I copied it below if interested.

[1] addresses the question:

First, weights shouldn't be set to zeros in order to break the symmetry when backprogragating:

  Biases can generally be initialized to zero but weights need to be initialized carefully to break the symmetry between hidden units of the same layer. Because different output units receive different gradient signals, this symmetry breaking issue does not concern the output weights (into the output units), which can therefore also be set to zero.

Some initialization strategies:

[2] and [3] recommend scaling by the inverse of the square root of the fan-in
Glorot and Bengio (2010) and the Deep Learning Tutorials use a combination of the fan-in and fan-out:

for sigmoid units: sample a Uniform(-r, r) with $r=\sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
for hyperbolic tangent units: sample a Uniform(-r, r) with $r=4 \sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).

in the case of RBMs, a zero-mean Gaussian with a small standard deviation around 0.1 or 0.01 works well (Hinton, 2010) to initialize the weights.
Orthogonal random matrix initialization, i.e. W = np.random.randn(ndim, ndim); u, s, v = np.linalg.svd(W) then use u as your initialization matrix.

Also, unsupervised pre-training may help in some situations:

  An important choice is whether one should use
  unsupervised pre-training (and which unsupervised
  feature learning algorithm to use) in order
  to initialize parameters. In most settings
  we have found unsupervised pre-training to help
  and very rarely to hurt, but of course that
  implies additional training time and additional
  hyper-parameters.

Some ANN libraries also have some interesting lists, e.g. Lasagne:

Constant([val]) Initialize weights with constant value.
Normal([std, mean]) Sample initial weights from the Gaussian distribution.
Uniform([range, std, mean]) Sample initial weights from the uniform distribution.
Glorot(initializer[, gain, c01b])   Glorot weight initialization.
GlorotNormal([gain, c01b])  Glorot with weights sampled from the Normal distribution.
GlorotUniform([gain, c01b]) Glorot with weights sampled from the Uniform distribution.
He(initializer[, gain, c01b])   He weight initialization.
HeNormal([gain, c01b])  He initializer with weights sampled from the Normal distribution.
HeUniform([gain, c01b]) He initializer with weights sampled from the Uniform distribution.
Orthogonal([gain])  Intialize weights as Orthogonal matrix.
Sparse([sparsity, std]) Initialize weights as sparse matrix.

[1] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

[2] LeCun, Y., Bottou, L., Orr, G. B., and Muller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade.

[3] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." International conference on artificial intelligence and statistics. 2010.


What should be the criteria to select features using correlation factors between features?

There are several questions here, but let me try to answer the overall issue.  How do I know which features to drop or add to make the best model?

There is no rule that can be used to say “always drop all but one feature if X correlation exists”.  I usually start with > +/- 0.8 but you should test on a per dataset basis.
feature engineering, creating new features or eliminating features, is the art side of analytics.  Take into account what you know about the dataset, what you can learn from data analysis and experimenting.   Often all of the experiments don’t get captured in data kernels (especially those that don’t have positive impacts) so it just looks like magic to people reviewing the final/posted solution.


Factorization Machine - prevent over fitting

Here are some excerpts from the original paper that I think are key to understanding the question:

  Instead of using an own model parameter for each interaction,
  the FM models the interaction by factorizing it. We will see later on,
  that this is the key point which allows high quality parameter
  estimates of higher-order interactions under sparsity.

  Factorization machines can estimate interactions even in these
  settings (sparse data) well because they break the independence of the interaction
  parameters by factorizing them. In general this means that the data
  for one interaction helps also to estimate the parameters for related
  interactions.

In other words, instead of fitting an independent parameter for every second order interaction, it factorizes the parameters which reduces the parameter space and the model complexity, and thus making the model less prone to over fitting.

Hope this helps understand some aspects of the question.


Categorical and non-categorical data in the same column

If you provide more information about the detail of why there is no data for some data points that would make it easier.

That being said, I would split it into three columns as follows:

col_1: it includes 0 and 1 >>> 1 for those samples that have a value of -4 in the original column. For the rest of the samples: 0

col_2: it includes 0 and 1 >>> 1 for those samples that have a value of -1 in the original column. For the rest of the samples: 0

col_3: the real values of the original columns. If the value is -4 or -1, there are two options as follows:
1. Use 0 for those samples
2. If they are missing values replace them using one of the methodologies used for replacing the missing values.


Are there any very good APIs for matching similar images?

Never used this, but http://matelabs.in has something close... pretrained models on jewelry. You can probably extend those models. The new google dataset may also have labeled versions of the stuff you need. I cannot find the link I saw origionally, but it has bounding boxes for 20K classes down to 'teacup' 'teacup handle', 'tea', so I would not write google off before testing either.

Since this is DataScience SE, I will give the more DS answer as well. The thing you are looking for sounds like a multi-category classifier. You can probably find good resources and pretrained models here, and on the end, sklearn has apis for common classifiers.

The general idea is run your image through a Convnet, minus the classification part. You then have 1K-8K 'features' which a second algorithm set of classifiers, [decision forest](https://pdfs.semanticscholar.org/.../99dff11dcb3a48dee07a19052b07fdd2e7fe.pdf
), logical compisitional model, embedding, etc... gives a score for each binary attribute.


Identifying which known groups are the most similar or most dissimilar

You only have five groups, so full blown clustering is probably not a good idea here, but looking at similarity scores between the group vectors may be insightful. An easy one to try at first would be cosine similarity, which essentially measures the angle between each of your group vectors:

similarity = $cos(\theta)$ = $\frac{\mathbf{A} \cdot \mathbf{B}}{||\mathbf{A}||_2 ||\mathbf{B}||_2}$

where $\mathbf{A} \cdot \mathbf{B}$ gives the dot product of two vectors, and $||\mathbf{A}||_2$ is the magnitude of a vector $\mathbf{A}$.

All of your vector values are positive, so the result will be in the range $(0, 1)$. The closer this value is to $1$, the more similar the vectors. $0$ means they are completely decorrelated (vectors are orthogonal). You can compare the cosine similarities of each pair of groups to work out which ones are most similar/dissimilar.


Forcing a multi-label multi-class tree-based classifier to make more label predictions per document

You seem to experience a class imbalance situation where some classes dominate the others by the number of samples they have, so that your algorithm finds it wise to predict less of the rare classes or non-predict them to decrease the loss at the end. You can manually set the class weights for the Random Forest Classifier, making loss function treat unevenly to different classes, but even in total.

For details you can refer to:
https://stackoverflow.com/questions/20082674/unbalanced-classification-using-randomforestclassifier-in-sklearn

Note: Random Forest is not robust to class imbalance, this is a known situation. You can refer to: https://stats.stackexchange.com/questions/242833/is-random-forest-a-good-option-for-unbalanced-data-classification

Hope if I could help. If not, I will be around for further discussion.


Predicting what time will a John go to the gym tomorrow?

I would start by plotting some of the raw data. You might look at histograms of start times/stop times to see what the range of values are. You could also plot the start times as a time series to see if there are any long range patterns (i.e maybe as the week goes on John starts getting tired and showing up at the gym later).


Preparing training data for NLP machine learning task

Nlp like spacy remove stop works and identify proper nouns and nouns.  It does fairly well at identifying nouns but is not perfect.  Try a sample of your data and see how accurate your percentages become.  Use the medium model


What's the fastest way to do a text analysis over user reviews on a website for a beginner?

Welcome to DS.SE, @shubham.

Since the question is a bit general, I can just provide you some link to start with. Once you have more specific requirements, the community can help further.

An easy-to-use tool that I knew for sentiment analysis was IBM Alchemy which seems retired now in favor of IBM Watson NLU.

Despite IBM, other analytical platforms such also provide sentiment analysis APIs/tools. For example, see Google AI & Machine Learning Products
 and Deep AI. This article on Medium reviews some this tools in brief.


Is there any way to get samples in under each leaf of a decision tree in Sklearn ?

Very good and analytical answers are provided here and here.

For me, the easiest way would be to find the leaves where each sample belongs and then split the dataframe into clusters using iloc, to further analyse the clusters. This can be done using:

clf.apply(iris.data)

or

clf.predict(iris.data)


How to Identify Repeating Data Entries when the Repeated Entries are Spelled or Constructed Differently

For R: Have a look and the stringr package. I would use for example the str_detect() function as follows: str_detect(column_of_different_names,"DOE|company_name"). This will return TRUE for each string that includes "DOE" or the company name in "company_name".


How to handle invalid values like this is an extremely common problem in machine learning, since most datasets contain errors of some kind.

There are a few ways to do it. For example, you could set them all to 0:

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = 0

Or you could fill them with the avg(SoilHumidity), and create an extra feature to flag to the model that they were missing:

import numpy as np

df['SoilHumidityInvalid'] = np.where(df.SoilHumidity < 0, 1, 0)

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = df.SoilHumidity.mean()

Or, you can try to impute them somehow. Either by back or forward filling (I.E. taking the value from the next or the previous row in your dataset) or by creating a model that uses the other features of your dataset to predict what these invalid values should be.

The right method can depend; sometimes domain knowledge guides you (i.e. if you know the sensor can mistakenly read negatives when it should read 0, then you know to fill with 0). Failing that, I would just try a couple of methods and use cross-validation to see which improves your model the most.


Book about distances for data science (can't remember the name)

Not a book, but a webpage with Open Source code is the ELKI Data Mining library of distance functions.


How do I find the minimum value of $x^2+y^2$ with a genetic algorithm?

Genetic algorithm

This consists in 4 crucial steps: initialization, evaluation, selection and combination.

Initialization

Each individual in the population is encoded by some genes. In our case the genes represent our $[x, y]$ values. We will then set our search range to [0, 1000] for this specific problem. Usually you will know what is naturally possible based on your problem. For example, you should know the range of possible soil densities in nature. We will create 100 individuals in our population.

Evaluation of the fitness

This step simply asks you to put the $[x,y]$ values into your function and get its result. Pretty standard stuff.

Selection

There are many ways with which you can select parents. I will always keep the alpha male. The best individual in the population, he will be cloned to the next. Then I will use tournament selection. We will repeat the following until the next generation population is full. Pick four parents at random, take the best individual from the first two and the best from the last two. These will be the two parents which will gives us our next offspring.

Combination

From the two parents we will build the new genome for the child using the binary values of the $[x,y]$ values of the parents. The resulting binary value for each codon in the genome of the child is selected from the two parent genes by uniform random.

The code

class Genetic(object):

    def __init__(self, f, pop_size = 100, n_variables = 2):
        self.f = f
        self.minim = -100
        self.maxim = 100
        self.pop_size = pop_size
        self.n_variables = n_variables
        self.population = self.initializePopulation()
        self.evaluatePopulation()

    def initializePopulation(self):
        return [np.random.randint(self.minim, self.maxim, size=(self.n_variables))
                           for i in range(self.pop_size)]

    def evaluatePopulation(self):
        return [self.f(i[0], i[1]) for i in self.population]
        #return [(i[0]-4)**2 + i[1]**2 for i in self.population]

    def nextGen(self):
        results = self.evaluatePopulation()
        children = [self.population[np.argmin(results)]]

        while len(children) < self.pop_size:
            # Tournament selection
            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p1 = self.population[randA]
            else: p1 = self.population[randB]

            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p2 = self.population[randA]
            else: p2 = self.population[randB]

            signs = []
            for i in zip(p1, p2):
                if i[0] < 0 and i[1] < 0: signs.append(-1)
                elif i[0] >= 0 and i[1] >= 0: signs.append(1)
                else: signs.append(np.random.choice([-1,1]))

            # Convert values to binary
            p1 = [format(abs(i), '010b') for i in p1]
            p2 = [format(abs(i), '010b') for i in p2]

            # Recombination
            child = []
            for i, j in zip(p1, p2):
                for k, l in zip(i, j):
                    if k == l: child.append(k)
                    else: child.append(str(np.random.randint(min(k, l),
                                                             max(k,l))))

            child = ''.join(child)
            g1 = child[0:len(child)//2]
            g2 = child[len(child)//2:len(child)]
            children.append(np.asarray([signs[0]*int(g1, 2),
                                        signs[1]*int(g2, 2)]))
        self.population = children

    def run(self):
        ix = 0
        while ix < 1000:
            ix += 1
            self.nextGen()
        return self.population[0]

Then you can use the code by

f = lambda x, y: (x)**2 + y**2
gen = Genetic(f)
minim = gen.run()
print('Minimum found      X =', minim[0], ', Y =', minim[1])

  Minimum found      X = 0 , Y = 0

f = lambda x, y: (x-6)**2 + y**2
gen = Genetic(f)
minim = gen.run()
print('Minimum found      X =', minim[0], ', Y =', minim[1])

  Minimum found      X = 6 , Y = 0


ML in R (caret-package) missing hyperparameters

It appears that the defaults are used, except for lin, which is inferred from the type of the target variable: [source code]

Note too that you can set any of the other RSNNS parameters through the dots.


How to deal with a binary classification problem, where the instances in the negative class are very similar?

The CNN in this case does not learn what is a cat but rather what differentiate an image with a cat from one without a cat.
If all of your "no-cat" images contain spider(s), the CNN could also converge only by detecting the spider(s) : images with spider(s) belong to "no-cat" and others belong to "cat". That explains why you have such a good confusion matrix on testing data and such poor performance in real-world.
You have some options to avoid such a situation:

add other images to your "no-cat" label as you said (using public datasets for instance)
try one-class classification approaches. If you are familiar with neural networks, you could dig into reconstruction error of autoencoders (see this post: How to use a dataset with only one category of data or this sklearn guide)

Hope it helps


Linear Regression. Interpret weights as an effect on a prediction?

In a basic linear regression, can I use the weights of each
  explanatory variable to describe their relative effects on the
  predicted value? If parameter A has a weight of 100, and parameter B
  has a weight of 10, can I say that parameter A has 10 times more
  effect on the outcome than parameter B?
  Blockquote

Somewhat yes. Precisely it means if A changes by one unit the outcome will change 100 times, on average (while B is frozen because it is a multiple linear regression -> you have more than one independent variable).

There are tons of examples and blogs explaining it, see for example here .
Also you can analyze your regression model more by doing some statistical tests like identifying how variables are correlated using Variance Inflation Factors and estimate the statistical significance (p-value, F-test), here is a very nice intro.

  An example: From a weather data set, I used two parameters, humidity
  and pressure, to make a prediction on temperature.

I do not see any problem with your analysis. As you said you see an inverse correlation between humidity vs. temperature in your plot as well as via your -.32386 weight. While the weight of -.02219 for pressure is nearly zero and negligible and obviously you won't see such relationship between pressure-temperature.


Machine Learning - Range of Hypothesis space and choiceof Hypothesis function type

I will answer your question as I understand them, the clarity you are looking for comes from the Maths behind ML.

There is no fix range of values of the coefficients or weights. Finding the value of these weights is the work of ML algorithms. although using regularization you can reduce the value of these weights but range depends on algorithm. no. of weights you can decide
Yes, formula is loosely fixed for a ML algorithm.
e.g. for linear regression in your case

h(w,x) = w0*x0 + w1*x1 + w2*x2 +w3*x3 (where x0 =1)

the weights will change after each iteration though. for neural network it is multiplication of a weight matrix with an input matrix for each layer.
The output of h(w,x) is the prediction of ML which you can compare against y.

All the ML algorithms available are your hypotheses space(H)

e.g., quadratic,

linear equation(the one above),

high degree polynomial,

complex neural network.


Cross-validation can be used to tune any hyperparameter.
One way tune Extreme learning machines (ELM) with cross-validation is to:

Fit a model to a subset of the training dataset. Measure performance on a validation dataset with a evaluation metrics.
Change the number of neurons hyperparameter.
Fit the updated model to a subset of the training dataset. Measure performance on a validation dataset with a evaluation metrics.
Compare the two models and see which model is better on a evaluation metric on their respective validation dataset.
Repeat steps 3-4 until satisfied.


What does it mean if a high or low number of my componenets describe a percentage of the cumulative explained variance?

I always think that Principal component analysis is a very interesting tool. Many know its applications, few knows exactly what is going on.
If you care about the math, you need to study eigen values and eigen vectors, but here let me explain pca in a very simple way. Imagine that you have a point in a 2-d space (x, y), this same point can be represented by using another two values, let us say (r, theta), where r is the distance from the origin and theta is the angle with respect to the +v x-axis.
Given any point (x,y), you can find the point (r, theta) corresponding to the point (x,y). You should be able to move between the two coordinates easily. Now let us ask the following question:
If I decide to ignore the theta dimension (for example set it to zero) how many points in the (x,y) you still can recover? All the points across the +ve x-axis can be recovered correctly. Now what about the r dimension, if we set it to zero, we can recover only one point which is the origin point.

Principal component analysis helps find this new coordinate system depending on the data, number of dimensions and variation across each dimension. The higher variation across dimension the more important to reserve this dimension. Very important to know that pca proposes both new coordinates and a measure of how much each dimension is important to recover the points in the original coordinate.

For example, let us take these points,
(1,1)
(2,2)
(3,3)

Both x and y dimensions are important and they experience the same variation.
If we transform, these data points into a new coordinate system (k,m), such that
K=x.  And.  M=0
Then. K is very important 100% and m is not important at all 0%
You can recover the original points by using, x=k. And y=k. Note that the variation across the m dimension is zero and it is high across the k dimension.

How much you want to reserve depends mainly on the application, usually > 95% is good.
For the second question, it is possible to recover less than 100% when you use less dimensions, but when you use 200 of 200 you should be able to recover 100%.


Unsupervised text clustering using a driving list

Your direction of using k-means on term-document matrix is correct. To go further, you can

Use Weighted k-means

You can also try artificially boosting the weight for certain columns/variables (corresponding to your pre-specified list of words). This will force the similarity calculation to be dominated by those words.

Use Constrained k-means

You can read up more about constrained k-means algorithm: you can force certain points to be in the same cluster. In other words, if there are two documents containing words from the pre-specified list, they are forced to be in same cluster.

Alternative: Classification

If you know that there are only two clusters to be formed, why stick to only unsupervised learning? Depending on the application and availability of labelled data, you may think of it as a binary classification task using nearest neighbor or similar algorithm.


The type of ML problem you are trying to solve is a regression problem.  Essentially, given your attributes, predict a continuous variable.

There are multiple ways to solve your problem, but (assuming attributes are numerical) they all boil down to "plot your data points, then draw a line through them".  The different methods calculate the placement of this predictor based on various different measures of error; the most basic probably being least squares, which aims to minimize the sum of squared errors.

Metacademy is a great website for learning ML related stuff online, because they provide roadmaps that tell you what you need to learn before learning ______, as well as how to learn those prerequisites.

Here is a link to the metacademy page for linear regression, which is where you're going to want to start.  I'd recommend watching the coursera videos from Andrew Ng.  If you're just getting started with ML, his course is a wonderful introduction.  However, a common complaint for the coursera class is it's too shallow, and is to be considered as an overview of machine learning.  If you consider yourself a little more mathematically savvy, I hear the Stanford course (also taught by him) may be a more in depth resource for you.

You should implement your own basic linear regression at first, just to get a deeper understanding of how it works.  Afterwords, you can just use a python ML library called scikit learn to perform the regression (link here).  I strongly recommend you take this approach for all ML algorithms; strive to understand at least how the algorithms work at a basic level.  It is far too easy for someone just starting to learn about ML to see all the implemented algorithms in scikit learn go on to use the algorithms as black boxes without understanding what the algorithm is actually doing.

Lastly, in order to evaluate the different methods, you should split your dataset into two different parts; a training set with which to train your models on, then a testing set with which to test how accurate the predictions are.  The idea here being that your data is representative of the "wild" data that you'll see, so by purposely hiding some data, you can effectively simulate how the algorithm will perform in "real world" conditions.

Generally, people go with something along the lines of 70% of your data being used to train, and 30% of it being used to test.  Later on, as you get into complicated models with multiple parameters, it will help to split your data into 3 parts: training, used to train your model, validation, used to adjust model parameters, and finally, a test set, used to measure how accurate the model is.

Good luck!


Yes,  it is technically possibly. The easiest algorithm you can try out is siamese networks. Basically siamese networks try to learn a similarity measurement of two inputs (images, sounds, whatever) and work quite well for problems with little labeled data.
You can also find hands-on tutorials guiding you through the details like tutorial 1 or tutorial 2.

For more advanced check out prototypical networks.


Creating a loss function with two minima?

You basically don't want to have a loss like that in any case.

The objective of Machine Learning is define a loss function that is convex to find the minimum by approximating it with the given data.

Not having a convex loss function means that you will have multiple minima and won't be able to determine if the one you're in is a global minimum or just a local one so you won't know if it's optimal or not.

The only kind of Machine Learning without convex function is Deep Learning. And you see how complicated it is to work with it and having "good results" without understanding what is happening under the hood.

Deep Learning models are (most of the time) stochastically with batch gradient descent, that helps getting out of local minima, but that's not even easy.

We can't even prove mathematically that these systems are well-optimized.

So basically, if you want a good model, you can avoid creating such a monster loss :)


Classification model for recommender system?

One option could be to model the output slightly different. Instead of returning the type of product 1, 2, ..., n or 0 in case of no product maybe you could return a list of pairs (product, probability), if the highest probability is the one corresponding to 0 i.e. "no product" you could just recommend the 2nd best.

An advantage of this approach is that you could recommend more than one product, maybe the top 3, or a list of products whose probability is higher than a specific threshold, etc.

If the customer is new and there is no information you could base your recommendation on, maybe you could recommend the product that you would offer to a similar customer (if you can cluster the customers in any way) or simply the top selling products.


How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans

It sounds like you are grappling with large data set sizes, for which I first suggest switching to mini-batch k-means. Mini-batch scales better so will be less frustrating.

Regarding apriori estimates of the inertia_, I suggest using a sample data set to approximate the inertia_ with appropriate margins of error. But, mini-batch may just preclude your need for apriori inertia_.

Hope this helps!


Interactive dashboard for time series data

D3 (data Driven Documents) is a great and powerful tool, but does require a certain afinity for JavaScript. There are a few tools that piggy-back off it though and are more friendly to the aspiring data scientist and plotly is behind a few others. All are easier to use compared to D3 in my opinion.

Check out a few of the following:

Bokeh (python)
Dash (python - based on plotly, looks very promising!)
Plotly (many languages)
Shiny (R)

Here is a comparison of Bokeh and Dash.

Here is a related question on StackOverflow.


How to choose between discounted reward and average reward?

The Average reward in that figure is used as a measure of performance. In other words, the score of the agent playing the game. You do not track reward per episode as this doesnt indicate a general improvement in the learning process. Instead you track the average reward over training epochs. If it steadily increases this means that your agent indeed is learning.

The discounted reward is used to create some kind of future-reward dependencies and is used in the learning equations. So, instead of evaluating how good is a particular state according to the immediate reward you received, you also take into account the future reward from your next state. In RL you attempt to max your expected return and some methods estimate the expected reward from every state.

Please note that my answer gives a high level description and is not referring to a specific RL algorithm (as there are many variations). I would suggest you to understand very well the simple tabular form of Q-learning before moving to RL and function approximators combinations.


How does one deploy a model, after building it in Python or Matlab?

what comes after training a model is persisting (in layman terms, saving) a model, if you want to save the parameters for later deployment.

All companies that do serious machine learning spend many hours (sometimes even days or weeks) training the models. Obviously, when you need rapid predictions in a deployment mode, you use a pre-trained model to generate said prediction.

In python, for example, there are a number of way to persist models. The most common libraries used are pickle and cpickle (i/o much faster I believe as the core is in C).

Here is a link to model persistence in sklearn:

http://scikit-learn.org/stable/modules/model_persistence.html


The memorisation capacity of an LSTM (real numbers)

Welcome to the site! If you're referring to a series of numbers, like what you would get during tokenization/NLP then, yes, LSTM can certainly handle that without many issues. If you are talking about a much larger range than that, then you might want to consider a scenario where you scale your inputs instead.


Detecting hand-written objects from images

You could try to run some OCR like tesseract. It'll give you locations of detected text. It'll probably fail for handwritten and skewed text. You'll get your mask, though this is a bit hacky as some false negatives may appear.


ML questions are concerned mainly around predictions, but we can extrapolate to (in a certain sence) causality.

First of all these are two different modelling approaches:

Causal inference is focused on knowing what happens to $Y$ when you change $X$.  Prediction is focused on knowing the next $Y$ given $X$

Some of the current causal approaches are randomised testing, do-calculus etc...

So how can we extrapolate to causal inference of standard predictive ML models?

Counterfactual Explanations we can simulate counterfactuals for predictions of machine learning models where we simply change the feature values of an instance before making the predictions and we analyze how the prediction changes. Read more about it here, and there is python library called alibi that implements it.


Calculating the average of gradient decent

Starting from the last part, as the entire dataset is used, number of epochs(run over entire dataset) equals number of iterations. Instead, one can do the calculation in "mini batches" (of 32, for example), then the run over each 32 samples is called an iteration.

As for the rest of the question, you can chose a batch that is equal to the entire dataset - this is called "batch gradient descent"; or update after every single sample (a batch size of 1) which is "stochastic gradient descent". Any other choice is called "mini-batch gradient descent.

Deep Learning course on Coursera offers a relatively better explanation of these matters compared to Nielsen's book or 3B1B videos. You can watch the videos for free. In particular here is the video on  Gradient Descent.


I would say depends on the requirements and how much effort you want to give the task.

Using regex is definitely easier but in the some time you will not be able to cover everything specially if the text you have is not structured as you expect all the time. So in this case you will be updating you regex patterns each time you miss something ...
If you go for machine learning approach then you will need data to train your models on it ( a lot of this data). Time for training, and enhancing the quality of your model. Hopefully you will get something good.

As a conclusion, I think, if you can cover the requirement with regex go for it. If regex will not be a good solution then start thinking in machine learning solutions.


I am not getting classification output by predict_generator()

m.predict_generator returns a numpy array. you can take the maximum probability of a class by numypy argmax function or use predict_classes. This link might help-

How to predict class label from class probability given by predict_generator for testdata?


Why do train, test, validation datasets need to have the same distribution?

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. Further you have to estimate its performance in this world by testing it on more examples from this world. If you train your model in one world, test and validate on another and deploy it into a third, you have no idea how well the model was trained for the real world or if it was trained for the real world at all.

Let's look at a little example. I created a little toy data set and ran regression with the same network, same optimizer and everything the same, except the initialization.

All 5 models give the same prediction in the domain where there is data (they even over-fit a little bit in the same way). But for x-values far away from the training data, we get widely different predictions from different models.

So what does this tell us? It tells us that our models are basically worthless for inputs that are "far" from the training data. "Far" means "from a different distribution" in this sense. But if our models are worthless for inputs that do not come from the original distribution, then any quality measures based on test and validation datasets are also worthless.

I used the word "far" a bit loosely here, basically as if I was talking about a distance. I think this is somewhat correct, but it's not the whole story. Even if the support of two distributions is the same, they can be "far" from each other if one distribution puts a lot of weight on events that the other only weighs little.

This is also a problem, because this would mean you get many samples from one distribution where the other gives only a few. The amount of training data you have for a particular input domain will influence how well your model generalizes to this domain. If now in the test or validation set you shift the bulk of the distribution, you basically over-emphasize input values that the model has hardly been trained for, while at the same time hardly testing it in domains that it has been trained for well. So again your measure of model performance does not really tell you how well your training worked.

This is why training, test and validation dataset should stem from the same distribution, but also the inputs once the model is deployed need to be from the same distribution, otherwise your quality measures and your predictions will be highly inaccurate.


I think you answered yourself by "values of B are not comparable". Learning for predictions is based on a fundamental assumption which is the data for prediction has the same joint distribution as the data for learning. This is the link between those processes.

Now, if you want to handle that in a meaningful way you have to know somehow the source type. In your example the device type. One way would be to introduce the device type as a different column in your data set, so that the model can have the chance to differentiate between source type. Obviously you have to have training data for all the device types. Supposing you have 2 device types, A and B. Your training data should have some columns for the signal and also a factor column with type A and B. Also you have to have enough instances of data with type A and type B.


high accuracy on non trained class in tensorflow model

It is normal because the model has never seen such data so the probability for such unseen data is like random, it can be high or low.
You should add all other images (like the ones of tomato) into a new dedicated class OTHER where you will put everything which is not in the 4 original classes.
That way, for an image of tomato your model will give a high probability for the OTHER class, and low probability for the apple classes.


Implementing back translation as a data augmentation for text classification

One option, which I have discovered, is back-translation via the Unsupervised Data Augmentation repository made public by Google Research. This is based on this paper.


bad regression performance on imbalanced dataset

Your residuals are huge, which is not surprising, given that your data is very variable, a linear model may not be the best choice for this task. You could try transforming your data (log, sqrt) depending on the nature of your data to reduce the variability, but as I said, your variability is huge.

Alternatively you could try modeling the variance with a mixed model if it makes sense for your data, given some additional knowledge of some variable.

Other then that you could try a different algorithm for this task.


Statistical machine translation word alignment for FR-ENG and ENG-FR: what is p(e) and p(f)?

You are right that $p(e)$ is the probability of the English sentence. Estimating the probability of a sentence is achieved by a language model.

This kind of machine translation model is known as the noisy channel model. The noisy channel model says that given a french sentence $f$, its best English translation is

$$e^* = \arg\max_{e\in E} p(e)p(f|e)$$

In this equation the $p(e)$ is the language model. Back in the era of IBM models (which are built upon the noisy channel approach), it is usually an n-gram based language model, calculated as (assuming bigram) $$p(e_1e_2...e_n)=p(e_1|<s>)p(e_2|e_1)p(e_3|e_2)...p(</s>|e_n)$$

And $p(f|e)$ is the translation model where you need to use the EM algorithm to solve. Inside the EM algorithm you do not update the language model parameters, so yes, $p(e)$ and $p(f)$ don't affect the optimization problem.


Perform k-means clustering over multiple columns

There is no difference in methodology between 2 and 4 columns. If you have issues then they are probably due to the contents of your columns. K-Means wants numerical columns, with no null/infinite values and avoid categorical data. Here I do it with 4 numerical features:

import pandas as pd
from sklearn.datasets.samples_generator import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=10, centers=3, n_features=4)

df = pd.DataFrame(X, columns=['Feat_1', 'Feat_2', 'Feat_3', 'Feat_4'])

kmeans = KMeans(n_clusters=3)

y = kmeans.fit_predict(df[['Feat_1', 'Feat_2', 'Feat_3', 'Feat_4']])

df['Cluster'] = y

print(df.head())

Which outputs:

     Feat_1    Feat_2    Feat_3    Feat_4  Cluster
0  0.005875  4.387241 -1.093308  8.213623        2
1  8.763603 -2.769244  4.581705  1.355389        1
2 -0.296613  4.120262 -1.635583  7.533157        2
3 -1.576720  4.957406  2.919704  0.155499        0
4  2.470349  4.098629  2.368335  0.043568        0


Clustering algorithm for a distance matrix

There are hundreds of algorithms to choose from.

Hierarchical clustering in it's myriad of variants. Cut the dendrogram as desired, e.g., to get k clusters
PAM, the closest match to k-means on a distance matrix (minimizes the average distance from the cluster center)
Spectral clustering
DBSCAN
OPTICS
HDBSCAN*
Affinity Propagation
...


Is a good practice to sum each rate (i.e. crime rate per 100.000 people)?

This can be fairly good if you deal with few years.

In general it's not a good practice, since asymptotically will lead you to have almost the same fraction of crimes per all the considered years. But unless you have data from the last 10000 years, you can simply ignore the asymptotic behavior a continue the way you are doing.

Otherwise, you can normalize the data to a reference value, say the maximum number of crimes on the series. This is more robust.


Getting negative r2_score with new set of dimensions

A negative $R^2$ on your training set typically means you didn't fit an intercept. A negative $R^2$ on your test set means the model is simply very bad. Indeed flight ID is not a meaningful scalar value and can't be in a linear regression.


Yes, that is a reasonable approach. Also try neural network based representations such as doc2vec. I suppose you know how to do the classification part?


A confusion matrix is indeed a very useful way to analyze the results of your experiment. It provides the exact number (or percentage) of instances with true class X predicted as class Y for all the possible classes. As such it gives a detailed picture of what the system classifies correctly or not.
But a confusion matrix is a bit too detailed if one wants to summarize the performance of the classifier as just one single value. This is useful especially when one wants to compare two different classifiers, since there's no general way to compare two confusion matrices. That's why people often use evaluation measures: for binary classification, the most common ones are:

Accuracy, which is simply the number of correct predictions divided by the total number of instances.
F-score, which itself is the harmonic mean of precision and recall. Quite ironically, F-score gives a more accurate picture of performance than accuracy because it takes into account the different types of possible errors.


How to benefit Data augmentation when it yields to different classes

You could rotate images manually (without using ImageDataGenerator) and save it to disk.
That way you would know which images you have rotated - so you would know which images have changed the class.

After it, when using ImageDataGenerator, you need to set rotation_range to small value in order to be sure that it won't change the classes of images.

Examples how to rotate image manually can be found here: https://stackoverflow.com/a/43864937/5997950


How can I predict the acceptance of an article by publisher?

Applied Predictive Modeling book has a case study of acceptance rate of grant proposals, which was a Kaggle competition. You can get many good ideas from there.

from the dataset, what kinds of features you would look at
from the book, step-by-step explanations of how to build models
from the Kaggle competition, approaches by the contestants

I think it is very hard to predict the quality of the paper from the text of the paper, but you might have much more luck by including features such as academic rank of the author(s), their past publishing history and success, acceptance rate of the journals the paper was submitted to, and things such as even the length of the paper. Such proxy features for paper quality actually might be very successful...


SGDClassifier : Get the weights and bias for each training step

You can use the partial_fit method to get the results of a single training epoch for sklearn.linear_model.SGDClassifier.


Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

Both negative sampling (derived from NCE) and sampled SoftMax use a few samples to bypass the calculation of full SoftMax.
The main problem comes from this comment in the linked pdf:

Sampled Softmax
(A faster way to train a softmax classifier)

which is only used for sampled SoftMax, although, negative sampling is as fast for the same reason that is working with few samples. If their performances are at the same level, this could be the reason why researchers are not convinced to switch over to sampled SoftMax. In academia, it is almost always the case that older methods are preferred over new, but equally-competent methods for the sake of credibility.
Negative sampling is NCE minus the logistic classifier. Roughly speaking, it only borrows the term "F(target) + sum of F(negative sample)s". Negative sampling is most prominently introduced in the Word2Vec paper in 2013 (as of now with 11K citations), and is backed by the mathematically rigorous NCE paper (2012). On the other hand, sampled SoftMax is introduced in this paper (2015) for a task-specific (Machine Translation) and biased approximation:

In this paper, we propose an approximate training algorithm based on
(biased) importance sampling that allows us to train an NMT model with
a much larger target vocabulary

Noting that negative sampling also allows us to train "with a much larger target vocabulary".


Best format for recording time stamp and GPS

As Spacedman put it, "best" is pretty subjective. However, as we have found, a good format for time is Unix time (aka POSIX time, aka Epoch time). Most databases support it and it is still pretty human readable.

For location, we like decimal degrees as it is easy to read and stored and is compatible with Google Maps API. It's also easy to convert to other formats if needed.


Will oversampling help with generalization (small imbalanced dataset)?

The bigger issue might be the small n. With 60 samples and 2:1 ratio, you only have 20 samples in the minority class. Generalization, no matter what machine learning technique is used, will be limited with just 20 samples.