What does it mean to "share parameters between features and classes"

I will try to answer this question through logistic regression, one of the simplest linear classifiers.

The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,1\})$and only one input feature ($x \in R$). In this case the output of logistic regression would be:

$$
\hat y = σ(w \cdot x + b)
$$
where $w$ and $b$ are both scalars. The output of the model $\hat y \in [0,1]$ corresponds to the probability that $x$ will be of class $1$.

We'll try to break down the phrase "linear classifiers do not share parameters among features and classes" into two parts. We will examine the cases of multiple features and multiple classes separately to see if logistic regression shares parameters for any those tasks:

Do linear classifiers share parameters among features?

In this case, for each example, $y$ is a scalar that takes binary values (like before), while $x$ is a vector of length $N$ (where $N$ is the number of features). Here, the the output is a linear combination of the input features (i.e. a weighted sum of these features plus the biases).

$$
\hat y = σ \left(\sum_i^N{(w_i \cdot x_i)} + b\right) \;\; or \;\; σ( \mathbf w \cdot \mathbf x + b)
$$
where $ \mathbf x $ and $ \mathbf w $ are vectors of length $N$. The product $\mathbf x \cdot \mathbf w$ produces a scalar. As you can see from above there is a separate weight $w_i$ for each input feature $x_i$ and these weights are independent by all means. From this we can conclude that there is no parameter sharing among features.

Do linear classifiers share parameters among classes?

In this case $x$ is a scalar, however $y$ is a vector of length $M$ (where $M$ is the number of classes). To tackle this, logistic regression essentially produces a separate output $y_j$ for each of the $M$ classes. Each output is a scalar $y_j \in [0,1]$ and corresponds to the probability of $x$ belonging to class $j$.

$$
\mathbf{ \hat y} = w \cdot \mathbf x + \mathbf b, \;\; where \;\; \mathbf{ \hat y} = {\hat y_1, \hat y_2, ..., y_M}
$$

The easiest way to think of this is as $M$ simple independent logistic regressions each with an output of:

$$
\hat y_j = σ(w_j \cdot x + b_j)
$$

From the above it is obvious that no weights are shared among the different classes.

multi-feature and multi-class:

By combining the two cases above we can finally reach the most general case of multiple features and multiple classes:

$$
\mathbf{ \hat y} = σ( \mathbf W \cdot \mathbf x + \mathbf b)
$$
where $\mathbf{ \hat y}$ is a vector with a size of $M$, $\mathbf x$ is a vector with a size of $N$, $\mathbf b$ is a vector with a size of $M$ and $W$ is a matrix with a size of $(N \times M)$.

In any case, linear classifiers do not share any parameters among features or classes.

To answer your second question, linear classifiers do have an underlying assumption that features need to be independent, however this is not what the author of the paper intended to say.


How can I do classification with categorical data which is not fixed?

It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience:

Avoid one-hot-encode as much as possible (contrary to what was suggested above). The reasoning is that it won't work. A model with one-hot-encode features only works when all those sublevels had been existed in the training data. The model won't be able to do prediction, unless somehow it is manually tweaked. If you search you will find many people ran into this issue when splitting their data into train/test, and faced the issue of some sublevels of a particular feature were not present in the training split and subsequently failed to do prediction on the test. Said aside, if you have very high cardinal features (maybe like your city with let's say 200 city names), this will increase the dimensionality of your data unnecessarily! If for some reasons you may need to do one-hot-encoding, just keep these in mind.
Use Other Encoding Methods. Maybe try learning more about other methods that are robust to this issue, at least for the time being like target-based coding, hashing (see some references below). If you are with Python there is a nice package offer mant encoding options. You may be surprised to see that other simple methods often works just fine.
Retrain your model. Theoretically when learning your train/set should have had the same distribution (mostly this is thought of as target distribution, but can be true about variables as well). Now with new items comes into play, your test (unseen) data distribution has changed. Then it is best to retrain the model again so that those new cities will be accounted for.
Put Newly Added Subcategories (and least frequent ones) to Others. While earlier point is true theoretically, it is very likely that the test distribution (of that particular category) won't change that drastically in most of the cases e.g. very few items goes top of the categories in the training set. Perhaps like in your case, you may have 100 cities in the city feature, and just very few new ones comes over time. What I would consider would be looking at the let's X-quantile of that particular category, and put the least frequent ones into Others subcategory. Assuming your newly added data point is only little, it will very much goes into the Others group. You will certainly lose a level of granularity by doing this one, but once again the point of learning is that not only your model learn the training data, but most importantly to be able to generalize very well on unseen data and if those new added categories are very data points, grouping them altogether into the Others group won't hurt.
Other Recent Not-Yet Mature Solutions like Cat2Vec (borrowed from Word2Vec from NLP) or Similarity Encoding. These are very recent, check the paper for the former and its github and an example (based on Word2Vec) in Kaggle, and this paper for the latter and its implementation. The idea of the former is to convert categories to vectors. As much as I have to say it really make sense to work, but I have no experience using it. The latter, so-called dirty_cat, looks quite promising and easy to use. Whether it is robust to having unseen cardinal category in your test data is not clear to me!

P.S.: I would like to add that the idea of city to a geographical location given in the first comment is really nice and it is actually not complicated as they are many Python API e.g. by Google or HERE that allows you to do that. But it is noted that this is just a way to engineer new features and certainly not to be replaced by the city feature itself.

Interesting references to check first, second, third, fourth (no particular order!)

All above-mentioned points are practical solutions rather concretely theoretically correct, and surely subject to further discussions. And I am more than happy to learn more.


How to add jpg images information as a column in a data frame

I assume you a list of the filenames called movie_images

# Could get filenames with:
# import os;  movie_images = os.listdir("./folder/with/images/")
movie_filenames = ["11_lion_king.jpg", "22_avengers.jpg"]

First create a mapping between the ID values and the filenames:

# Use the "_" to split the filename and take the first items, the ID
mapping = {f.split("_")[0]: f for f in movie_filenames}    # <-- a dictionary-comprehension

Now add a column of some empty values (whatever you like) that will hold the movie_image values:

data_movie["movie_image"] = pd.Series()    # will be filled with NaN values until populated

Now iterate over this mapping, inserting the movie filenames for the correct movie IDs:

for movie_id, movie_image_filename in mapping.items():
    df.loc[df.movie_id == movie_id, "movie_image"] = movie_image_filename

This should produce the output dataframe you described.

As a side note (in case you are ever tempted): never load the actual images into a  pandas dataframe. It is best to load them as NumPy arrays or something similar. Pandas DataFrames are in essence just annotated NumPy arrays anyway.


Is there potentially data leakage during imputation for time-varying sensor data?

If you are using information from the future to impute missing data would be data leakage as you would not have this extra information when the model is in production and trying to predict future values. To prevent data leakage, make sure to only use values that are available at the date/time you want to predict. If you were to impute the missing data based on historical data you would not be leaking data since you have this data available at the moment of prediction (as the term "historical" implies).


Looking for a machine's breakdown dataset

I would recommend the NSSDC NASA data of 550 space spacecraft

https://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html
https://pds.jpl.nasa.gov/


What does the number after a machine learning model name mean?

Sometimes it refers to a version (like windows 10), sometimes it refers to a size of the parameters, like the size of a layer, the number of parameters (like for GPT models), or the size of parameters in memory. They are just names so we can know what we are talking about. I don't think there is a general convention.


Classifying based on irregular number of features

The simple approach is:

To use a learning algorithm which can handle missing values. You should be careful about whether these missing values occur randomly or not though, as they could cause the model to be biased.
Triplets can be separated into individual features, there's no simple way to process them differently anyway.


I don't think there is a standard method to do this. But if you use a probabilistic model you can use the predicted probability together with thresholds on each class to only allow classifications that you deem certain enough. Then if the class with the highest probability does not meet the threshold you can set it to the default class.

I tested it out with sklearn and numpy, this could be an approach:

# Train probabilistic classifier
clf.fit(X_train, y_train)

# Get probabilities
probas = clf.predict_proba(X_test)

# Get the class with highest probability
highest_proba_class = np.argmax(probas, axis=1)

# Set different thresholds per class
thresholds = np.array([0.9, 0.2, 0.5])

# Init our prediction array
predictions = np.zeros_like(highest_proba_class)

# Set a default class to set if we don't reach threshold
default_class = 2

# Loop over predictions
for idx, highest_class in enumerate(highest_proba_class):

    # Threshold check if threshold was met, otherwise set default
    if probas[idx][highest_class] >= thresholds[highest_class]:
        predictions[idx] = highest_class
    else:
        predictions[idx] = default_class


Why can we not split train test data with 0.01 as parameter or 99% training data

The idea of splitting 80%-20% is to use as much data as possible for training while keeping enough (labeled) data in the test set in order to reliably evaluate the performance. It's fine to use a smaller test set if it's big enough, but if it's too small the evaluation might not reflect the real performance of the model.

Your idea of using future test data is fine, it simply means that the reliable evaluation is delayed until you obtain enough test data. In the meantime you cannot be sure that your model performs well, so I would avoid using it for real trading on the stock exchange ;)


What is the point of getting rid of overfitting?

Basically, prediction error can be decomposed into three terms, two of which that you can control. MSE (mean square error) = Variance + Bias^2 + irreducible error. A model with high variance implies that small changes in the training dataset yield large changes in the fit of the model (in this context). A model with low bias essentially fits the training dataset extremely well. Irreducible error is the difficulty inherent to the problem itself (and therefore, not controllable) so lets ignore that.

In an ideal world, we would obviously want low variance AND low bias. However, this in reality is extremely difficult to do because intuitively, the two forces of variance and bias work against each other. The reason why is that I can arbitrarily make my model fit the data extremely well (and therefore, have low bias) by simply increasing the complexity of the model, modelling every quirk and essentially do what you refer to as "memorizing the dataset". You see, data are realizations of random variables that can potentially take on many different outcomes and therefore, there is noise in the dataset. Hence, modelling the exact pattern/trend in the dataset is not going to generalize well because the data itself is not a perfect representation of the population (which is our goal) but a sample of it.

By increasing the complexity of my model to arbitrarily improve the fit of the model on the dataset, I have effectively traded away all of my bias for large amounts of variance. I did this because clearly, by increasing the complexity of my model to the point where my model has even memorized the noise in the dataset, a relatively small change in the dataset with different observations (another random sample from the same population) will change the fitted model drastically. This is because noise is inherently random, not predictable and unique to that specific dataset (sample) only, not the general population from which the data comes from. As a consequence of my model changing a lot with my dataset that I trained it with (what many refer to as an unstable model), my models predictions will inherently be very random as well and therefore, my models performance on new held out data will be no better than say, a random guesser. This is what people mean when they say their model is "overfit". I've fit the training data set perfectly, at the cost of not modelling the actual signal from the data but a bunch of noise unique to the training set only.

So what people have figured out is that we need to make a tradeoff between fitting the training data extremely well whilst also being mindful of not modelling noise. Thus, stuff like dropout layers in NN, L1/L2 regularization, pruning in trees, learning rates, early stopping, and cost complexity penalties all work in purposely sacking some bias (fitting the training set not as well) in hopes of reducing variance (having a stable model and therefore, stable non random predictions). Striking the balance between the two is in essence what predictive modelling is about and all ML methods try to address this problem in their own way.

In your example, removing overfitting would imply moving the validation accuracy closer to the training accuracy, not the other way around. But of course, this is typically a pipe dream and you will almost always overfit somewhat. The degree to what you find acceptable, given the problem at hand, is key. If I have a training accuracy of 0.99 and a validation accuracy of 0.89 in many contexts I would still be satisfied with the model, for example.


Why training model give great result but real data gives very bad result: Azure ML Studio

Your question is not clear. There's 2 ways to understand it. Which dataset did you use to train your model?

You trained and tested on a premade dataset. The result  is great. Then you applied this model to real dataset and the result is really bad.

If this is the case, you should retrain on your real dataset or apply some Transfer Learning techniques to your current model.

You trained and tested on a premade dataset. The result is great. Using the same model, you trained and tested on real dataset but the result is much worse.

I can't tell exactly the reason for this. Normally, real data is much more noisy. Did you handle missing data and do some feature engineering before training?


Where do I find the .csv file for the submission after creating it?

When you are done with Colab instince files saves will not persist. You have to configure saving them to google drive. Try something like this;

  from google.colab import drive
  drive.mount('/content/gdrive', force_remount=True)
  root_dir = "/content/gdrive/My Drive/"
  base_dir = root_dir + 'fastai-v3/'

so now you can access and save files to google drive directly.


Search for hyperparameters whith different features using Random Forest

By performing GridSearch I understand you want to say searching for the best hyperparameters.

For sake of simplicity, let say that you want to fit a linear regression with a penalty (lasso/ridge) with 1 feature and with 100 features. The hyperparameter that you are looking for is the $\lambda$ penalty.

It is easy to see that with 1 feature your model might need some parameter, it could be that the feature has a perfect distribution and it doesn´t need any penalty at all. But when we go for 100 features there is some noise and you might need penalties to make sure that your model is generalizing well. So $\lambda$ will be different

With this example, my idea is to show that hyper-parametrization is a specific problem for each task.

For Random Forest, it would be the same. Different features will require different parameters so yes, you have to do a GridSearchCV with every subset of features if you want to achieve optimality


What methods are available to evaluate similarity between different clustering algorithms?

If you just want to see how similar the clustering is between 2 algorithms, using the sklearn.metrics.adjusted_rand_score() function is a good starting point. This will work for unsupervised learning, no need for a label.

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html

Or are you looking at choosing the best combined grouping overall? I think "clustering the cluster" results is not a common approach, however there are a few articles online on how this could be done. But I don't know of any packages that will do this for you.
One idea is to create a new column for each algorithm with the grouping chosen and then do some comparisions or calculations on these columns to get a final composite clustering.


How can I add custom numerical features for training to BERT fine tuning?

With BERT I am assuming you are using finally the embeddings for your task.

Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model for the task.

Solution 2: Here you will play with the network.

Now here left one is the normal BERT, in the right we have another MLP network to deal with other numeric features, on top we are combining embeddings from both and then passing through one more MLP to give the final output.
Here the advantage is, while backpropagation, the BERT weight updates will have some contribution because of the loss due to numeric features.

As you mentioned you are new to TF and Bert, refer https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b to see how BERT can be implemented, and with this only doing the tweak to create suggested network will be easy.
Feel free to reach out to me in case of any further questions.


Standardization on training and split data

You should only fit your scaler on training data. Your scaler is part of your model and fitting your scaler to some data can be considered as learning from this data.
Test data is used to evaluate your model on previously unseen data, so if you fit your scaler to test data, it is not "unseen" data anymore.


How to assign a weight for classifiers when using weighted majority voting?

You can test each base classifier on a hold-out dataset and come up with a performance metric say accuracy for each model. You can then use each model's accuracy as a weight when combining predictions in the ensemble.
Here is a useful article.


Normalized 2D tensor values are not in range 0-1

The normalisation you do does not re-scale to $[0,1]$ range! It normalises to have mean $0$ and std $1$ instead.
To scale the tensor to be in $[0,1]$ range you should subtract min value and divide by absolute max-min value.


Correcting Datasets with artificially low starting values

Yes, the general idea is to add a baseline small count to every category. The technical term for this is Laplace smoothing. Really it's not so much of a hack, as encoding the idea that you think there is some (uniform?) prior distribution of the events occurring.


Confused about polynomial regression with multiple variables

Not exactly,
Polynomial regression means that the dataset is not linear and we have to transform it to a specific polynomial degree based on the dataset, so that we may map the Linear model
 Decide a polynomial degree first, let's say 2
$y=b_0+b_1*x_0^2+b_2*x_1^2+...b_n*x_n^2$
If we want to add feature interaction,
$y=b_0+b_1*x_0^2+b_2*x_1^2+b_3*x_0*x_1....$
Basically, you have to implement this Class of Scikit-Learn
PolynomialFeatures


What happened was that

xs[0] = np.zeros((numInputs, 1))

Generates an array of ((4, 1)), and

xs[t + 1] = x[t]

Generates an array of ((1, 4)). To solve this, I transposed x[t] so now it is the same shape as above - ((4, 1)):

xs[t + 1] = np.array([x[t]]).T

Annoying bug and hard to solve, but a fairly easy solution.


Can data science be applied to establishing the author of a text?

Yes - this is an active area of study, especially famously with respect to disputed plays of Shakespeare. Features commonly extracted include frequency of key words, novel vocabularly, word length, and these are well adapted to being used as data mining features.

For example, here is a thesis that data mined Shakespeare's extant plays to test whether they could have been the work of a number of authors often touted as being the "true Shakespeare" -> http://www.cs.dartmouth.edu/~datamining/Final.pdf


Ideas for model to predict wind speed in a local kitesurfing spot

Predicting weather is very complex, so it's unlikely you'll improve upon any predictions from services like WindFinder.
I would first try to prove that predictions from a service like WindFinder are inaccurate, i.e. collect historical predictions and compare it your recordings of sensor data at your local position.
After this, encode your heuristic model which allows you to predict the "good wind" based on other sources into code, but with current information I'm not sure where the ML fits in, rather it seems to just be aggregating sensor & weather site information into a single metric.


ROC AUC score is better if test data is imbalanced

ROC normally reports higher scores for imbalanced data sets because it does not take into account false discoveries. For this reason, it is recommended to only use ROC when the positive and negative sets are roughly equal in size. Otherwise, the suggestion is to compute the area under the Precision-Recall Curve instead.

You can read a bit about the reasons here.

https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/

  However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval , have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.

  — The Relationship Between Precision-Recall and ROC Curves, 2006.


TL;DR: Relative scale of multiple different rewards can be important. However, granting +10 for a win and -1 for a loss in a game will not improve speed of learning how to win any better than tuning the learning rate.

  from a given state if a agent takes a good action i give a positive reward, and if the action is bad, i give a negative reward.

Usually you do not know what a "good action" or "bad action" are, and the reward system is based on the immediate outcome of the action from a certain state.

You may already know that, and I will phrase the rest of this answer as if that is what you meant by "good action". However, that is an important detail, so if you are not sure what the difference is, you could ask another question about it.

  So if i give the agent very high positive rewards when it takes a good action, like 100 times positive value as compared to negative rewards, will it help agent during the training?

The ideal reward scheme is based on easy to measure outcomes that you care about. E.g. reaching a destination, solving a puzzle, how many items are collected, the score in a game, winning a game against an opponent.

If you have both negative and positive rewards, they usually need to be scaled relative each other in a way that makes sense given the problem. It is quite common to have a low negative reward for every time step for instance, if some resource such as time, fuel or money is being used up just from the agent acting and not solving the task. In that case it may make sense to have e.g have a small -0.1 reward for "bad actions" and a larger +10 reward for "good action = completing the task".

Scaling up rewards is like increasing the learning rate. There is usually some optimal learning rate where the agent learns fastest. Too high and the learning is not stable. Too low and the learning is too slow. Given that is the case, usually you just care about getting relative sizes of rewards correct, then you can scale the learning rate to get the best learning speed.

There is no specific benefit to scale up positive rewards only. You should generally do so only when the problem definition allows you to. It might help with speed that the agent learns when you test it, but most of that effect will be the same as scaling the learning rate.

This is different to training animals, teaching small children or rewards for humans in general, where you may be advised to use positive rewards and positive signals more often than negative. However, this is likely to be related to problem domain of general survival and generalist learning in living creatures, which most RL does not replicate. It definitely is not advice that extends to the simpler statistical agents built using RL such as Q-learning.


LSTM loss value not change, accuracy stucked at 50%

The dataset contains ~25K class '0' samples and ~10M class '1' sample. This clearly tells us that the LSTM would be learning patterns more popular in class '1' instances. Here are some improvements you can try:

Instead of undersampling the class '1' labels, oversample the number of instances of class '0'.
Accuracy is not a very good metric in cases of unbalanced datasets. Use 'Macro-F score to evaluate the performance of your model.
Use tensorboard to see how the weights and the gradients of each layer are changing.


You want to fit the model $H = \beta_0 + \beta_1 T + \beta_2 H + \epsilon$

The standard way to enter the data to fit this problem is to enter 3 columns, one for T, one for F, one for H.

T    F     H
20  8179    40
30  8044    40
40  7923    40
20  8343    60
30  8250    60
40  8130    60
20  8528    80
30  8430    80
40  8320    80

In R, the model can be fitted with the command:
lm(H~T+F)

Without checking the appropiatemes of the model, the result is:
Coefficients:
(Intercept)   $\;\;\;\;\;$         T     $\;\;\;\;\;\;\;\;\;\;\;\;\;$      F
  -845.4072 $\;\;\;\;\;$    1.1911   $\;\;\;\;\;$    0.1056
with Rsqr= 0.9958.

When F = 8179 and T = 20.5, the predicted value of H = -845.4072 + 1.1911*20.5 + 0.1056*8179 =  42.39791


(Pattern-Matching) what is the best measurement can be used to match drug names from different countries

One of the workable approaches:
If you have sufficient data, and, you need to so this often, you can train your own custom NER model. Write a pipeline to match drugs and compare price and deliver a small app that solves the problem once and for all!
Custom Named Entity Recognition models helps you identify a named entity from a given chunk of text. There are plenty of models available that provide out of the box support of healthcare related named entities, like, drug!
Once you have extracted named entities like drug, dosage etc. (btw, dosage can also be solved by writing a regex), you can compare the cosine similarities to find the right match. You can try different word embeddings to have more accurate match.


Can I use the test dataset to select a model?

Your teacher is correct. The test dataset is unseen data. You could not select the final model using the test set. In competitions, to be fair, the test dataset is not revealed until you submit the final trained model.

To select the hyperparameters of the final model (e.g., activation functions, the number of hidden layers, the number of units, learning rate, dropout, ...), you should use the validation set.


What does test score provide in terms of real application?

Evaluation is a crucial part of any serious ML project, but when it comes to evaluation choices there is no perfect answer. Generally speaking one evaluates a system in order to know how well it performs (i.e. how reliable are its predictions), usually to know which level of quality to expect when used in production (but not only). However the evaluation results are only useful to some extent:

the evaluation method/measures should be chosen so that it actually represents "quality" for the target task. In general this is imperfect because no evaluation score can fully represent the diversity of a specific task (and that's assuming good evaluation choices)
the test data should be representative of the production data, and in reality it rarely is from the same distribution.
There's almost always a chance factor that one tries to eliminate using cross-validation or other methods, but again that's imperfect.

Besides these unavoidable simplifications, an evaluation score can be interpreted for exactly what it is. For example a precision score of 80% for class X means that one should expect an instance predicted X as truly of class X in 80% of the cases in average. Predicting a specific probability for a particular prediction requires either using a probabilistic model or devising a specific model which predicts a confidence score/probability... But in both cases the model could still can be wrong about the probability it predicts!


Add ID information from one dataframe to every row in another dataframe without a common key

Well, it turned out I accidentally answered my own question in the title.  My answer to this is to insert a new common key into both dataframes, then delete the new key after the 2 are merged.  Maybe there is a more elegant way, but this works.

df.insert(0,'mykey','abcdef')
df2.insert(0,'mykey','abcdef')
df2=pd.merge(df,df2,how='outer',on='mykey')
df2.drop(columns=['mykey'])

Where mykey is the new common key, but make sure it will not match any existing columns, while 'abcdef' is random, but needs to be the same added to df1 and df2.


The features for a token in a NER algorithm are usually binary. i.e The feature exists or it does not. For example, a token (say the word 'hello'), is all lower case. Therefore, that is a feature for that word.

You could name the feature 'IS_ALL_LOWERCASE'.

Now, for POS tags, lets take the word 'make'. It is a verb and hence the feature "IS_VERB" is a feature for that word.

A gazetter can be used to generate features. The presence (or absence) of a word in the gazatter is a valid feature. Example: the word 'John' is present in the gazetter of Person names. so "IS_PERSON_NAME" can be a feature.


When is there enough data for generalization?

It is my understanding that random sampling is a mandatory condition for making any generalization statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, clarifying the @ffriend's comment, I believe that you have to calculate needed sample size, based on desired values of confidence interval, effect size, statistical power and number of predictors (this is based on Cohen's work - see References section at the following link). For multiple regression, you can use the following calculator: http://www.danielsoper.com/statcalc3/calc.aspx?id=1.

More information on how to select, calculate and interpret effect sizes can be found in the following nice and comprehensive paper, which is freely available: http://jpepsy.oxfordjournals.org/content/34/9/917.full.

If you're using R (and even, if you don't), you may find the following Web page on confidence intervals and R interesting and useful: http://osc.centerforopenscience.org/static/CIs_in_r.html.

Finally, the following comprehensive guide to survey sampling can be helpful, even if you're not using survey research designs. In my opinion, it contains a wealth of useful information on sampling methods, sampling size determination (including calculator) and much more: http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm.


Is there an intuitive interpretation of precision always higher than recall?

Generally lower recall means that the system is too strict, i.e. it predicts an instance as positive only when it has clear indications in the features that it's indeed a positive. As a consequence, it misses the true positive instances for which the indications are not so clear.
But when looking at macro-recall it's more complex: it depends primarily on how many classes there are and their distribution. In case of imbalance a very common problem is that the system predicts the majority class too often, causing a low recall for the minority class(es) and therefore a lower macro-recall. The only way to check for this is to look at the performance of individual classes.


In that section, within the for-loop, n is an integer (you loop over a range().

If you use the square brackets to access the Pandas DataFrame (or a Series, in this case), you are trying to access the index of the table. In this case, the index is a timestamp - have a look at the df in section Out [7].

So you need to get values by timestamp, or to use the row number itself, you must do as Vaalizaadeh said in a comment and use the .iloc accessor to the pandas object.

So just change shift = dominantTheta[n] to this: shift = dominantTheta.iloc[n].


Accuracy makes no sense for an autoencoder, because you're essentially doing regression (from X to X), you should use a different metric instead, like MSE, MAE, RMSE.


Efficient environment for machine or deep learning in Python

Try Kaggle, they have kernels with GPUs (free but limited 9 hours runtime I think)

Google cloud is currently beta testing jupyter notebooks on their infrastructure. You do not need to spawn your own compute instance, or even start a Docker: you have your jupyter kernel in your browser in a matter of minutes (https://cloud.google.com/ai-platform-notebooks/). There runtime is illimited but if your connection is glitchy the notebook might disconnect and you can loose your data.

But as a rule of thumb if your runtime is several hours, it might be better to prototype on a subset in notebook and then train on your full dataset in command line.


NLP text autoencoder that generates text in poetic meter

A starting point that comes to mind is creating a cost function for a sentence being in IP. Now, while normally this is a binary affair (either a sentence is in IP or not - or so I would assume), this does not lend itself readily to the task. You should devise a cost  function that measures how close your sentence is to IP (so I assume that sentence with 11 syllables would have a lower cost than one with 12. Verbal stresses and their associated costs are left to you to figure out ;) ).

After you have a cost function, you should take some sort of pretrained deep neural network that can translate from one language to another, such as those that use Word2Vec, set it to translate from English to English, and train it with two costs - one for keeping the meaning of the text (probably with another instance of word2vec or something to that effect), the other for being in IP. The relative weights of the two costs should be determined experimentally (you may even want to change them while training).

This is of course a shot in the dark. More research and experimentation will be required, but I hope this may serve you as a starting point in your endeavor.

Be advised that while this is an intriguing project, in my opinion it might not be the best choice for beginners.


The problem is that you want to do supervised learning without having labels. You have some target in mind, but it might not be how the underlying data is split. Without any information on what is anomalous or not, your clustering algorithm won't try to work out that information. It will give you groups of users that have different behaviors (people that prefer weight vs. elliptical), but not necessarily the splitting that interest you.
First you need to think about and define what would be an anomaly for that problem : an unbalanced value ? an extreme value ? an impossible one ? An anomalous session vs. an anomalous user ? It's not clear from your question and generally speaking, it might help you work trough your problem, eventually by labelling your data.
To solve your ML problem, after working on your dataset (see below), you might consider different approaches :

If you want to continue on the unsupervised path : augment the N number, it might provide you more clusters of which some of them might correspond to what is anomalous to you. Then you might consider techniques like hdbscan that find outliers outside of the mains clusters.

Label the data then consider some fully supervised technique (a small random forest might be enough). You might need to define exactly what is anomalous to you (someone that come very often ? someone that badges multiple times thus giving 0 minutes sessions ? someone that stay the whole day ?).

Label the data for those you consider some sort of anomaly and use a semi supervised algorithm, like self organised map, that will help you retrieve those anomaly you missed in your labelling process.

Generally speaking it seems you also need to work on your dataset beforehand:

It seems you only have too few predictors. As is, it is unlikely your algorithm find meaningfull anomalies, that you couldn't find / work out without ML. There is two path : try do to some explanatory data analysis (frequency of training, distribution of time spent), that might help you understand what are your anomalies and design a non-ML solution. Or try to get more variables, that will help you detect anomalies and give them meaning.

You consider each entrance separately, based on the user name. First it may cause some problems if two user have the same identification. You might want to consider a unique identifier for user identification. Then, if you want to identify anomalies about user, not sessions, you might want to group entrance by user. That would require you to build other features, like frequency, average time spent... etc.


Why does exploration in DQN not lead to instability?

All Reinforcement Learning (RL) control algorithms have to deal with the exploration/exploitation trade-off. Do you trust that you have learned enough about the environment that you can act optimally, or do you still need to check alternative actions.

Related to this, there are two basic approaches when trying to learn an optimal policy in RL. An agent can either learn:

on-policy where the agent assesses the actions that it is taking, and adjusts behaviour directly. To solve the exploration/exploration trade-off with on-policy learning, a common approach is to decay $\epsilon$ (or a similar parameter which controls the amount of exploration), based on some idea of how much learning is necessary to learn the task.
off-policy where the agent has separate behaviour and target policies. The agent takes actions from an exploring behaviour policy, and adjusts the results it gets mathematically so that it calculates a different target policy, which is often optimal in some way.

Q learning is an off-policy algorithm, which learns the action value function of following deterministic greedy action choices at all times. In the simplest single-step version, the adjustment required is really simple - instead of estimating the action value based on what the behaviour policy does, Q learning updates using whatever the best next action would be, regardless of what gets chosen.

Q learning is proven to be stable and to converge when using a table of q values (i.e. not a neural network). However it is actually not very stable when used with neural networks for estimating $\hat{q}(s,a, \theta)$ (where $\theta$ is the NN weights). This is why DQN needs to use experience replay and a separate target network - in order to combat the instability. The instability is not directly caused by random actions though, it is more complex than that.

  If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?

The agent will not behave consistently on each run, as it is still exploring. However, if it has been set up correctly, it will still converge to and learn an optimal policy, and in fact the constant exploration should help it get values for non-optimal action choices correct, and maintain the correct maximising actions.

When you use a Q learning policy after it has finished learning, typically you would set epsilon to zero and have the agent act greedily.

It is worth noting that even with a deterministic policy, RL is designed around the concept of an environment where random events can happen. For instance it will learn the best way to move when navigating, even if sometimes the distance travelled is not the same due to random events. When you test an agent in a non-deterministic environment, then different things will happen each time, and this is not a problem for the learning process. It might be a problem for you to evaluate the agent - you would need to run a test multiple times and take an average to assess how well it is performing.


Metric to combine average with number of accurance

The metric you are looking for is the standard error. From Wikipedia:

  the standard error equals the standard deviation divided by the square root of the sample size.

So as the sample size increases the standard error decreases. You can also see this for an explanation with some code in R.


How to deal with changing video frame sizes in a CNN?

In your example, you are seeing the effect of a stabilisation algorithm applied to otherwise jerky movement - and the effect is a moving "window" inside the video frame.

The actual video file has a fixed frame size. As far as I know there are no video formats that switch size frame to frame. You may have different areas of focus to worry about, but should never need to worry about frame size switching on you mid video. You can rely on your video processing library to present a consistent frame size for the whole of a video.

If you find that the border of the view inside the video frame interferes with the performance of your CNN once trained, then you could deal with it in a few ways. A couple of simple approaches might be:

1) Crop out a centre portion that always reliably has content in, and use just that portion for your input features.

2) Detect and flood-fill the black border with the mean pixel value, using an image/video library (like OpenCV) so that edges around it cause less activity in your CNN.


When do I have to use aucPR instead of auROC? (and vice versa)

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-AUC) lies in its tractability for unbalanced classes.  They are very similar and have been shown to contain essentially the same information, however PR curves are slightly more finicky, but a well drawn curve gives a more complete picture.  The issue with PR-AUC is that its difficult to interpolate between points in the PR curve and thus numerical integration to achieve an area under the curve becomes more difficult.

Check out this discussion of the differences and similarities.

Quoting Davis' 2006 abstract:

  Receiver Operator Characteristic (ROC)
  curves are commonly used to present results
  for binary decision problems in machine
  learning. However, when dealing
  with highly skewed datasets, Precision-Recall
  (PR) curves give a more informative picture
  of an algorithm’s performance. We show that
  a deep connection exists between ROC space
  and PR space, such that a curve dominates
  in ROC space if and only if it dominates
  in PR space. A corollary is the notion of
  an achievable PR curve, which has properties
  much like the convex hull in ROC space;
  we show an efficient algorithm for computing
  this curve. Finally, we also note differences
  in the two types of curves are significant for
  algorithm design. For example, in PR space
  it is incorrect to linearly interpolate between
  points. Furthermore, algorithms that optimize
  the area under the ROC curve are not
  guaranteed to optimize the area under the
  PR curve.

This was also discussed on Kaggle recently.

There is also some useful discussion on Cross Validated.


What's the most effective way to measure the accuracy of my multi-class classification NN?

Keras has some accuracy metrics for multiclass neural networks.

CategoricalAccuracy
Top K  Categorical Accuracy
Sparse Top K Categorical Accuracy

There are some others if you want to implemented from scratch. Taken from: https://www.sciencedirect.com/science/article/abs/pii/S0306457309000259

These work for most problems rather nicely combined with a good cross validation approach and leak prevention.
Having an imbalanced dataset always makes it challenging but picking a metric that doesn't punish results too much like Top K Categorical Accuracy combined with n-fold cross validation like this article suggests, is often enough to have a solid evaluation procedure that you can work on.


Can a Warm-up loss period easily be implemented in Keras?

To my knowledge it is currently not possible to compile a Keras model first with a warmup loss function and later re-compile it with another. That would be the cleanest solution in my opinion.

You can however, with a little use of tensorflow (assuming you are using that as a backend judging from your tags) do something along these lines:

WARMUP_ROUNDS = 10

def loss(y_true, y_pred):
   seen = tf.Variable(0.)
   seen = tf.assign_add(seen, 1.)

   loss = tf.cond(tf.less(seen, WARMUP_ROUNDS),
                  lambda: warmup_loss(y_true, y_pred),
                  lambda: actual_loss(y_true, y_pred))

   return loss


Does batch_size in Keras have any effects in results' quality?

After one and a half years, I come back to my answer because my previous answer was wrong.

Batch size impacts learning significantly. What happens when you put a batch through your network is that you average the gradients. The concept is that if your batch size is big enough, this will provide a stable enough estimate of what the gradient of the full dataset would be. By taking samples from your dataset, you estimate the gradient while reducing computational cost significantly. The lower you go, the less accurate your esttimate will be, however in some cases these noisy gradients can actually help escape local minima. When it is too low, your network weights can just jump around if your data is noisy and it might be unable to learn or it converges very slowly, thus negatively impacting total computation time.

Another advantage of batching is for GPU computation, GPUs are very good at parallelizing the calculations that happen in neural networks if part of the computation is the same (for example, repeated matrix multiplication over the same weight matrix of your network). This means that a batch size of 16 will take less than twice the amount of a batch size of 8.

In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.


What's the Seminal Book on data science?

As already pointed out by the first comment, data science is a very broad field and not sharply defined.
But I think since you like to get started with data science and you're looking for resources, I share some of my recommendations with you (in no particular order).

Very applied, less academical, with lots of code examples: Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron
The Deep learning bible, more academical, great overview: Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville
A classic for Machine learning, covers a lot of topics, very educational with great explanations: Pattern Recognition and Machine Learning by Christopher M. Bishop
If you're more into statistics: An Introduction to Statistical Learning: with Applications in R by James, Witten, Hastie & Tibshirani
Machine Learning: A Probabilistic Perspective by Kevin P. Murphy
The classic RL reference: Reinforcement Learning by Richard S. Sutton & Andrew G. Barto

This is only a partial list focusing more on theory and classic data science algorithms. Keep in mind there are many many more books, e.g. I didn't mentioned any resources for Big Data, Data Analytics, Data Mining etc. whatever one would throw into the data science bucket. Further it's possible that this list might be outdated in a few years.


How to verify a CNN encoder works as expected?

In order to verify the encoder efficiency than it is better to design some down stream task like classification, similarity , etc. Based on that u can validate your encoder. U may refer this link for downstream Task.


Work experience in Data Science: Couple hours with A-level student

Since you're busy at the moment, perhaps you could take a section of a dataset that you've worked with previously (and have the relevant processing stored somewhere) and let them loose on it. Then in a week or two go back and see where they've got with it.. it has the two prong approach of throwing them in at the deep end and seeing what they can do as well as providing a ready-made learning opportunity for you.


I have worked on similar projects (using medical images such as PET to predict outcomes).  A method being used more and more for predicting cancer treatment outcomes is texture analysis: https://www.ncbi.nlm.nih.gov/pubmed/21321270

Another method of texture analysis uses wavelet transforms: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3505569/

After deriving the texture features, you could then use those in a ML model for making predictions.

A benefit to texture analysis is that it is mostly independent of the size of the target.  It is however, more dependent on resolution. If your images are produced on different scanners, you may need to resample in order to normalize the images.

There are fewer studies about deep learning for medical imaging, but it definitely has some exciting potential.  The question is whether or not a CNN can pick up subtle differences.  From what I've seen (I'll look for a reference), CNN hasn't outperformed any other method yet.

Maybe add a few more details so we can get a better idea of what your objective is?


Has anyone heard of a model similar to a random forest which fits a linear regression model in its leaf nodes?

M5P Model Trees are the closest thing that I'm aware of.  In these trees, the leaf nodes are linear models.  The difference is that the leaf nodes learn a multi-dimensional linear model instead of single-dimensional.
(If you had a single-dimensional model at the leaf, how would you know which feature to use in the regression?  What is the advantage over multi-dimensional?)
Of course a model tree is just a single tree, not a forest.  But you could easily build a random forest composed of model trees.


A = np.array(['A','B','C','A','B','A','B','C','A','A','B','C','D','E'], dtype=object).reshape(-1,1)
B = np.array([1, np.nan,np.nan,2,np.nan,3,np.nan,np.nan,4,5,np.nan,np.nan,np.nan,np.nan], dtype=object).reshape(-1,1)
arr = np.hstack((A,B))

df = pd.DataFrame(arr, columns=['A','B'])
df.fillna(method='ffill',inplace=True)

cols = np.append(['num'],df['A'].unique())
data = pd.DataFrame(columns=cols)

data['num'] = pd.Series(df['B'].unique())

for i, row in data.iterrows():
    for j, column in row.iteritems():
        check = df[(df.B==i+1) & (df.A==j)]
        if len(check) != 0:
            data.loc[i,j] = j

data.fillna('',inplace=True)
data

Some engineering might be needed on boundary conditions if pattern changes with new data.


Confusion Matrix and AUC in univariate Anomaly Detection

It's difficult to answer precisely without knowing the data and the task.

Assuming it's a single column of values with no order involved, it boils down to finding the optimal threshold to separate regular values vs. anomalies. Given that you know which ones are the anomalies (labelled data), the problem can be treated as a binary classification task: a value is either regular (negative) or an anomaly (positive).

The idea is simply to evaluate the performance for every possible threshold, i.e. count the number of True/False Positive/Negative for every value. Let's say you take the set $S$ of all the distinct values in your column. For every value $t \in S$ you calculate the confusion matrix (i.e. how many TP/FP/TN/FN) obtained by predicting every value $x$ as positive if if $x>t$, negative otherwise.

From there you can plot a ROC curve and/or find the optimal threshold according to some evaluation measure (typically the one which maximizes the F1-score).

I assume that there are libraries which do that automatically, but it's simple enough so personally I would do my own code for this kind of thing.


Which scaling size is better? [0,1] or [-1,1] for LSTM?

I think both scalings can potentially behave well. The important matter is to put all the features in the same range, otherwise some features will add up more in the first linear transformation of your layer (remember that any neural network is a linear transformation and a non-linear one), which is undesirable, and will hinder the learning.

Best option is then to try both, and take the one that gives best performance.


input and output to fully connected layer

A Fully Connected network looks like this:

You input layer can be of any size, and your output layer can be of any size as well.

The hidden layer in the middle has a size that you need to define.

Each layer is a Fully Connected layer because it has a connection to all nodes in the previous layer. So output nodes have connections to all nodes in the hidden layer. And, all nodes in the hidden layer have access to all nodes in the input layer.


How to encode an array of categories to feed into sklearn

If the sequence of watches is meaningful, then you do need some kind of classifier that creates user/item embeddings from the sequence of watches, so you're probably looking at GRUs and LSTMs. Neural nets aren't overkill here per se. This is a pretty good treatment of the topic: https://towardsdatascience.com/introduction-to-recommender-system-part-2-adoption-of-neural-network-831972c4cbf7

However this simple approach above does mean you have a multi-label class with a lot of dimensions. This can be hard to scale and may not train well without much data. Feature hashing doesn't help so much in the output because you need to map each back to the right video

If you're willing to forego the sequence, which often doesn't carry that much information beyond simple recency (recent watches are more important). Yes the usual way of dealing with this is as a huge matrix factorization problem. Sparse SVD is possible but maybe overkill; ALS is simpler and about as effective, and works well for implicit input like you have here. You can readily add weights to watches with ALS, which can be used to deemphasize older watches.


Accumulators in Spark (PySpark) without global variables?

The problem with sharing nothing is that for reduce functions like sum you need results from multiple elements. There are Spark ways to do this and there is also a sum already implemented, but the easiest code for reduction is using variables and looping. You need to get used to functional programming that Spark uses in order to write Spark way code. Functional programming also allows easier parallelization and faster distributed code.

It would probably be faster to use Spark dataframes like here. And in my opinion they are easier to understand than map-reduce logic.

A possible map-reduce approach (this is how the whole code looks like):

rdd1 = sc.parallelize([{0: 0.3, 1: 0.8, 2: 0.4},
                       {0: 0.2, 1: 0.4, 2: 0.2},
                       {0: -0.1, 1: 0.4, 2: 1.6}])

def acc_fn(acc, row):
    res = {}
    for key,value in row.items():
        res[key] = acc[key] + value
    return res

vector_acc = rdd1.reduce(acc_fn)

print(vector_acc)

vector_acc is now defined as a Python dictionary and not a class. You should access it as a dictionary.


What is the current state-of-the-art within aspect-based sentiment analysis?

Basically, there are two types of sentiment analysis.

Symbolic approach - the sentiment of the sentence is classified based on the kind of words used in the sentence. Involves a lexical database such as wordnet and some rules based on the grammatical structure of the language as knowledge representation.
Vector models - rather than detecting the sentiment in a sentence, the query sentence is classified into one of the category based on the training dataset and the features are represented as word vectors. Distributed and Distributional representations are types of it.

Whether it is coarse-classification (positive - negative) or fine-classification (types of emotions in it), a training dataset is needed if you are going to approach it via vector models as it is much more easier to play with.
The skip-gram model and continuous bag of words model developed in Google based on Recurrent neural network language model made a breakthrough in Natural language processing seems to be the state-of-art model in Distributional representation.
Since the code has been open-sourced, the c-code is available here and the python library is maintained here.
It doesn't matter if you are going to use reddit comments, tweets or reviews. What matters is how well the sentences are classified for the training data. Because garbage in is garbage out.


Set the errors parameter of the to_numeric function to 'coerce'. The value 'ignore' will leave any invalid value as is while 'coerce' will set it to NaN.


Whether solution of Kernel ridge regression through inverse calculation yields global optimal solution or not?

In the case that the inverse exists, it is the same as the pseudo inverse. Unless the method crashes due to the fact that the inverse does not exist, the results will be the same.


k-means and LDA for text classification: how to test accuracy?

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering.
Secondly, you asked for testing accuracy. As your problem is pure Clustering, there is no evaluation for that.
The last but not least is the problem of "Short Text Understanding". In short texts, LDA of TF-IDF based approaches (like LSA) do not work well as they rely on co-occurrences of words in texts.
Considering these two facts let's discuss the solution. I would recommend that you use a pre-trained model (I recommend S-BERT which is implemented in Sentence-Transformer Python package). Simply follow this semantic similarity search piece of code and you can implement it easily in a few lines.
Probably fine-tuning the model will be tricky according to the nature of your task, so just use a pre-trained model and see how it works.
For evaluation, I recommend to capture some similar tweets manually (more the better) and check the performance on them.
For Topic Modeling approach you can use the pytho implementation of the paper I mentioned above here.
Two above mentioned can also be combined creatively (search similarity with S-BERT and compare to bi-term topic model for example)
Hope it helped.
Good Luck!


Yes. The centroid will converge to the center of all your data and this will occur in a single iteration. This is due to all the data points belonging to a single centroid, thus it will be centered according to all these instances immediately.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

# Generate data
class1 = np.random.randn(1000, 2)

class2 = np.random.randn(1000, 2)
class2[:,0] = 4+class2[:,0]
class2[:,1] = 4+class2[:,1]

class3 = np.random.randn(1000, 2)
class3[:,0] = -4+class3[:,0]
class3[:,1] = 4+class3[:,1]

data = np.append( class1, class2, axis= 0)
data = np.append( data, class3, axis= 0)
print(data.shape)

# Plot the data
plt.scatter(data[:,0], data[:,1])
plt.show()

# Cluster
kmeans = KMeans(n_clusters=1, random_state=0, verbose = 1).fit(data)

# Plot clustered results
plt.scatter(data[:,0], data[:,1], c=kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:,0], kmeans.cluster_centers_[:,1], c = 'r')
plt.show()

# Show the cluster centers
print(kmeans.cluster_centers_)

  Initialization complete Iteration  0, inertia 81470.055 Iteration  1,
  inertia 48841.695 Converged at iteration 1: center shift 0.000000e+00
  within tolerance 8.140283e-04


Forecast vs Prediction: What is the difference?

Prediction is more general term.
It can be defined as a an expectation for a combination of different predictors (random variables).

Forecasting would be a subset of prediction and mostly related to time-series. Any time you predict into the future it is a forecast. All forecasts are predictions, but not all predictions are forecasts.

Moreover, forecasting always have error, that is why we have confidence interval of 5%, 2.5%, 1% and residuals for modeling. There is no such thing as so called "perfect forecasting", but we can maximize the model accuracy.

On contrast, prediction sometime can be perfect. I also believe that model should follow data not the other way around.


They match up fairly well.

The first Goodfellow description is Karpathy's final "many to many" image. The output at each time step is based on the previous hidden state of the net and the input.

The third Goodfellow description directly corresponds to Karpathy's "many to one" image. This model reads an entire input sequence, and then produces one output.

The only difference is that the second description from Goodfellow's text isn't captured by Karpathy's image. Here's my rendition of what that description states.


Classifying multilabel images with TensorFlow

I suggest you try building multiple networks: one network for building type (outputs "house", "apartment", or "condominum"), another network for build year, another for garage (yes vs no), and so on.  This keeps the number of classes small for each network, and allows each network to tune itself for the specific task it is focusing on.

If you want to avoid multiple networks:

Conceptually, share the first n-1 layers, and have a separate nth layer for each of these tasks.  In other words, the last layer is made wider, with elements for building type, elements for building year, and so on.  You end with multiple softmax elements.  A simple example would be one softmax for building type outputs ("house", "apartment", or "condominum") and one softmax for build year ("new" or "old").  Thus, you'd have 5 output wires.

You can of course experiment with other splits, say, where the first n-3 layers are shared and then for the last 3 layers you have multiple different networks in parallel.


Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half?

You basically answered your own question:

  Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? Is this procedure prone to overfitting if you perform this operation on all the train data?

Yes. You have already used the target values and the first half of the training set. In other words, the filled-out NaN values have information about the target values in them, so if you were to perform any cross-validation for performance estimation, you would end up with an over-estimation of your performance metric. It's best practice to leave that used part out of any further training steps to avoid information leakage.


Can we fine-tune a model on the same dataset which it is pretrained on?

In the context of that paper, pre-train then fine-tune on the same dataset does not really make sense, as the pre-training is unsupervised, and the fine-tuning is with labelled data.
But, generally, if you have trained on dataset X for N epochs, and then you fine-tune one more epoch using the whole of X, it is just another way of saying you trained for N+1 epochs. Nothing wrong with it, except as you note in the question, if you were starting to overfit after N epochs, you are even more over-fitted now.
What does make sense, and something we have used in production models, is to do the initial training on a large dataset that is the combination of X1, X2, X3, ... and so on. Then once the learning curve starts to level out, we take a copy of the model and then fine-tune on e.g. just X1.
This is in the context of NLP Transformer models, so may not make sense for other domains, but we try to do initial training on as much data, from all domains, as possible, then we might fine-tune on just medical papers, or just economic reports, depending on what the model will be used for. (We've also done a final fine-tune on just a subset of the data to have it learn a specific style. It is amazing that even with just one epoch of this final fine-tune it can switch to using a different way of writing numbers, for instance.)


Trouble with grouping by and counting in r

The dplyr package should be useful for this.

library(dplyr)

World %>%
  group_by(name) %>%
  mutate(num_distinct_languages = n_distinct(language)) %>%
  filter(num_distinct_lanagues > 4)


How can I check the correlation between features and target variable?

Your data can be put into a pandas DataFrame using

import pandas as pd
data = {'Loan ID': ['LP001002', 'LP001003', 'LP001005', 'LP001006', 'LP001008'],
        'Married': ['No', 'Yes', 'Yes', 'Yes', 'No'],
        'Dependents': [0, 1, 0, 0, 0],
        'Education': ['Graduate', 'Graduate', 'Graduate', 'Not Graduate', 'Graduate'],
        'Self_Employed': ['No', 'No', 'Yes', 'No', 'No'],
        'Income': [5849, 4583, 3000, 2583, 6000],
        'Coapplicant Income': [0, 1508, 0, 2358, 0],
        'LoanAmount': [123, 128, 66, 120, 141],
        'Area': ['Urban', 'Rural', 'Urban', 'Urban', 'Urban'],
        'Loan Status': ['Y', 'N', 'Y', 'Y', 'Y']}
df = pd.DataFrame(data)

Now to get a correlation we need to convert our categorical features to numerical ones. Of course the choice of order will affect the correlation but luckily all of our categories seem to be binary. If this is not the case you will need to devise a custom ordering.

df = pd.DataFrame(data)
df['Married'] =df['Married'].astype('category').cat.codes
df['Education'] =df['Education'].astype('category').cat.codes
df['Self_Employed'] =df['Self_Employed'].astype('category').cat.codes
df['Area'] =df['Area'].astype('category').cat.codes
df['Loan Status'] =df['Loan Status'].astype('category').cat.codes

Now we can get the correlation between the 'LoanAmount' and all the other features.

df[df.columns[1:]].corr()['LoanAmount'][:]

Now using some machine learning on this data is not likely to work. There just is not sufficient data to extract some relevant information between your large number of features and the loan amount.

You need at at least 10 times more instances than features in order to expect to get some good results.

To only obtain the correlation between a feature and a subset of the features you can do

df[['Income', 'Education', 'LoanAmount']].corr()['LoanAmount'][:]

This will take a subset of the DataFrame and then apply the same corr() function as above. Make sure that the subset of columns selected includes the column with which you want to calculate the correlation, in this example that's 'LoanAmount'.


Explaination or Description of clusters after clustering

It depends on the clustering technique you use. Since you tagged this post with k-means I will assume this is what you are using. Cluster centers should already be somewhat informative for laymans, but since you should be/are scaling this can lose some of it's interpretation.

What you could do is assign class labels to each sample based on in what cluster they ended up in. Then you could fit a multi-class decision tree to your data and use the decision rules for interpretation, like 60% of cluster 1 has $x_1 < 0.9$.


Machine Learning Based Algorithm for Image Inpainting?

If I were you I would use deep learning. You can use an autoencoder format for this. Essentially you would feed the image in, then each layer yields a smaller output. Then you would feed the output in reverse, using transposed weight matrices, which would yield progressively larger outputs. The final output from the backward pass would be the filled in image. Since your image has such a small color range I would recommend representing pixel colors using one-hot encoding.


How to deal with a machine learning model which affects future ground truth data?

One option is reinforcement learning (RL).
RL frames the problem as sequential decision-making under uncertainty. There is an agent making decisions and after the decision is made, the agent collects new data. The next round of decisions uses the new data.
What you are describing is a scenario where the distribution of "ground-truth" changes over time. Many statistical and machine learning models assume a stationary process where the underlying distribution does not change over time. Some types of RL does not assume a stationary process.


Possible harm in standardizing one-hot encoded features

With unpenalized linear models, there is no difference.  The coefficients will just scale to counteract the new scale of the variables, and the intercept will shift to compensate for the centering.
With penalized linear models though, there will be a difference.  Since the standard deviation of a binary variable is at most $1/2$, you'll be increasing the overall scale of the variable by standardizing.  That will cause the unpenalized coefficient to decrease in magnitude, which will change the balance of how the penalty applies to different features.  I suspect there is no "better" approach then: sometimes the penalty improves performance when the dummies are scaled, and sometimes degrades performance.


Input Normalization for Transfer Learning

You should select a single preprocessing scheme and keep it constant for all experiments!

In your case, you want to scale your input features to $0$ mean and a standard deviation equal to $1$:

$$x_{scaled} = \frac{x-μ}{σ}$$

During the initial training compute the mean $μ$ and standard deviation $σ$ of each feature and store them. Then use the same values to scale the data for all subsequent experiments.


Can you reduce many Naive Bayes training samples to a few big ones?

Your assumption seems to be correct that both approaches are actually equivalent. I can not see any difference here in the propabilities and conditional propabilities for either one way or the other.

That's why the predictions should also be the same for both ways. It is more likely that there is something wrong with your implementation.


Linear Discriminant Analysis (LDA) before or after k-fold cross-validation?

It depends on what you want to achieve.

If you want to visualize the results of your SVM classification, then you should do it after.

If you want to reduce noise, speed up training ... or whatever reason you want to reduce the dimensionality of your problem. You should do it before.

One idea here that can be useful, is to do LDA inside a pipeline and choosing the best hyperparameter within the CV.

In this example you can check a PCA and then a logistic regression. Your case will be similar but with LDA and SVM.


The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is

For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.
When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.
How then is the covariance matrix calculated in such case when each node have only partial data?

I shall address that, which hopefully will clear the matter to a degree.
Let us look at a common form of covariance calculation, $\frac{1}{n}\sum(x-\bar{x})(y-\bar{y})$
This requires you to calculate the following:

$\bar{x}$
$\bar{y}$
$x-\bar{x}$ and $y-\bar{y}$
Multiply the $(x-\bar{x})$ and $(y-\bar{y})$

in a distributed manner. The rest is simple, let us say I have 100 datat points (x,y), which is distributed to 10 Apache Spark workers, each getting 10 data points.
Calculating the $\bar{x}$ and $\bar{y}$: Each worker will add $x/y$ values of 10 data points and divide this by 10 to arrive at partial mean of $x/y$ (this is the map function). Then the Spark master will run the aggregation step (in Spark DAG of the job) where the partial means from all 10 workers are taken and again added, then divided by 10 to arrive at the final $\bar{x}$ or $\bar{y}$ (the aggregate/reduce operation)
Calculating the $(x-\bar{x}) \cdot (y-\bar{y})$: Same way, distribute the data points, broadcast the $\bar{x}$ and $\bar{y}$ values to all the workers and the calculate the partial $(x-\bar{x}) \cdot (y-\bar{y})$, again run aggregation to get $\sum (x-\bar{x})(y-\bar{y})$
The above method is used for distributed calculation, you shall get the covariance, for multi-dimensional data, you can get the covariance matrix.
The point is to distribute the calculation for stages that can be distributed and then centralize the calculation stages that cannot be distributed. That is in effect one of the important aspect of Spark architecture.
Hope this helps.


How to prepare Audio-text data for speech recognition

In general audio training tools requires data segmented on small chunks. It is not a big problem to segment, you can use segmentation scripts, for example
segment_long_utterances.sh as described in Kaldi group discussion. Many other segmentation tools beside that like dsalign or gentle or aeneas.
Overall, training model from scratch is pretty complicated process which requires time and data (>5000 hours of speech properly annotated). It can take months to build a new model.
You'd better investigate first the reasons your error rate is bad, it might not be solvable with training. If information is already lost, you can not really repair it. A better microphone, no compression for data and many other tricks can improve recognition accuracy significantly. You can also just adapt the language model to improve accuracy in case standard language model is not good for your content. It is easier than acoustic model training and you just need texts as described here.


What are Hybrid Classifiers used in Sentiment Analysis?

In sentiment analysis you may want to combine a number of classifiers. Let's say: a separate classifier for emoticons, another one for emotionally loaded terms, another one for some special linguistic patterns and - let's say - yet another one to detect and filter out spam messages. It's all up to you.

You can use either SVM, Naive Bayes, or anything else that best suits your problem. You may use majority voting, weights (for example based on cross validation results), or any other more advanced technique to decide which class is the most appropriate one.

Also, googling for hybrid sentiment returns tons of papers containing answers to the questions that you have stated. Please, don't ask us to rewrite this papers here.


How to find feature importance with multiple XGBoost models

I suspect you may be confusing the terms "test set" and "training set". Normally, the model is trained on the training set, and evaluated on the test set. The feature importance is independent of the test set, it's a property of the model that you trained.

The obvious answer is to combine all months into a single training dataset, and train a model on that, but you say it’s not an option.

Then I think your only option is to combine the results from individual months, for example by summing or averaging them and presenting the final ranking. Whether this is valid depends on the criteria used to train the trees (how the splits are determined). If there is a valid way to combine them, I still recommend against it, because I suspect it will obscure reality and discussion over the methods may distract from the result.

So I would sidestep the problem by presenting a graph like is sometimes shown for sports leagues or polls before elections. On the horizontal axis are the months, on the vertical axis the rankings. This way you visualize all info easy to grasp and you also show any trends and seasonal effects.


Will stochastic gradient descent converge for multivariate linear regression

For an MSE Loss, you will have a convex surface

Gradient Descent on a convex function is guaranteed to converge if the learning rate is not too high. The only that can happen is a high enough learning rate which diverges the learning

Impact of one record per learning step - Light on computing but a zigzag gradient

Working on a single instance at a time makes the algorithm much faster because it has very little data to manipulate at every iteration.
On the other hand, due to its stochastic (i.e., random) nature, this algorithm is much less regular than Batch Gradient Descent: instead of gently decreasing until it reaches the minimum, the cost function will bounce up and down, decreasing only on average.
$\hspace{2cm}$[Ref] - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow,by Aurelien Geron

Impact of multi-variate gradient -
Multi-variate means, now you have multiple participants to decide the resultant Gradient. Everything will be the same and the learning will follow the net Gradient path.
One thing that will change is that with a high dimension, we have a long and flat surface in most of the places in the Loss space which can slow down the learning if the learning rate is not scheduled. Convergence is still guaranteed if LR is not too high.
For Intuition -
For more than 3-D, you will have to trust on the extrapolation and vizualize in your own way.
$\hspace{3cm}$
$\hspace{3cm}$Individual record Gradient and Overall gradient - 1 Feature
$\hspace{3cm}$
$\hspace{6cm}$A Loss surface in 3-D - 2 Features
$\hspace{3cm}$[3-D plot credit] - https://www.monroecc.edu/faculty/paulseeburger/calcnsf/CalcPlot3D/


As far as I know the nltk lemmatizer works on words or rather ngrams. Your example is a trigram, an easier way to work through this is:

word="web based technologies"
splits=word.split()
word=" ".join(lemmatizer.lemmatize(w) for w in splits)


I once heard a data scinetist state at a conference talk: "Basically, you can do what you want, as long as you know what you are doing."

This also applies here. The more statistically sound way would be to transform all variables prior to additional steps such as PCA or factor analysis. Then you still know the scale of your variables and can interpret the rescaling in the context of your application. If you have no such interpretation, but good reasons for rescaling your principal components due to computational issues arising if some values are to close to zero while others are quite large, rescaling the components makes sense. However, reversing this process and still being able to interpret the effect of the rescaling operation in your context will become almost impossible.


How to get a similarity vector from two vectors?

If I understand your question correctly, you have 2 sentences and you converted those sentences into 2 vectors, you want to know how those sentences are similar.
If this is the case use cosine similarity between 2 sentences and cosine similarity is a scalar not a vector its the dot product of your 2 embedding vectors.

from numpy import dot
from numpy.linalg import norm

cos_sim = dot(a, b)/(norm(a)*norm(b))

where a and b are your vectors.

There is direct formula as well:

from sklearn.metrics.pairwise import cosine_similarity

cosine_similarity(vector1,vector2)


How to compare the performance of different number of mixing components for EM algorithm?

1.

The simplest and most common way is to use AIC or BIC. You would pick a model with the minimum AIC/BIC value. AIC/BIC work well here because you have a likelihood function.

Bayesian model selection is another possibility. It's more advances than AIC or BIC, but you get chance to add your own prior distribution. Section 5.3 in

  Machine Learning - A Probabilistic perspective (Kevin P.Murphy)

has the details. There're also lots of papers on Google.

Alternatively, you can use the cross-validated likelihood as a performance measure, although this can be slow, since it requires fitting each model N times, where N is the number of CV holds.

Slide 17 in

  https://www.doc.ic.ac.uk/~dfg/ProbabilisticInference/IDAPISlides13.pdf

is a good reference.

2.

EM algorithm is a technique for estimating the most likely parameters and is not limited to mixed Gaussian. The algorithm is useful when there's no closed form formula for the solving maximum likelihood problem with one or more latent variables.


Get bounding boxes for adjacent instances of a single class in image

My first thought would be not to full deep learning on this - It is hard to see but it looks like your regions are bound by vertical lines with many horizontal ones spanning those regions. You can try doing just simple canny filters to detect those lines (maybe with [opencv] - [https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_houghlines/py_houghlines.html]), then find the points where horizontal and vertical lines intersect to form vertical bounds for regions.

Another idea that may help is the sweep-plane algorithm,:[https://scicomp.stackexchange.com/questions/8895/vertical-and-horizontal-segments-intersection-line-sweep]
I am just spitballing here, but where notes and horizontals meet will form connected regions. Finding connected regions that contain horizantal lines gets you part of the way. Then slicing those with the output of the vertical line detector (maybe it is a search over similar length groups starting with longest vertical using the tree-based strategy of sweep-plane) is worth a try.

On the lines of the RPN, I have had good experience with SSD for a similar problem (detecting individual drawings on an architectural drawing). SSD differs in that it returns something like 8K proposals with confidences, and then a second pass of tuning the confidence threshold and finding non-overlapping regions got me pretty close, but my intuition says that your dataset is structured enough to have another answer.

I am curious how many pages you have in the dataset. If you have less than a few thousand annotated lets say, it may be harder to train a big neural net, and would lean toward the canny filter/hough transform direction. Also are the pages that are annotated represent a diverse enough sample of the production data?


How to Include Features that Apply to Specific Classes

If I understood correctly, you said that  Pipe Size does not have a correlation with the Predicted Hours. If you are sure that one or more variables do not possess the information about the target then drop them.
Other than that, if a feature (or some of your features) has a relation only with a specific task, then replace the null values with 0.
But in your case, I think you should also try the simpler algorithms like Linear Regression. But be careful about the sample size vs a number of features balance. Since you have few samples, use at most 1 variable for every 15 samples (In rare cases, 10 can be used too, but it is not recommended). As this source mentions:

In summarizing the findings by Schmidt, Green suggested that the minimum number of SPV ranges from 15 to 25.

Also, you need to evaluate your model's performance, but since you may have an unbalanced number of samples for each Task Type, be careful while splitting your data to train and test, make sure your test data has enough samples from each Task Type.
Some other important things:

Try to do bivariate (between the target variable and an independent variable) analysis before modeling, so you can eliminate unnecessary variables.
In the case of using Linear Regression, be careful about multicollinearity if you will use the coefficients to interpret your results. Because if your model has multicollinearity, then your coefficients will be incorrect although your predictions will note be spurious i.e. they will still be trustable.


What does 'Linear regularities among words' mean?

By linear regularities among words, he meant that "Vectorized form of words should follow linear additive properties!"

V("King") - V("Man") + V("Woman") ~ V("Queen)


Extracting the feature map in Tensorflow Object Detection API

import tensorflow as tf

# Sample frozen model
MODEL = "frozen_inference_graph.pb"

# An existing operation from the frozen model
OP_NAME = "WeightSharedConvolutionalBoxPredictor/BoxPredictionTower/conv2d_2/BatchNorm/feature_0/beta/read/_360__cf__363"

# Load the graph from the frozen model
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(MODEL, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

# print all operations
for op in detection_graph.get_operations():
    print(op.name)

# print tensor ( without :0 you will get the operation itself )
print(detection_graph.get_tensor_by_name("{}:0".format(OP_NAME)))


If you've got prior information then you should certainly not use simple mean in a split test. I assume you're trying to just predict which group will produce the greatest amount of revenue overall, by trying to emulate the underlying distribution.

Firstly, it's worth noting that any metrics you choose will actually reduce to mean in a pretty trivial way. Eventually mean will necessarily work out, though using a standard bayesian method to estimate the mean is probably your best bet.

If you've got a prior then using a standard bayesian approach to update the prior on your mean revenue is probably the best way to do it. Basically, just take the individual results you get and update a multinomial distribution representing your prior in each case.

If you want some more full background on multinomial distributions as bayesian priors are pretty well, this Microsoft paper does a pretty good job of outlining it. In general, I wouldn't care so much about the fact that your distribution is technically discrete, as a multinomial distribution will effectively interpolate across your solution space, giving you a continuous distribution that is a very good approximation of your discrete space.


Using EM (Expectation Maximization) algorithm for Training Logistic Regression

Use Algorithm 1 from this paper. The rest of paper is more about the analysis of algorithm and theoretical background which is very nice but not necessary for your purpose. Implementation of the algorithm in Python or R is pretty straightforward.

Hope it helps :)


Equation of a Multi-Layer Perceptron Network

You are forgetting one element of the MLP which is the activation function. If your activation function is linear - then you can simply flatten out all the neurons into one single linear equation. The advantage of MLP however is its non-linearities so I suspect in your network you do have some activation (sigmoid? tanh? relu? etc..).
As for your graph - you could simply output predictions from your MLP and plot the exact scatter plot you have above. The only difference would be you wouldn't have a simple way of expressing this network in algebraic notation (as you have done on the existing x-axis).
To describe networks effectively in text you should look into matrix notation describing the weights and inputs of each layer. Maybe take a look at something like this to get started: https://www.jeremyjordan.me/intro-to-neural-networks/


Neural Network regression negative performance

It is quite possible that the original model does already a very good job - it is not possible to improve it.
For example the true relationship between original input variables and target might be linear, so a neural network doesn't add anything.
As a test, I would increase hidden_layer_sizes and set early_stopping=False and even just doing a grid search (without CV): RMSE Training should get better than RMSE Feature. Most likely RMSE Testing will be worse, but at least you will have evidence there are no other unexpected circumstances (eg. a bug in the code).
It does make sense to use predictions of a base model as inputs. Especially if that original base model is constrained somehow, and you expect your new model to outperform the original one because in the new model you don't have that constraint.
For example the original model might be linear (ARIMA or ARIMAX), while yours is nonlinear - a neural network.
Even if the true relationship between inputs and target is nonlinear, you don't have too much playroom here for a neural network (remember we want to outperform a base model). As I understand you have only ~ 4*365 observations. You can easily overtrain with many neurons, but with only a few neurons training can stuck in local minima.


Is this a good approach to classify tickets which contains description and logs?

First, you should get clear about your non-functional requirements (speed, memory consumption). Make sure you have a clear idea who is going to maintain the software and what the effort will be if you make some changes. Higher accuracy might not be worth it.

The simplest way to combine n models is averaging their predictions. This technique is often used on Kaggle. See chapter 2.6 of https://arxiv.org/pdf/1707.09725

Did you try a simple MLP with tf-idf features? How well does that perform? Usually it executes way faster than SVMs, if you have many classes.


There is one answer to know: try both methods and take the one that gives the best result. I would say that in general pre-trained embeddings usually gives better results. You can also start with pre-treained embeddings as initial conditions and let the embeddings train maybe with a smaller learning rate.

In any case, the current state of the art for text classification is ULMFIT (https://arxiv.org/abs/1801.06146), which actually doesn't do any of this. It pre-trains embedding and RNN with a language model in the wikipedia and in the target text and then fine tunes the whole model with the target text.


sklearn "balanced_accuracy_score" sample_weights not working

The point of sample_weights is to give weights to specific sample (e.g. by their importance or certainty); not to specific classes.
Apparently, the "balanced accuracy" is (from the user guide):

the macro-average of recall scores per class

So, since the score is averaged across classes - only the weights within class matters, not between classes... and your weights are the same within class, and change only across classes.
Explicitly (from the user guide again):

$$\hat{w}_i = \frac{w_i}{\sum_j{1(y_j = y_i) w_j}}$$

i.e. the i-th sample is re-weighted by dividing its weight by the total weights of samples with the same label.
Now, if you want, you can just use the simple accuracy score, and plug in weights as you see fit.
In the following example:
from sklearn.metrics import balanced_accuracy_score, accuracy_score

y_true = [0, 1, 0, 0, 1, 0, 1, 1, 1, 1]
y_pred = [0, 1, 0, 0, 0, 1, 1, 1, 1, 1]

some_sample_weights =[10, 1, 1, 1, 10, 1, 0.5, 0.5, 0.5, 0.5]
weights_by_class =[1 if y==1 else 1000 for y in y_true]

print('with some weights: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred, sample_weight=some_sample_weights)))
print('without weights: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred)))
print('with class weights in balanced accuracy score: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred, sample_weight=weights_by_class)))
print('with class weights in accuracy score: {:.5f}'.format(accuracy_score(y_true, y_pred, sample_weight=weights_by_class)))

class_sizes = [sum((1 for y in y_true if y==x))/len(y_true) for x in (0,1)]
weights_by_class_manually_balanced = [w/class_sizes[y] for w, y in zip(weights_by_class, y_true)]

print('with class weights in accuracy score (manually balanced): {:.5f}'.format(accuracy_score(y_true, y_pred, sample_weight=weights_by_class_manually_balanced)))

you get:
with some weights: 0.58
without weights: 0.79
with class weights in balanced accuracy score: 0.79
with class weights in accuracy score: 0.75012
with class weights in accuracy score (manually balanced): 0.75008

As you can see:

using class weights in the balanced accuracy score didn't matter; they just got adjusted back into class sizes.
using class weights in the accuracy score is very close to 75% (3 of out of 4 the 0 labels are correctly classified), and re-adjusting the weights according to class sizes doesn't matter much (the accuracy is a bit less, because the 0 class is larger)


What features most contribute for label y?

I have good news and bad news.

The bad news is that due to the nonlinear nature of neural networks, the same feature will contribute differently depending on the other features.

Here's a simple nonlinear function: f(a,b,c) = y = ba^3 + bc + c.  One could argue that 'a' contributes most to this function, but when b is negative, a more positive a might make things worse.  When b is zero, c contributes most.  When b is negative, a very negative a will contribute most.

The good news is that you may be able to visualize some simple single-layer networks.  There's a fun trick for this used in deep learning.

Assuming you've got x for your examples, W for your weights, and y for your outcomes, set the feature you want to show in y to 1 and the rest to zero, then multiply by W-transpose.  You should get some output values for x which show you how connected everything is.  It's not a perfect solution, but if you get some huge values coming out after the multiplication, you can say those values are highly-connected for y.  (Usually.)


Identifying Customers who are more likely to purchase a given product category- Which model to use?

Actually, I'd propose to tackle it from a different angle - assuming you want to predict product categories, and those are usually few, you can see it as a regular classification problem.

Every product group is a category, which you will try to predict. So you can use simple models, which are easy to train (logistic regression, decision trees etc.), easy to build, debug, explain and test. And you can experiment with various features - which ones are the most helpful when predicting product category (this is crucial, since many people will probably ask you why the model predict this category for this person).

Once you master this, then the regular approach to this is any type of recommender system (collaborative filtering, matrix factorization etc.). Plus you can use your simple approach as a benchmark.


Feature reduction by removing certain columns in dataframe

It depends, generally speaking it might be okay as those columns only seems to holds very few information. However there are some case, notably when the outcome is unbalanced, where some skewed features are usefull. Say you are trying to predict something that happen 1% of the time, a feature with only 2% of non-zero value but causally leading to the outcome, might be very predictive for your target.
So when removing those it is important to check (before) there is no clear dependency to the outcome and (after) the performance doesn’t drop too much when removing them.


Python - accessing dictionary values for math operations

The reason is that ticker = (stocks[name]) is outside the if-statement. Since the loop does not stop when name == stock it continues to loop until the end of stocks and name will always be assigned to the last key in the dict.
One of multiple solutions is to move it within the if-statement:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

for name in stocks.keys():
    if name == stock:
        ticker = (stocks[name])
        print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)

print(amount)

This gives:
How many shares? 10
What's the stock? FB
The price for FB is 255
2550

And you get the same output even without a loop:
shares = input('How many shares? ')
stock = input('What\'s the stock? ')

ticker = (stocks[stock])
print('The price for', stock, 'is', ticker)

amount = int(ticker) * int(shares)

print(amount)

This gives:
How many shares? 10
What's the stock? FB
The price for FB is 255
2550