The estimation quality of the mean improves with sqrt(n).

So with more and more data, your mean will get more precise. But the improvements get slower and slower.

It won't help with all the other issues of k-means, such as problems with different cluster diameters and outliers. And more data probably means more outliers...

You can try a similar approach to that paper with k-means with some slight modifications. Try plotting the sum of squared deviations between the holdout set, and the nearest centroid each. As long as this improves, the clusters fit your holdout set better.


Logistic Regression optimal threshold is a negative value

I'm not familiar with the way you are obtaining the optimal threshold, but It might be a little bit easier.
What you are looking for is the leftmost point in the x-axis (false positive rate) and the rightmost point in the y-axis (true positive rate) So by calculating the difference between the two you will have so.
from sklearn.metrics import roc_curve
yhat = best_model.predict_proba(X_train)[:,1]

fpr, tpr, thresholds = roc_curve(y_train, yhat)
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]

This threshold will give you the lowest false positive rate and the highest true positive rate
EDIT
I just notice that you are passing the $P(Y = 0| X)$ i.e yhat = yhat[:, 0] try passing yhat = yhat[:, 1]


How to split data into training, validation, test data sets if the data is non-stationary?

In general, for time series data, you can split it in such a way that the temporal order is as is in all your splits (training, validation and testing sets).
Once you make your series stationary (via differentiation for instance), you can make use of the TimeSeriesSplit cross-validator by scikit-learn, with wich you get the time-ordered indices of each train-validation split, so that you can use them later on with the validation strategy you might want to use, something like:

As a possible validation strategy, you could use the walk-forward validation (nice info here)


Keeping DataFrame in Memory while working - R vs Python in VS Code

This is more about the IDE than the programming language. When you use RStudio, you can import the data and the IDE saves it in its memory so you don't have to reload it when you write subsequent code in that same session.

To do this in Python, consider using a Jupyter Notebook or something similar. That way, you only need to import your dataframe once, and you can then perform exploratory analysis, visualizations, and build models as needed.


Is there a limit in the number of layers for neural network?

The maximum number of layers that a network can have depends on many factors, such as the model architecture, activation function, optimization method, and others. For example, as you alluded to, ResNets is huge in increasing the number of layers possible, even up to 1000 layers for CIFAR-10. Another example is that when ReLU functions became more popular, Kaiming initialization allowed deeper networks to be trained by keeping the variance of each layer's activations at around 1, as the other popular method of Xavier initialization is more suitable for sigmoid activation functions.

So in general, there's no known limit except to keep experimenting and coming up with new techniques, including but not limited to finding better activation functions.


You should convert the categorical variables to dummies. For each individual variable in general you want to have equal number of elements of each class, or at least the numbers should be close. If not, you can cluster smaller classes to form a larger one. For example, let's assume you have a categorical variable with 5 different categories. You want each class to be approximately %20 of the data. If it is not, you can define a new class which combines smaller classes to make each class approximately equal.

For the second part, if you can actually quantify how much A1 is better than A2, or able to assign a relative value to them based on some heuristics; you can convert them to numerical variables.

You can find an example of this in this notebook (section titled "Aggregating categorical variables"). It is from the course "Principles of Machine Learning: R Edition" on edX. You can watch the videos on audit mode for free; and the notebooks are on github.


Are there databases specializing in scientific data

SciDB is an open-source DBMS for scientific data.


What's the difference between GridSearchCrossValidation score and score on testset?

Gridsearch cross-validation can be used to learn the hyper-parameters of a prediction function. Consider that learning and testing of the model on the same data is a big mistake. The chance of having a perfect score but failing to predict anything useful on yet-unseen data (i.e., overfitting) is very high when using the same data for learning and testing. It is common practice when training a model to hold out a part of the data as a test set to prevent overfitting and measure the performance of the model. Also, note that the best hyper-parameters can be determined by grid search techniques and the score resulted from grid search should not be used as a criterion to measure the performance of the model. Please refer to this page for more information
That being said, best_score_ from GridSearchCV is the mean cross-validated score of the best_estimator. For example, in the case of using 5-fold cross-validation, GridSearchCV divides the data into 5 folds and trains the model 5 times. Each time, it puts one fold aside and trains the model based on the remaining 4 folds. Then, it measures the performance of the model based on the left-out fold.  Finally, it returns back the mean of the performance of 5 models as the final score.
Now, let's answer this question: what does the best estimator mean? it is the estimator that was chosen by the search or the estimator which gave the highest score (or smallest loss if specified) on the left-out data. GridSearchCV's goal is to find the optimal hyperparameters.  It receives a range of parameters as input and it finds the best ones based on the mean score explained above. Grid search trains different models based on different combinations of the input parameters and finally returns the best model or the best estimator. Hence,  best_score_ is the mean score of the best estimator.
It is notable that tuning hyperparameters with cross-Validation in the above context is one of the methods that helps you to prevent overfitting.
In your case, 0.8923046854943018 is the mean score of the best estimator. Let's call this score cross-validation score.  For your case, I would go with the second case, because in that case there is no overfitting and the cross-validation and test scores are almost the same. In the first case, the cross-validation is significantly higher than the unseen test score and there is overfitting. It means that the model works very well on the train data but not on the unseen data.


Are there better techniques for doing video classification compared to using temporal segment networks?

I would suggest to train the network to detect bird species in image space without the temporal dimension. This will make the network have much less parameters to tune and you will likely get better results. Then for a novel video, you can detect the species for each frame and do a majority vote to decide what species was in the video.

Are there any preprocessing steps that should be done to improve accuracy?

If you train in image space then you can use a DataGenerator on the image and transform it in all kinds of ways, the position and rotation of the bird does not affect the species. This can be done through the Keras pipeline. You can check how to do this in general here.

How to train the model to generalize better to be less sensitive to lighting and weather?

You can try to pass your images through some pretrained segmentation network to identify background and foreground and then set that area to a static color. Perhaps black (i.e. all 0's). This will make the decision boundary for the network easier. However, I think you should focus on the other points, the network should inherently detect what features are the most salient for discriminating different species, if you have sufficient training samples the weather and background will be low information features and will thus not be used.

How can the model be trained to classify unknown species?

This would be called anomaly detection. This is a much harder problem. If you have birds of other species available in your dataset then you can add a class label for other. This is still standard supervised learning. If this is not available it will be very difficult. Most anomaly detection algorithms do not work well for high dimensional datasets. Then you will need a way to compress the information in your input images. This can be achieved with some transfer learning on a pretrained autoencoder network trained on natural images such as ImageNET, then you can use transfer learning using your bird images to further tune the network for your specific use case. You can then use this compressed feature space using one of the anomaly detection techniques I described here.


When to stop showing content on recommendation engines?

Interesting question, but I would say this really depends on the recommendation system itself and the purpose of your goal. Let's think it through:

Indeed we could collect the number of logins and create a relative feature of times ignored until chosen. This allows us also to inspect the amount of time a user ignores a show. On the first look, this makes perfect sense to me. However, this will only work if the user decides each time for a new show, from my own experiance I can say this is not valid for all user :D!

Therefore we would need to include also the probability or the fact of the user choosing a new show for given login. To keep things simple let's say we collect only those logins in which the user is deciding for a new tv-show or movie. Then our new feature would make way more sense and is a bit closer to what we can expect from reality.

But there is more. As far as I know (please correct if I'm wrong), Netflix has a sort of prediction or classification on what is the best picture to display for a certain user on each tv-show. In case the pictures change over time, our login feature from above is less useful. Maybe the image made the trick to convince the user of a tv-show or movie - honestly I don't have a clue how to measure this straight away, except maybe with another feature or collected value if the image was different than the last time.

So far so good. Let's say we collect the following:

selected/not selected for each movie or tv-show recommended in a
session the user picked a new tv show or movie
picture changed for each tv show or movie recommended after the last
time the user picked a newly recommended item.
And maybe to improve our picture algorithm also what kind of picture
could have had an impact on the user's decision

Sounds like a plan. But the more I think about it the more comes to my mind, that often my wife is watching on the same Netflix account and our tastes are pretty different for tv-shows or movies we watch in absence of the other. Now, one could say: "yes buddy, this is why netflix offers you profiles". But honestly, I'm not into watching tv that much, so I usually don't care about the profile I'm using. Hence I won't be the only one who shares his profile with a person with another preference, this could have serious influence on our recommendation as well. The profile will be mixed and the precision of our recommendation system will probably decrease.

I'm not sure if this is valid for the mass of Netflix users. We don't have any numbers, so I would assume this is valid for more or less than 30% (just guessing). In this case, you could try to track active user behavior. Like on what kind of recommendations does the user stops for how long, on which tv-shows is he interested in how many seasons it has, how often did the user hover over a movie or show?

The more we think about it the more edge cases and conditions come to my mind. Since I personally don't have a lot of experience with such advanced recommendation systems I would assume that from a technical point of view it would make sense to split the system into two parts. One which gives us a baseline of a larger amount of predictions and one which adjusts the recommendation live, while the user is behaving on the open application to adapt current data and to decide which profile is relevant and applicable for the real person behind the user.

This is really interesting. I'm curious what others (probably with more experience) thing about these questions and about what I provided as an idea.


I don't know much but maybe my answer would helps you. For machine learning, we have to do feature engineering to extract the important features from the data. Here we have images and the methods above to extract features of each object and image, then classify them using one of machine learning algorithms.
In CNNs, you do all these together and that's why the the CNNs object detection approach is better than ML approaches.
If you look at any of famous object detection models, you will see a CNN model such as ResNet or VGG which are used for feature extraction as a first part of the model.
Also, you can use the model for feature extractions and then apply ML algorithms to classify such as KNN, SVM, etc.


You should use scaling instead of normalizing. By normalizing the column you make the norm 1, while you are interested in making the expected value 0 with a standard deviation of 1. With the norm, the more samples you have the smaller your samples will become. Use the scikit-learn StandardScaler instead. Despite this, with these huge numbers you will still get weird results (although most of them will be negative with this skew). What could be an alternative is using a log transformation on your dollars first, and then scaling them. Conceptually you usually look at relative values between dollars, the difference between 10 and 20 dollars is similar to that of 10,000 and 20,000 which the log transformation will capture.


Finding number of common elements between different columns of a DataFrame

You can build something like below. You can leverage set().intersection() to find the intersection between list. You need to loop one column with other columns.

You would Notice that I changed the values of df columns into a list in order to use a set

import pandas as pd

df = pd.DataFrame({'A': ['cat','lion','bull','horse'], 'B': ['ox','dog','ox','eagle'],
                   'C': ['bull','cat','horse','bull'],'D': ['horse','zebra','tiger','giraffe']})

df_out = pd.DataFrame()

for col in df.columns:
    # making a list of all column which has to be compared with col
    other_col = [x for x in df.columns if x!=col]
    for oCol in other_col:
        #using a set we can find a intersection between 2 list and count them
        count = len(set(df[col].values.tolist()).intersection(df[oCol].values.tolist()))
        #storing all count with their respective column in separate df
        df_out = df_out.append([(col,oCol,count)],sort=True,ignore_index=True)

df_out.columns = ['Column','Comparison','Count']

df_out.sort_values(by=['Column','Count'],ascending=[True,False],inplace=True)

Output would look something like this:

Column Comparison  Count
1   A   C   3
2   A   D   1
0   A   B   0
3   B   A   0
4   B   C   0
5   B   D   0
6   C   A   3
8   C   D   1
7   C   B   0
9   D   A   1
11  D   C   1
10  D   B   0


First try a simple model: The input layer and the output layers dimension are defined by your data / your problem definition. Then train a model without any hidden layer.
See how good it performs. Is it good enough? If yes, you're done. If no, continue
Add a hidden layer of reasonable size or adjust a hidden layers size. Go to step (2).

The "reasonable" size part might be difficult. As a guidance: If you have only a single node, it is certainly too small for a 1000 class problem. It might be big enough for a 2 - 3 class problem. I would usually suggest to keep the size of the features per layer roughly constant or at most reduce it by 1/10 or triple it. But that is only gut feeling.

The reason for my preference for simple models is Occam's razor, the fact that they are often faster, easier to analyze and to manually improve.

For more information about topology learning and rules how to design neural networks, see:

  Thoma, Martin. "Analysis and Optimization of Convolutional Neural Network Architectures." arXiv preprint arXiv:1707.09725 (2017).

Especially chapter 2.5 and chapter 3.


You could consider each month as an explanatory variable. And the year as response variable. Because, for each year, you'll have 12 monthly variables.

With a model, you could find the relationship that has each month value with the final output. For that, you can perform a linear model (your categorical variables will or must be recoded as dummy variables for that estimation).

If you want to check the effect of the variable by itself. Yearly Target = a * Monthly Variable. I agree with @Leevo.


why does ID3 Decision tree algorithm not give the best decision tree?

If you want the best decision tree, you need the "optimal decision tree", which you can get nowadays. Just have a look into the vast literature.


I think it would be more helpful for us if you put all yours code and language which are you using. I suppose that's are you using Python. In this case try Python library scikit-fuzzy. Some examples at the link:
https://pythonhosted.org/scikit-fuzzy/auto_examples/index.html


It often is appropriate to use quantile bins.

I.e. to get four bins, choose the smallest 25%, then up to the median, then the Q75, ...


Batch Normalization vs Other Normalization Techniques

Cosine normalisation is result of the fact that we bound dot
product and hence decrease the variance, when we use cosine similarity or centered cosine similarity instead of
dot product in neural networks (which is quasi ground-stone in NN)

Main benefit of cosine normalisation is Cosine
normalization bounds the pre-activation of neuron within
a narrower range, thus makes lower variance of neurons.

Also it does not depend on any statistics on batch or mini-batch
examples, and performs the same computation in forward
propagation at training and inference times. In convolutional networks, it normalizes the neurons from the receptive fields rather than the same layer or batch size.

Have a look at this paper showing emipirically comparison between normalisations you mentioned. C.N. comes on top.


Why does an L2 penalty prefer smaller and more diffuse weight vectors?

You answered this in your question. "Prefer" means "produces a smaller penalty", and you've identified that the penalty in the first case is smaller. Why would this be a good thing? It amounts to preferring an explanation based a bit on many features, rather than one based entirely on one features. That's often a good bet to avoid overfitting.

These two weight vectors do not produce the same dot product with other vectors in general. If the vector X contained all identical values, they would.

If the 4 input features were regularly identical or nearly so, then it means they're redundant, and you may prefer to use just 1 of the features (your second case), instead of a bit of each. In this case, an L1 penalty would at least be indifferent to the two, not penalize the second one.


A few questions to understand a random forest blog

I had a look at your questions and a brief look at the blog you posted. I will not go into detail and answer all your questions since it will take time and there should be one question. Nonetheless, I would like to give you some heads up on this and help you where possible.

When doing a confusion matrix is usually on a test set, you can see also where is used confusionMatrix(data = rparty.test, reference = testing$Sex). What is the usual is to have a training set, cross-validated to pick hyper-parameters and so forth and then test it on the test set. There is no rule just following a logical procedure. If these are new I would recommend diving into what are cross validations vs training set, test set and validation set.
Probably is talking about SMOTE upsampling (aka oversampling). This theory is for creating new artificial variables by creating synthetic ones (by combining features of the unprecedented class). Again I would recommend having a look there are many ways of dealing with an unbalanced dataset. Let's name a few: undersampling, oversampling, ensemble sampling (combing the previous) or you can tune certainly your model in order not to be fair towards a class and more.
Will not go into much detail about the ROC curve but I would recommend checking the AUC graphical plot instead of in this example and try to understand. Hint, the training dataset was balanced hence is reasonable a model would expect a balanced dataset for a test set instead is unbalanced.

All in all, I had a quick overview didn't go into depth of your blog and I didn't the find it very comprehensive plus uses confusing terms (as I remember), for instance, oversampling instead of upsampling or precision instead of specificity and recall for sensitivity. I wrote the answer as I empathise you as new in the area but next time please post only one question and make these sort of questions at the blog if applicable instead of here where the author directly can answer. Hope I helped you little bit, but you should also understand you opened a lot of topics.


sklearn's cross_validate does not work with catboost

I found out that adding xtrain.astype('O') works.

Apparently catboost doesn't work with pandas Categorical dtypes yet: https://github.com/catboost/catboost/issues/814


Good "frequent sequence mining" packages in Python?

I am actively maintaining an efficient implementation of both PrefixSpan and BIDE in Python 3, supporting mining both frequent and top-k (closed) sequential patterns.


Updating Google News Word2vec Word Embedding?

Transfer-learning is one possible approach :

Design and implement a neural net to match Google Word2Vec's design
(In terms of number of layers, activation functions and etc.,).
Pre-initialize weights with these vectors
Retrain with domain-specific corpus

This is an implementation that can be used as base and modified for step #1


Understanding Bernoulli Trials, Bayesian Setting

What is a Bayesian setting and why is Bayesian thing everywhere?

In very simple terms:

  Bayesian is a statistical setting, where the likelihood of an event
  happening (called the posterior) depends on the prior trials or
  observations (called the prior(s)).

Bayesian networks is an extension of the above, forming a chain or a network of inferencing.

  Are there any theorems/theories that I must know?

For understanding the Bayesian paradigm, you need to know the Bayesian theorem/relation, which is basically:

$$P(\theta|d) = \dfrac{P(d|\theta)P(\theta)}{P(d)}.$$

  Any book/notes/resources where I could learn about these stuff in a
  relatable way (I have very little maths background, but I can learn)?

I would highly recommend "Doing Bayesian Analysis" by John Krushke


I think your input features are lacking. Firstly, The budget allocation is not simply done based on the previous year turnover and size of the company, there are myriad other factors involved which directly correlate towards a budget allocated to any department.
Secondly where is the relation of profit vs budget allocation? For maximizing profit according to budget allocation, your training set needs to reflect that.
Thirdly you could simply make target profit an input feature and model your data accordingly to allocate budget keeping in mind the targeted profit as well. Provided you incorporate features highlighted in the first point.


Word2Vec: Identifying many-to-one relationships between words

Those vector relations are not exact. Rest assured that king - queen ≠ man - woman. What we do is finding the closest vectors to the result of king - man + woman. One of the closest vectors is queen.
Nevertheless, when we try the "parallelogram approach" to verify word relations, in most cases, the closest vector is the original one. The fact that we disregard the original word as a sensible result is due to the original implementation of word2vec, as shown in the study Fair is Better than Sensational:
Man is to Doctor as Woman is to Doctor. This implementation quirk is actually one of the sources of gender bias attributed to word embeddings.
There are other articles studying the limitations of the analogies based on word embeddings, like this and this.
That being said, there are also studies of the relationships that can be found in word representation spaces, like this one, which studies different types of relationships, e.g.  class-inclussion, part-whole, attributes, etc. You may find the main figure of the article interesting:


Scaling features in artificial neural networks

As per your assumption, if you're scaling the features so that all of them are transformed between 0 and 1, the case where you're inputting 7 will result in a rescaled feature which will be greater than 1 (7/6 > 1). So, this tells us that your scaling technique of dividing inputs by a specific number isn't a really good scaling approach.

For your specific problem, scaling won't really be necessary, as explained here.
The important chunk says:

If the input variables are combined linearly, as in an MLP, then it is rarely
strictly necessary to standardize the inputs, at least in theory. The reason is
that any rescaling of an input vector can be effectively undone by changing the
corresponding weights and biases, leaving you with the exact same outputs as you had
before. However, there are a variety of practical reasons why standardizing the inputs
can make training faster and reduce the chances of getting stuck in local optima.
Also, weight decay and Bayesian estimation can be done more conveniently with
standardized inputs.

But if you're still interested in scaling the data, here's a nice discussion which can be explored to get an idea about standardized scaling/normalization techniques.


What is the difference between "equivariant to translation" and "invariant to translation"

Equivariance and invariance  are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by @Xi'an, you can find previous uses (anterior to Convolutional Neural Networks) in the statistical literature, for instance on the notions of the invariant estimator and especially the Pitman estimator.

However, I would like to mention that it would be better if both terms keep separate meaning, as the prefix "in-" in invariant is privative (meaning "no variance" at all), while "equi-" in equivariant refers to "varying in a similar or equivalent proportion". In other words, one in- does not vary, the other equi- does.

Let us start from simple image features, and suppose that image $I$ has a unique maximum $m$ at spatial pixel location $(x_m,y_m)$, which is here the main classification feature. In other words: an image and all its translations are "the same".
An interesting property of classifiers is their ability to classify in the same manner some distorted versions $I'$ of $I$, for instance translations by all vectors  $(u,v)$.

The maximum value  $m'$ of $I'$ is invariant: $m'=m$: the value is the same. While its location will be at $(x'_m,y'_m)=(x_m-u,y_m-v)$, and is equivariant, meaning that is varies "equally" with the distortion.

The precise formulations given (in mathematical terms) for equivariance depend on the class of objects and transformations one considers: translation, rotation, scale, shear, shift, etc. So I prefer here to focus on the notion that is most often used in practice (I accept the blame from a theoretical stand-point).

Here, translations by vectors  $(u,v)$ of the image (or some more generic actions) can be equipped with a structure of composition, like that  of a group $G$ (here the group of translations). One specific $g$ denotes a specific element of the translation group (translational symmetry). A function or feature $f$ is invariant under the group of actions $G$ if for all images in a class, and for any $g$,
$$f(g(I)) = f(I)\,.$$

In other words: if you change the image by action $g$, the values for feature or function $f$ are the same.

It becomes equivariant if there exists another mathematical structure or action  (often a group again) $G'$ that reflects
 the
transformations  (from $G$) in $I$  in a meaningful way. In other words, such that for each $g$, you have some (unique?) $g' \in G'$ such that

$$f(g(I)) = g'(f(I))\,.$$

In the above example on the group of translations, $g$ and $g'$ are the same (and hence $G'=G$): an integer translation of the image reflects as the exact same translation of the maximum location. This is sometimes refered to as "same-equivariance".

Another common definition is:

$$f(g(I)) = g(f(I))\,.$$

I however used potentially different $G$ and $G'$ because sometimes $f(.)$ and $g(.)$ do not lie in the same domain. This happens for instance in multivariate statistics (see e.g. Equivariance and invariance properties of multivariate quantile and related functions, and the role of standardisation).
But here, the uniqueness of the mapping between $g$ and $g'$ allows to get back to the original transformation $g$.

Often, people use the term invariance because the equivariance concept is unknown, or everybody else uses invariance, and equivariance would seem more pedantic.

For the record, other related notions (esp. in maths and physics) are termed covariance, contravariance, differential invariance.

In addition, translation-invariance, as least approximate, or in envelope, has been a quest for several signal and image processing tools. Notably, multi-rate (filter-banks) and multi-scale (wavelets or pyramids) transformations have  been design in the past 25 years, for instance under the hood of shift-invariant, cycle-spinning, stationary, complex, dual-tree wavelet transforms (for a review on 2D wavelets, A panorama on multiscale geometric representations). The wavelets can absorb a few discrete scale variations. All theses (approximate) invariances  often come with the price of redundancy in the number of transformed coefficients.
But they are more likely to yield shift-invariant, or shift-equivariant features.


predicted bounding boxes that stretch beyond grid cell (Andrew NG CNN course)?

1) Exactly. You have two anchor boxes in Andrew's current example, so the algorithm is going to output two predicted bounding boxes for each grid cell.

2) Your statement below is not true:

  "Because we know that each object can belong to one grid only based on the midpoint"

I don't remember that being said on the course. The center of the object belongs to a single cell.

TO CLARIFY: The fact that an object spans at a region greater than the grid cell it is assigned to has nothing to do with the grid cell's size itself. The object can and will be larger than its assigned grid cell. However, the output assigns each object to one grid cell because that grid cell contains its midpoint.

Anyway, the receptive field of the neurons is much larger than the single cell they process (i.e. they cover the whole image). Anchors are initialized on a certain width and height but will be resized during inference, based on the identified object size using the final feature map. So one might consider that yolo predicts the occurrence of an object as well as its size.

More on the receptive field of CNNs

Check the: "A regressor rather than a classifier" part

  A taste: For every positive position, the network predicts a regression on the bounding box precise position and dimension

This has been also asked here


Is there any package in python that can identify similarity between alphanumeric alias names of a parameter?

If you know there are only specific variants, you can obviously make a look-up table yourself (i.e. a Python dictionary).
Otherwise you could try using a fuzzy matching library, like fuzzywuzzy.
This will give you a "closeness" score for your search term, based on your list of parameters (measurements). Here is an example of how you could use it:
In [1]: from fuzzywuzzy import process

In [2]: measurements = ["Voltage", "Current", "Resistance", "Power"]

In [3]: variants = ["VOLT", "voltage_in", "resistnce", "pwr", "amps"] # notice typos etc.

In [4]: for variant in variants:
   ...:     results = process.extract(variant, measurements, limit=2)
   ...:     print(f"{variant:<11} -> {results}")  # See which two were found to be closest
   ...:     best = results[0]                     # Take the best match by score (first in the list)
   ...:     if best[1] < 70:                      # Set a threshold at 70%
   ...:         print(f"Rejected best match for '{variant}': {best}")

VOLT        -> [('Voltage', 90), ('Current', 22)]
voltage_in  -> [('Voltage', 82), ('Resistance', 30)]
resistnce   -> [('Resistance', 95), ('Current', 38)]
pwr         -> [('Power', 75), ('Current', 30)]
amps        -> [('Voltage', 26), ('Resistance', 22)]
Rejected best match for 'amps': ('Voltage', 26)

So most worked out pretty well, including the typo example.
Obviously this does not kind of semantic search, as so amps do not get related to Current in any way.

To go the way of semantic encodings, you might want to look into "word embeddings", which do indeed try to match the real meaning of words, based on their semantic meaning. To start here, you could look into Word2Vec or GloVe` embeddings. Perhaps there is even a tool or library that already offers this capability.
These approaches will not inherently deal with things like typos, so for best results, you could even combine the two approaches.


How can you decide the window size on a pooling layer?

When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and see what works better. In the case of pooling layers it is actually relatively interpretable. Why do we use pooling? To downsample our feature maps. This is done to decrease the resolution of our processed image but also to increase the receptive fields of our following layers. Because it is a form of downsampling we throw away all the information that was not the maximum in case of max-pooling, or we throw away information about the spread with mean-pooling. By increase our pooling filter size, we decrease our resolution even further and lose more information, but we will also need less parameters in our network if we have a fully connected layer at the end and our next convolutional layers will have a bigger receptive field, but with less attention to detail.


How to calculate growth function for a threshold function

Yes, your interpretation is correct.

Each member of $H$ is one such function, they are parametrized by $i$, the feature selected and $\theta$, the chosen threshold. Different $\theta$ might corresponds to the same function but effectively there are only $m+1$ such function for each $i$. Hence, the set $H$ consists of at most $(m+1)N$ elements.


How does keras train without disrupting the data set order

By default, Keras will shuffle training data before each epoch (shuffle=True).

If you would like to retain the ordering of your dataset, then set shuffle=False (docs here).


How to use one hot encoding of string categorical features in keras?

For string data, use get_dummies() (from Pandas). to_categorical() takes integers as inputs.

There are two important differences between
Keras: to_categorical() and
Pandas: get_dummies().

Keras: to_categorical()

to_categorical() takes integers as input (no strings allowed).
to_categorical() generates dummies starting at 0 by default!

Looking at the help function:

print(help(to_categorical))

Says:

to_categorical(y, num_classes=None, dtype='float32')
    Converts a class vector (integers) to binary class matrix.

    E.g. for use with categorical_crossentropy.

    # Arguments
        y: class vector to be converted into a matrix
            (integers from 0 to num_classes).
        num_classes: total number of classes.
        dtype: The data type expected by the input, as a string
            (`float32`, `float64`, `int32`...)
...

So if your data is numeric (int), you can use to_categorical(). You can check if your data is an np.array by looking at .dtype and/or type().

import numpy as np
npa = np.array([2,2,3,3,4,4])
print(npa.dtype, type(npa))
print(npa)

Result:

int32 <class 'numpy.ndarray'>
[2 2 3 3 4 4]

Now you can use to_categorical():

from keras.utils import to_categorical
cat1 = to_categorical(npa)
print(cat1.dtype, type(cat1))
print(cat1)

Which yields a matrix:

float32 <class 'numpy.ndarray'>
[[0. 0. 1. 0. 0.]
 [0. 0. 1. 0. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 1.]
 [0. 0. 0. 0. 1.]]

Note that the matrix contains five columns (starting at zero up to four, which is my max. value in the np.array). The first two columns (representing 0 and 1 in the original data) are 0 in the whole matrix, because none of these values are found in the original data.

to_categorical() also takes input which is not explicitly defined as np.array. For instance the statements below would also be legal.

alt1 = to_categorical([0,0,1,1,2,2])
print(alt1.dtype, type(alt1))
print(alt1)

alt2 = to_categorical((0,0,1,1,2,2))
print(alt2.dtype, type(alt2))
print(alt2)

Because the range of values now is between 0 and 2, the result would look like:

[[1. 0. 0.]
 [1. 0. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 0. 1.]
 [0. 0. 1.]]

Pandas: get_dummies()

When you have a Pandas df, you can convert some column to dummies using get_dummies(), regardless of the data type in the column. So it is also possible to convert a column of strings to dummies.

import pandas as pd
df = pd.DataFrame(data={'col1':["A", "A", "B", "B", "C", "C"]})
alt3 = pd.get_dummies(df['col1'])
print(type(alt3))

This gives:

<class 'pandas.core.frame.DataFrame'>
   A  B  C
0  1  0  0
1  1  0  0
2  0  1  0
3  0  1  0
4  0  0  1
5  0  0  1

Note that the result is (again) a Pandas df. So we need to convert it to a np.array.

alt3 = alt3.to_numpy()
print(alt3.dtype, type(alt3))
print(alt3)

This yields:

uint8 <class 'numpy.ndarray'>
[[1 0 0]
 [1 0 0]
 [0 1 0]
 [0 1 0]
 [0 0 1]
 [0 0 1]]

So that it is ready to be used with Keras.

Note that the matrix generated here does not (!) start at zero. Instead each distinct value in the chosen Pandas column gets it's own column in the dummy matrix.


How to get weightage of each factor for a prediction in a neural network classifier?

It's very difficult to understand the effect of each variable on the final output, since the weights of an input variable propagate their effect to all the units of the following hidden layers. Truth is, Neural Networks are very powerful predictors, but they are not very good when it comes to estimate feature importance.

One (very time consuming) way of doing it would be: substitute each variable with random noise that has the same mean and variance, and observe how the model's performance changes. Repeat this for all your variables. If performance gets significantly lower once you substituted a variable with random noise, then the contribution of it to the final accuracy has to be relevant.

As a more viable alternative, train a tree-based classifier (Random Forest or XGBoost) on the output. This class of models returns importance scores. You can use Neural Network for prediction, and the tree-based model to estimate what variable is more relevant for your task.


Benefits of stochastic gradient descent besides speed/overhead and their optimization

On large datasets, SGD can converge faster than batch training because it performs updates more frequently. We can get away with this because the data often contains redundant information, so the gradient can be reasonably approximated without using the full dataset. Minibatch training can be faster than training on single data points because it can take advantage of vectorized operations to process the entire minibatch at once. The stochastic nature of online/minibatch training can also make it possible to hop out of local minima that might otherwise trap batch training.

One reason to use batch training is cases where the gradient can't be approximated using individual points/minibatches (e.g. where the loss function can't be decomposed as a sum of errors for each data point). This isn't an issue for standard classification/regression problems.

I don't recall seeing RMSprop/Adam/etc. compared to batch gradient descent. But, given their potential advantages over vanilla SGD, and the potential advantages of vanilla SGD over batch gradient descent, I imagine they'd compare favorably.

Of course, we have to keep the no free lunch theorem in mind; there must exist objective functions for which each of these optimization algorithms performs better than the others. But, there's no guarantee whether or not these functions pertain to the set of practically useful, real-world learning problems.


Mismatching target dimension in Keras LSTM

With a single output unit model.add(Dense(**1**, activation='softmax')), Keras is expecting a binary outcome, and so the 'softmax' activation function and the 'categorical_crossentropy' loss are not appropriate.

If your output is binary, use: model.add(Dense(1, activation='sigmoid')) and model.compile(loss='binary_crossentropy'.

If your outcome is not binary, then the number of output units needs to be adjusted to match the number of output categories.


Working as data scientist for a nonprofit company

I see at least five ways to approach this problem of finding a data scientist position/work specifically at non-profit, non-governmental or similar organizations, as I describe below. I hope that this is helpful.

First, and the most obvious, way is to search major job portals, such as indeed.com, dice.com, monster.com, CareerBuilder, Glassdoor and others, for data scientist or similar positions, such as data analyst, data engineer, quantitative analyst, statistical modeler, or even market researcher.
The second and also obvious way is to tap your professional social networks and research or inquire your contacts for any potential data science work opportunities in the areas of your interests.
The third way is to search web sites, focused on the non-profit and related topics (they usually have a job listings or similar section) as well as specialized non-profit job portals. For example: http://encore.org, http://idealist.org, http://bridgespan.org, http://cgcareers.org, http://opportunityknocks.org, http://foundationcenter.org, http://thenonprofittimes.com, http://philanthropy.com, http://ynpn.org, http://philanthropyjournal.org, http://nonprofit-jobs.org, CoF Jobs (section isn't easy to find), http://careers.councilofnonprofits.org, http://nonprofittalentmatch.com.
The fourth way is to perform Internet search and research on non-profit or similar organizations that you might be interested in working with (criteria might vary from organization's size to industry focus or geographical locations). Based on the information presented  on their websites, make notes and then approach those organizations, directly inquiring or applying for positions of interest.
The fifth way is to consider various non-profit, for-profit and otherwise social good-themed data science-focused organizations, initiatives and Kaggle-like competitions, such as DataKind, DrivenData, DataLook and Data Science for Social Good.


Best Approach to Forecasting Numerical Value Based on time series and categorical data?

So the question is about to model the next repair date given the previous repairs.
If you have customer-specific data, where you have logged customer repairs to a specific customer, then it would be good idea to do time series, provided you have enough instances of customer repeatedly coming back.
If you have this scenario, then you can use something like an RNN (Recurrent Neural Network [https://towardsdatascience.com/understanding-rnn-and-lstm-f7cdf6dfc14e]) or LSTM, whereby you feed in the date time, type of damage at every time step to then get a single output date time.
If you have not logged a customer ID to the repairs to identify which repairs have been carried out for particular customers, then you could easily use a standard neural network for this.
Here you would train the model on all input data with the goal of getting a date time which equates to when the next repair is likely to happen.
In terms of data representation, you could represent the following features as the following:

Datetime: here, you could this as the day in the year, for example, Jan 1 could be represented as 1 and Dec 31 as 365. This can then be normalised (i.e. divided by 365) to reduce the scale of this feature.
Categorical variables like type of damage, etc: normally we represent these as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).

To make the input for a given entry, we then concatenate these features into one massive vector.
Good luck!


How to choose solution - Neural Neworks or Scikit-Learn/Numpy/Pandas?

There is no magic solution. You will have to try. The rule of thumb I would say is to try firstly classifiers which are less costly to train and deploy (i.e. Scikit-learn) then give it a try with NNs.

Anyways, for your problem, you may be interested to try the Genism library for topic modelling to extract what kind of topics are mentioned in an email.

Then spaCy is a great tool for NLP production tasks, there should be some text classification walkthroughs using it.

These are only a couple of ideas I am sure there are tons more. I believe the key thing is to understand which key indicators, features, words are responsible for an email to be labelled of a particular class.


Detecting off state in the magnitude of accelerometer data?

Seems to be pretty straight-forward thanks to your "well-behaving" data.

Simplest

The naive approach is to check histogram! As you see, the OFF state has generally a low magnitude. Knowing that level of magnitude from the labeld data you have above, you can threshold the histogram.

Why is it naive? :  because you see false positives (see the plot between 4 and 6 PM. there is a short state similar to OFF which is not OFF) and false negatives (immediately after 9 PM you see a peak in OFF state whose magnitude goes far beyond OFF)

what improves this naive approach? :

Taking the history into account (learning the sequential behavior)
Extracting more sophisticated features (e.g. in frequency domain using FFT or time-frequency domain using STFT, wavelet, etc.)
Applying an anomaly detection stage which helps you decide correctly about the peak which happened at 9 PM (that is pretty much an anomaly I assume)

A Bit More Complicated

Extract time frequency features (STFT, Wavelet, etc.) and in case you have labeled data, feed them to a classifier and in case you don't use those features to do some EDA to see if a clustering is possible there. According to your data I'm pretty confident you get good results.

Much More Complicated

I suppose Recurrence Analysis would be a wonderful option (and pretty fun for you!). Specially if the dynamic of the signal is non-stationary (many signals are in practice). The methods explained in the wikipedia page or in this page are not sophisticated.

Good Luck!


Are there any reports on Hardware errors affecting experiments?

Memory corruption seems to be an issue that is important enough for companies to buy expensive ECC memory for their clusters.
The Wikipedia article on ECC memory lists some causes for memory corruption, including (to my surprise and delight) cosmic rays and ingenious hackers.


The usual normalisation for Euclidian distance is NOT to scale each input to unit length, but to scale each column to mean 0 and variance 1. The scaling of each data is possible but it is not common.

I dont know

The whole point of SMOTENC is not to do the one-hot encoding. One hot encoding is a way to transform categorical data into numeric data (on multiple dimensions) for algorithms that cannot deal with categorical data. SO my suggestion is not to convert the categorical columns and let SMOTENC deal with them

The Pipeline in imblearn does the right thing - it only applies the oversampling (of other imbalance strategies) on the training set not on the test set. See this question in StackOverflow: https://stackoverflow.com/questions/63520908/does-imblearn-pipeline-turn-off-sampling-for-testing


How to define the input channel of a CNN model in Pytorch?

The defining factor is which dimensions you want your 2-dimensional convolution sweep over, e.g.:

In images, you want the 2D convolution to sweep over the height and width dimensions, and the extra dimension (the color space) is the channels; for grayscale images, you have a single channel.

In a spectrogram, you want the 2D convolution to sweep over the time and frequency dimensions. As there are no further dimensions, there is only one channel, like with grayscale images.

In the cases you propose, e.g. "3 * 3 * 30", if we want the 2D convolution to happen in the two first dimensions, then the number of input channels would be 30. If we wanted the 2D convolution to sweep over two other dimensions, then the remaining one would be the number of input channels. The same for "30 * 4 * 5".
We should note, however, that 2D convolutions follow a strict convention in the ordering of dimensions. As described in the pytorch documentation, the convention is $(N,C_{in},H,W)$, which means that we should rearrange the dimensions in our input tensor (e.g. with torch.Tensor.permute) to ensure that the dimensions over which we want the 2D convolution to sweep are in the correct order (i.e. the last 2 dimensions).


Based on Keras documentation here, flow_from_directory should contain one subdirectory per class. Any PNG, JPG, BMP, PPM or TIF images inside each of the subdirectories directory tree will be included in the generator. See this script for more details. That means you should probably unzip your zip files.


How to implement Python's MLPClassifier with gridsearchCV?

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer.

If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$.

$(100,1)$ would mean that the second hidden layer only has one neuron.


How does SHAP values help us to determine importance of a feature for a model trained by gradient boost?

I think that the decision tree that appears in the second article is just illustrating the xgboost model that the shap is applied on.

I would like to suggest you to read Christoph Molnar tutorial book on explainability, especially the chapters about Shapley and about the shap algorithm.

The first term (shapely) helps to decompose the effect of each feature on the predicted output by trying all the combinations ("coalitions") of features with the input values (vector). By that, the method allows deconstructing how the output of the model been constructed by each of the input feature values. This approach looks on the features as players that their different strategies affect the final prediction, and it originated in the game theory domain.

The second term, SHAP, is basically calculating the same shapely values with some extensions for missing values that are not handled well in the first and improvements for the algorithm that estimates the shapely values themselves both by speed (the Kernel function that approximates them) and quality of the approximations (the weighting method that combines the combinations).

Besides does algorithmic differences, I think that the shap method is also the approach of using the Shapley algorithm for the usage of interpreting ML (tree-based) models, mainly, by exploiting their additivity property to describe the prediction as a decomposition of the sum of features contributions.


What is the relationship between AI and data science?

Though neither are well defined, as commonly used they are somewhat orthogonal concepts.

In my opinion, AI has a fairly narrow definition - it is about optimization through actions.  AI is about decision making, either in deterministic or probabilistic environments.  Typically, this is operationalized as action selection to maximize some reward function, or equivalently to minimize some loss function.  Supervised or unsupervised learning (i.e. machine learning) about your environment can be helpful in using your experience to aid in selecting optimal actions.

As it's commonly used, data science has no rigorous definition - businesses use the term to refer to anything from creating charts Excel to deep reinforcement learning models that can win Go.  From the point of view of a practitioner, these have absolutely nothing in common. From the point of view of a business owner, the common thread is extracting meaning, and therefore value, from raw data.  A data scientist is a 'meaning extraction layer'.  How this operation is performed, and the techniques used (again in my experience), make no difference to the employer of a data scientist. The job title may as well be 'data magician'.  But the point is that data + data science = business value, whether that comes in the form of insights into customer trends, causal analysis of marketing campaigns, or an AI bot that is 'rewarded' when it recommends you a movie that you like.  I suppose that means that AI is a subset of data science - but you could also say the same thing about clear communication, so it's a bit of a non-statement.


Why does LSTM performs better when the source target is reversed? (Seq2seq)

Also the paper said, we do not have a complete explanation to this phenomenon.

But here is kinda explanation about that.

  While we do not have a complete explanation to this phenomenon, we
  believe that it is caused by the introduction of many short term
  dependencies to the dataset.

  Normally, when we concatenate a source sentence with a target
  sentence, each word in the source sentence is far from its
  corresponding word in the target sentence.

  As a result, the problem has a large “minimal time lag” [17].

  By reversing the words in the source sentence, the average distance
  between corresponding words in the source and target language is
  unchanged.

  However, the first few words in the source language are now very close
  to the first few words in the target language, so the problem’s
  minimal time lag is greatly reduced.

  Thus, backpropagation has an easier time “establishing communication”
  between the source sentence and the target sentence, which in turn
  results in substantially improved overall performance.


Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector

That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


Different performance for splitting into test/train data vs. using cross-validation

Reason for the discrepancy

Two aspects have to be considered regarding the split:

Is the split done in a stratified manner? (it should)
Is the data shuffled? (it should)

The line

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)

splits the data in a stratified manner by default (see parameter stratify) and it does shuffle by default (see parameter shuffle):

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

The line

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')

also splits the data in a stratified manner (see parameter cv:

  For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

), but it does not shuffle. This causes the bad results for this line.

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

Solution

Option 1: Shuffle the data beforehand:

import sklearn
scores = cross_val_score(clf, *sklearn.utils.shuffle(feature_matrix, df.eClass, random_state=42), cv=5, scoring='f1_macro')

Option 2: use appropriate cross-validaton object

I also looked into using an appropriate cross-validation sheme by using a different object:

import sklearn
skf = sklearn.model_selection.StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in skf.split(X, y):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    clf.fit(X_train, y_train.values)
    print('--------------------------------------')
    predicted_train = clf.predict(X_train)
    predicted_test = clf.predict(X_test)
    print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
    print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
    print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
    print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
    print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))
    print('--------------------------------------')

see:

https://scikit-learn.org/stable/modules/cross_validation.html
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html


Accuracy stagnated while training notMNIST data

Use tf.softmax_cross_entropy_with_logits function instead of writing it by yourself because tf.softmax_cross_entropy_with_logits is more computationally stable.

Try this,
loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))

Refrence: https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits


Do i need to handle missing values before EDA?

If the number of rows includes missing values are very small according to sample size I recommend dismissing it.
But if you decide to keep them according to not lose any information, you can do a bunch of things according to the feature that involves a null value.
You should understand the pattern of the feature column well before deciding the filling method below.

You can change the null value as;

mean of the column
median of the column
same as above or below
just zero
most repeated value along the column
etc.

If there is any categorical feature, you can group by a feature like gender and can do the same thing as above. For example, if there is a NaN height value for a male you can fill it with the mean of the heights of males etc.
Besides all you can decide to discard the whole column with:

Checking the correlation between the column and the dependent variable
Checking the column's representation level of source data with PCA


A couple good options would be to look at a Feature/Variable importance plot for your RF model.  Alternatively, depending on the model you could try extracting a couple individual trees from the model and examining them.  However, these methods  wouldn't be definitive; i.e. determining what variables are strong predictors for churn does not mean that they have a causal impact on churn, and an individual tree may be biased and not representative of the aggregation output presented by the RF model. To determine causation, you could use these methods as a starting point to design a test.


Following this example:

Suppose the last column in train_mat is the label that needs to be predicted, i.e. Global_Sales, then you construct the xgb.DMatrix as:

n <- ncol(train_mat)    # or if not the last, the position of the label column
dtrain <- xgb.DMatrix(train_mat[,-n], label=train_mat[,n])

To train the model, simply pass dtrain to the data parameter:

lm.boosted <- xgb.train(data = dtrain, booster= "gblinear", max_depth=3, nthread=1, nrounds = 2, eval_metric="error", eval_metric= "rmse", objective= "reg:linear")


Is the PNY NVIDIA Quadro RTX 4000 a good GPU for Machine Learning on Linux?

You seem to be looking at the latest Quatro 4000, which has the following compute rating:

You can find the complete list here for all Nvidia GPUs.

While it seems to have an impressive score of 7.5 (the same as the RTX 20180ti), the main draw back the memory of 8Gb. This is definitely enough to get started with ML/DL and will allow you to do many things. However, memory is often the thing that will slow you down and limit your models.

The reason is that a large model will require large number of parameters. Take a look at the following table (models included in Keras), where you can see the number of parameters each model requires:

The issue is that the more parameters you have, the more memory you need and so the smaller the batch size you are able to use during training. There are many arguments for larger vs. smaller batch sizes - but having less memory will force you to still to smaller batch sizes when using large models.

It seems from Nvidia's marketing, that the Quadro product line is more aimed towards creative developers (films/image editing etc.), whereas the Geforce collection is for gaming an AI. This highlights that Quadro is not necessarily optimised for fast computation.


What does Make Density Based Clusterer in Weka do?

Based on this paper, MakeDensityBasedClusterer is  a  metaclusterer  that  wraps  a  clustering  algorithm  to make  it  return  a  probability  distribution  and  density.   To  each  cluster  and  attribute,  it  fits  a discrete distribution or a symmetric normal distribution (whose minimum standard deviation is a parameter).


Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

It's because you need to fine-tune BERT for your specific task anyway. You can train it to classify based on either cls token, or mean of token outputs, or whatever.
In essence, CLS token of the last layer has connections with all of the other tokens on the previous layer. So, does it make sense to average manually?


How to find AUC metric value for keras model?

There is now a built-in function to compute (an approximation of) the AUC. See tf.keras.metrics.AUC. Apparently, you just need to do the following

...
model = tf.keras.Model(inputs, outputs)
model.compile('sgd', loss='mse', metrics=[tf.keras.metrics.AUC()])


What interactive graph package do you use for timeseries data?

I didn't get an answer so here's my take.  Ideally, I would use holoviews and the pyviz packages...to some extent i can, so i will continue to try this method.  However, many of the examples do not work.


Resume Parsing - extracting skills from resume using Machine Learning

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely.

Here are a few sources I found that might be helpful:

A resume parser
The reply to this post, that gives you some text mining basics (how to deal with text data, what operations to perform on it, etc, as you said you had no prior experience with that)
This paper on skills extraction, I haven't read it, but it could give you some ideas

For some personal suggestions :

I'm going to make the assumption you don't want to label those resumes by hand, and will try to explain how I would proceed first.
It might not be the best solution, as there's a lot of heuristics in it, but it's a beginning that's mostly aimed at studying your data and gain some insight on it.

You could see if the structure of the document helps: the skills you're looking for might often be appearing in a section with a specific title.
You could also work with gazeteers, which are lists of key words of interest, and combine these with colloquations (groups of words frequently appearing together). If, for example, you have a keyword appearing in your gazeteer, you could use ngrams to see what words appear around it the most, and turn your single word match into a multiple word skill (e.g : machine learning (2grams), natural language processing (3grams), etc)
Once you have an interesting list of keywords, you could move to ML and try word vectors, maybe use word2vec, and build vectors around your word of interests. If new skills come in, their vectors might be similar to the ones you have defined (sorry for being a bit unclear here, I only have a vague knowledge of how word2vecs work).

This is no silver bullet solution, but this should get you started.

Good luck !


How to extract paragraphs from text document?

It's not always possible to extract paragraphs from a pdf since sometime paragraph are split into multiple pdf frames so pdftotext split them into different paragraph even if there are actually linked. Similarly some frames ends collocated even they represent different information like the menu in the example pdf.

Here is a simple approach to split a text file into multiple paragraph using empty lines:

def txt2paragraph(filepath):
    with open(filepath) as f:
        lines = f.readlines()

    paragraph = ''
    for line in lines:
        if line.isspace():  # is it an empty line?
            if paragraph:
                yield paragraph
                paragraph = ''
            else:
                continue
        else:
            paragraph += ' ' + line.strip()
    yield paragraph


Background

I'll start with some background to help you research the solution yourself and then will add some specifics.  What you refer to as "biased data" is more commonly known as unbalanced classes in the data science world.  Also "customer turnover" is often referred to as churn.

Metrics

As hoards of Ng'ian devotees will undoubtably point out you need to start by designing a set of metrics that work better with unbalanced classes than accuracy. Accuracy does a poor job in testing the quality of predictions for unbalanced classes e.g. a cancer test for a cancer that occurs in 0.05 % of the population is 99.95% accurate if it always predicts "no cancer". I suggest using the F1-score as the key metric in cross-validating your model.  The F1-score is the harmonic mean of precision and recall and tends to work both for balanced and unbalanced classes.  There are other rations of harmonic mean, that could work in special cases, so be aware of these.

There are other metrics you should learn about also. ROC-AUC is likely at the top of the list for other metrics you should understand and know about.

Model Selection and Cross-Validation

Beginning a classification task with Logistic Regression is a fantastic strategy. I make a point to always use a linear regression for regression tasks and a logistic regression for classification tasks. The linear model provides significant insight into the feature importance and helps frame the problem.

But following this initial survey you should move on to other, more sophisticated models. Many will give you a litany of things to try. You should perhaps focus on one or two and develop the model while paying very careful attention to bias and variance as you cross-validate and test your model. A full bias-variance decomposition may be unnecessary, once you develop better intuition, but is a great place for newbs to start.

I suggest starting with an SVM and also eventually trying a random forest or naive Bayes model as this will traverse several regimes of model types (analogy, decision trees, bagging, Bayesian).

Finally... Unbalanced Classes

There are two typical methods for dealing with unbalanced classes. These include oversampling the minority class, and fixing the model by altering the hyperplane (SVM) or changing priors (Bayes).

There are lots of summaries of this problem and solution if you search for "unbalanced classes". But, it can still be a tricky problem despite all the literature.  Good luck...

Hope this helps!


This is one of the famous implementations for your task. It works well mostly. If you just need such a tool you can use it. However, if you want to develop your own tools you might want to analyze its structure.
It is also able to look for specific skills in the resumes as mentioned here.
As in your requirement, it accepts pdf and also doc and returns json.


How do I use LSTM Networks for time-series classification problems?

This can be done with RNN/LSTM/GRU (type of Neural Networks that are well-suited for time-series).

For example : https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

This example is quite similar to the problem mentioned in question (predict air quality based on ~10 parameters. Parameters are available as a time-series).


Look at the NLTK library for Python, there are functions to facilitate tokenizing sentences.


No you cannot. I had a discussion with someone in their development/support team on the MSDN forums and currently they don't support 'drag and drop' type of functionality. However you CAN serialize the model output and then de-serialize them in Azure.

Note that the answer in the image is a bit outdated and there is the 'Create R Script' module to replace the serialization-deserialization steps within Azure. However I believe you can still serialize outside Asure (in your Desktop version of R) and deserialize them in Azure.

Link to the conversation in Image:
https://social.msdn.microsoft.com/Forums/azure/en-US/5944c342-79ac-4ada-8006-8edf40f36ee1/r-script-as-a-trained-model?forum=MachineLearning


Group similarity by high dimension vector comparison

You can't actually prove that the two groups are similar but you can establish a confidence level/threshold. Furthermore, it is possible that the two groups won't be similar (depending on your threshold for similarity) if, for example, only one of the two groups contains strong outliers.

That said, you can make comparisons based on assumptions regarding the underlying data distribution. For example, if the data can be assumed to be distributed as a multivariate normal distribution, you can use Hotelling's two-sample T-squared statistic (a multivariate generalization of the Student's t-test) to test your confidence interval.


Quadratic approximation of L1 regularized cost function

The analysis becomes a little bit hairy because of the discontinuous derivative of the L1 penalty.  There may be a better way of phrasing the argument, but here's something.

Taking the derivative of $|w_i|$ gives $\operatorname{sgn}(w_i)$, unless $w_i=0$ which we will take as a special case.
In the main case, when $w_i\neq0$, taking the derivative w.r.t. $w_i$ equal to zero gives:
$$\begin{align*}
H_{ii}(w_i-w_i^*) + \alpha \operatorname{sgn}(w_i) & = 0 \\
\Longrightarrow w_i &= w_i^* - \frac{\alpha \operatorname{sgn}(w_i)}{H_{ii}}.
\end{align*}$$
Now for a trick.  From the original loss formulation, it's clear that the optimum $w_i$ will have the same sign as $w_i^*$ (or is zero): otherwise, switching signs decreases the first term and doesn't change the second term.  So we have
$$\begin{align*}
w_i &= w_i^* - \frac{\alpha \operatorname{sgn}(w_i^*)}{H_{ii}} \\
 &= \operatorname{sgn}(w_i^*) \left( |w_i^*| - \frac{\alpha}{H_{ii}} \right).
\end{align*}$$
And again, knowing that $w_i$ and $w_i^*$ must share sign, we must have that $|w_i^*|-\frac{\alpha}{H_{ii}}\geq0$.  If not, then this is not a location of zero-derivative, and we have to look to our other case.  In that case, simply $w_i=0$, and we can tack that on as a separate case here using the maximum given in the text.  (We should also confirm that $w_i=0$ is not the minimum when $|w_i^*|-\frac{\alpha}{H_{ii}}>0$, which is easy to do.)


Data science without knowledge of a specific topic, is it worth pursuing as a career?

Drew Conway published the Data Science Venn Diagram, with which I heartily agree:

On the one hand, you should really read his post. On the other hand, I can offer my own experience: my subject matter expertise (which I like better as a term than "Substantive Expertise", because you should really also have "Substantive Expertise" in math/stats and hacking) is in the retail business, my math/stats are forecasting and inferential statistics, and my hacking skills lie in R.
From this vantage point, I can talk to and understand retailers, and someone who doesn't have at least a passing knowledge of this field will have to face a steep learning curve in a project with retailers. As a side gig, I do statistics in psychology, and it's exactly the same there. And even with quite some knowledge of the hacking/math/stats part of the diagram, I would have a hard time getting up to speed in, say, credit scoring or some other new subject field.
Once you have a certain amount of math/stats and hacking skills, it is much better to acquire a grounding in one or more subjects than in adding yet another programming language to your hacking skills, or yet another machine learning algorithm to your math/stats portfolio. After all, once you have a solid math/stats/hacking grounding, you could if need be learn such new tools from the web or from textbooks in a relative short time period. But the subject matter expertise, on the other hand, you will likely not be able to learn from scratch if you start from zero. And clients will rather work with some data scientist A who understands their specific field than with another data scientist B who first needs to learn the basics - even if B is better in math/stats/hacking.
Of course, all this will also mean that you will never become an expert in either of the three fields. But that's fine, because you are a data scientist, not a programmer or a statistician or a subject matter expert. There will always be people in the three separate circles who you can learn from. Which is part of what I like about data science.

EDIT: A little while and a few thoughts later, I'd like to update this post with a new version of the diagram. I still think that Hacking Skills, Math & Statistics Knowledge and Substantive Expertise (shortened to "Programming", "Statistics" and "Business" for legibility) are important... but I think that the role of Communication is important, too. All the insights you derive by leveraging your hacking, stats and business expertise won't make a bit of a difference unless you can communicate them to people who may not have that unique blend of knowledge. You may need to explain your statistical insights to a business manager who needs to be convinced to spend money or change processes. Or to a programmer who doesn't think statistically. Caulcutt (2021, Significance) is a short article that says much the same, but gives more detail than I do.
So here is the new data science Venn diagram, which also includes communication as one indispensable ingredient. I have labeled the areas in ways that should guarantee maximum flaming, while being easy to remember.
Comment away.

R code:
draw.ellipse <- function(center,angle,semimajor,semiminor,radius,h,s,v,...) {
    shape <- rbind(c(cos(angle),-sin(angle)),c(sin(angle),cos(angle))) %*% diag(c(semimajor,semiminor))
    tt <- seq(0,2*pi,length.out=1000)
    foo <- matrix(center,nrow=2,ncol=length(tt),byrow=FALSE) + shape%*%(radius*rbind(cos(tt),sin(tt)))
    polygon(foo[1,],foo[2,],col=hsv(h,s,v,alpha=0.5),border="black",...)
}
name <- function(x,y,label,cex=1.2,...) text(x,y,label,cex=cex,...)

png("Venn.png",width=600,height=600)
    opar <- par(mai=c(0,0,0,0),lwd=3,font=2)
        plot(c(0,100),c(0,90),type="n",bty="n",xaxt="n",yaxt="n",xlab="",ylab="")
        draw.ellipse(center=c(30,30),angle=0.75*pi,semimajor=2,semiminor=1,radius=20,h=60/360,s=.068,v=.976)
        draw.ellipse(center=c(70,30),angle=0.25*pi,semimajor=2,semiminor=1,radius=20,h=83/360,s=.482,v=.894)
        draw.ellipse(center=c(48,40),angle=0.7*pi,semimajor=2,semiminor=1,radius=20,h=174/360,s=.397,v=.8)
        draw.ellipse(center=c(52,40),angle=0.3*pi,semimajor=2,semiminor=1,radius=20,h=200/360,s=.774,v=.745)

        name(50,90,"The Data Scientist Venn Diagram",pos=1,cex=2)
        name(8,62,"Communi-\ncation",cex=1.5,pos=3)
        name(30,78,"Statistics",cex=1.5)
        name(70,78,"Programming",cex=1.5)
        name(92,62,"Business",cex=1.5,pos=3)

        name(10,45,"Hot\nAir")
        name(90,45,"The\nAccountant")
        name(33,65,"The\nData\nNerd")
        name(67,65,"The\nHacker")
        name(27,50,"The\nStats\nProf")
        name(73,50,"The\nIT\nGuy")
        name(50,55,"R\nCore\nTeam")
        name(38,38,"The\nGood\nConsultant")
        name(62,38,"Drew\nConway's\nData\nScientist")
        name(50,24,"The\nperfect\nData\nScientist!")
        name(31,18,"Comp\nSci\nProf")
        name(69,18,"The\nNumber\nCruncher")
        name(42,11,"Head\nof IT")
        name(58,11,"Ana-\nlyst")
        name(50,5,"The\nSalesperson")
    par(opar)
dev.off()


you might want to look into 'approximate nearest neighbors' analysis, and particularly the annoy package from Spotify:

https://github.com/spotify/annoy

It's not as precise as an exact comparison, but it may be close enough to get you what you need.


Hmm the first few ideas of other things you could look for to make your model a little more sharp...

Examine id's of the images/metadata to see if a statistically significant pattern emerges for how to arrange the images. (i1.png,i2.png,i3.png,etc),(Largest image is always on top, etc)
They may purposely scramble the images in a way were they never let a picture be purposely on the right location which can be used to your advantage. You only have to decide in between three locations the correct location for any image.
You may want to see if you can incorporate word2vec and/or LSTM for getting image to->words to->story syntax/context correct.
If they allow two chances before reseting then your model will be more accurate if you can incorporate that. If it outputs a probability each of the images are at a location then you need to create two guesses not one.

Chances are low but I would be interested to see your results I think if they didn't make a terrible captcha you will have about 16% < x < 30 accuracy if they have systemic flaws could be a lot better.


Kappa near to 60% in unbalanced (1:10) data set

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics.

What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to the level of agreement that would occur through random chance. There's a very good overview of how to calculate Kappa and use it to evaluate a classifier in this stats.stackexchange.com answer here, and a more in-depth explanation of Kappa and how to interpret it in this paper, entitled "Understanding Interobserver Agreement: The Kappa Statistic" by Viera & Garrett (2005).

The benefit of using Kappa, particularly in an unbalanced data set like yours, is that with a 90-10% imbalance between the classes, you can achieve 90% accuracy by simply labeling all of the data points with the label of the more commonly occurring class. The Kappa statistic is describing how well the classifier performs above that baseline level of performance.

Kappa ranges from -1 to 1, with 0 indicating no agreement between the raters, 1 indicating a perfect agreement, and negative numbers indicating systematic disagreement. While interpretation is somewhat arbitrary (and very task-dependent), Landis & Koch (1977) defined the following interpretation system which can work as a general rule of thumb:

Kappa Agreement
< 0 Less than chance agreement
0.01–0.20 Slight agreement
0.21– 0.40 Fair agreement
0.41–0.60 Moderate agreement
0.61–0.80 Substantial agreement
0.81–0.99 Almost perfect agreement

Which would indicate that your algorithm is performing moderately well. Accuracy SD and Kappa SD are the respective Standard Deviations of the Accuracy and Kappa scores. I hope this is helpful!


How to make tool that's robust to user-generated typos?

TL;DR

Use fuzzy string matching to account for spelling mistakes.
Solving Jennifer for Alice will require you to know where to look in your DB for these cases, or talk to make a better Excel file to force people to only input first names (e.g. make a an entry to a cell restricted to a given list).

Fuzzy String Matching

In R, you can use adist or the stringdist package. These can be used to measure the distance from an entry (e.g. Aalice to a list of potential matches [Alice, Bianca, Chris].

Here is an articles explaining how to use both.

An extract from the article:

source1.devices<-read.csv('[path_to_your_source1.csv]')
source2.devices<-read.csv('[path_to_your_source2.csv]')
# To make sure we are dealing with charts
source1.devices$name<-as.character(source1.devices$name)
source2.devices$name<-as.character(source2.devices$name)

# It creates a matrix with the Standard Levenshtein distance between the name fields of both sources
dist.name<-adist(source1.devices$name,source2.devices$name, partial = TRUE, ignore.case = TRUE)

# We now take the pairs with the minimum distance
min.name<-apply(dist.name, 1, min)

match.s1.s2<-NULL
for(i in 1:nrow(dist.name))
{
    s2.i<-match(min.name[i],dist.name[i,])
    s1.i<-i
    match.s1.s2<-rbind(data.frame(s2.i=s2.i,s1.i=s1.i,s2name=source2.devices[s2.i,]$name, s1name=source1.devices[s1.i,]$name, adist=min.name[i]),match.s1.s2)
}
# and we then can have a look at the results
View(match.s1.s2)

All this assumes that you have a list of names that are actually valid.


I would normally suggest something like

merged_df = pd.merge(df_compiler, df_opt, on="instructions")

But the third image you posted does not really look like the inner merge of the other two. Can you describe what exactly you are trying to achieve in a little more detail? The third one looks like ten times longer than the other two.


Orange Linear Regression and scikit-learn linear regression gives different results

Orange and Scikit treat outliers in different ways. If you have outliers in your data, remove them and check. The answers then are similar.


Large no of categorical variables with large no of categories

I used to work a lot with such data in the past.
When you have many categorical features with high cardinality, you must avoid one-hot encoding them because it consumes too much memory and above all trees built from them will be too deep and worse than trees built without one-hot encoding.
You need a Tree based model library implemented with native categorical feature support (i.e. not needing the categorical features to be one-hot encoded).
I suggest you to use one of those 4 implementations :
- CatBoost
- LightGBM
- H2O GBM
- H2O Random Forest
Scikit learn and XGBoost implementations still need one-hot encoded categorical variables so I don't recommend using one of these libraries if your dataset has high cardinality categorical variables (i.e. with more than about 10 levels/categories).
See this :

  LightGBM offers good accuracy with integer-encoded categorical
  features. LightGBM applies Fisher (1958) to find the optimal split
  over categories as described here. This often performs better than
  one-hot encoding.


Why using a frozen embedding layer in an LSTM model

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that the embedding has learnt a type of association between words.

  A pretrained embedding like Word2Vec will produce vectors for words like school and homework which are similar to each other in the embedding space.

Many such associations are learnt after rigorous training mostly on high-end machines and precisely calculated parameters.

Why is the Embedding layer set to trainable=false?

As mentioned in the code, we have given a pretrained embedding matrix to the Embedding layer through the weights= argument. As the word suggests, its "pretrained" and requires no additional training.

We can enjoy the benefits of such an embedding by keeping it untrainable.
Additional training in the context of our task, may result in unusual behaviour of the Embedding layer and also distort the learned associations.

In some cases, the Embedding layer is kept trainable.


Unable to generate error bars with seaborn

I found this question when trying to find a way to add my own precalculated, custom error bars (standard deviations) to a Seaborn barplot with grouped values. I finally managed to find a workaround... The idea is to duplicate the observations drawing from a normal distribution with the desired mean and standard deviation. Then, the built-in "ci" option in sns.barplot does the rest. Not the cleanest solution... but it does the trick (at least for small datasets). An example:

import numpy as np
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt

dfBarPlot = pd.read_csv('my_precalculated_values.csv')
#duplicate observations to get good std bars
dfCopy = dfBarPlot.copy()
duplicates = 30 # increase this number to increase precision
for index, row in dfBarPlot.iterrows():
    for times in range(duplicates):
        new_row = row.copy()
        new_row['Y'] = np.random.normal(row['Y'],row['precomputed_std'])
        dfCopy = dfCopy.append(new_row, ignore_index=True)

# Now Seaborn does the rest
sns.set_style("whitegrid")
fig = sns.barplot(x='X',
                  y='Y',
                  hue='Cases',
                  ci='sd',
                  data=dfCopy)

plt.legend(loc='upper right')
sns.set(rc={'figure.figsize':(8,5)})
plt.show()


Find and count similar price for each store and output the max count and its corresponding rows

It might not be the fastest, but it'll work:
def condition(row1, row2, threshold):
    return abs(row1['price']-row2['price'])<=threshold

def row_counter(this_df, this_row, threshold):
    return this_df.apply(lambda row: condition(this_row, row, threshold), axis=1).sum()

def counter(this_df, threshold):
    return this_df.apply(lambda row: row_counter(this_df, row, threshold), axis=1)

threshold = 2
result = df.set_index('item_id').groupby('store_id').apply(lambda store_data: counter(store_data, threshold))

The result is:
store_id      item_id
1d632be3f72c  6dd5392a9991    2
              363a268ae1bc    1
              fcd248a3fe97    1
              20d197a04656    2
b5d61bc3e61   16740613e6af    1
b5d61bc3e6d1  54c6463ffc87    3
              1b62f63eac43    3
              4ed99ff1bcdf    2
              6e19d5b8e99b    3
              89c9b4655a9d    2
dtype: int64


How to force DecisionTreeRegressor to use polyfit equation instead of mse at leaf level in python SKlearn

This is an expanded answer based on my comment.

First of all, there are many interesting highly non-linear classifiers besides decision trees. Most widely used are neural networks and SVMs. (Nitpick but SVMs can be seen as neural networks.)

Within the world of decision trees, there are things like decision trees that have linear regressions in the nodes. However, these are very slow to train. The problem is that you can no longer take advantage of the fact that decision trees entropy or gini score can be calculated incrementally as the threshold $t$ is varied (where $t$ is the right-hand part of the node rule, $x_i < t$). This trick can not be used for these trees. There are of course many other rule induction algorithms besides decision trees, which are not as step-like.

Anyhow, usually ensembles are used. Ensembles are models made of several models, which then "vote" on the result. You can use them with any class of models, but they are particularly excellent when combined with decision trees because:

Decision trees can have very different results if the sample is slightly different. Ensembles fix that;
Decision trees decision boundaries are orthogonal to the features, which have "step" effects;
Decision trees have to be pruned, while some ensembles can be used without pruning;
Most winners of Kaggle and other data mining competitions have used decision trees ensembles;
Ensembles give you probabilities if you need them; probabilities from decision trees are highly dependent on how you prune them (and unpruned decision trees cannot give you probabilities at all).

See here the results of some decision trees ensembles:

There are two types of ensembles widely used:

Random Forest: where each decision tree is trained independently.
Gradient Boosting Trees: these are based on AdaBoost. The idea is that the next tree is trained by weighting more strongly the observations that the previous trees have not been able to qualify correctly.

Usually, Kaggle winners use gradient boosting trees, especially using the xgboost package. But they have more hyperparameters to tweak, so if you don't have much time to tweak them, just go with a random forest. Gradient boosting trees are more ticky to get right, and the results can be different if you change parameters like n_estimators. Random forests, on the other hand, tend to the expected behavior when n_estimators$\rightarrow\infty$.

Also, gradient boosting trees can overfit, albeit I think it's rare. (Random forests can also overfit, but it is because the underlying estimator is overfitting. They tend to the average behavior as you increase n_estimators. Tell me if this isn't clear.)

Note: n_estimators in sklearn is sometimes very low, and you should increment it. In real problems, people use thousands of trees.


For a linear model without intercept, why does the redundent term in one-hot encoding function as intercept?

Let's have a look how dummies work:
R Example:
# Some data
df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1))

# 1) Regression with constant and dummy
summary(lm(y~gender,data=df))

# 2) Regression without constant and dummy
summary(lm(y~gender-1,data=df))

# 3) Regression without constant and two dummies
summary(lm(y~gender+gender2-1,data=df))

Results:
Case 1:
Since dummies generally work as "contrasts" to some base category (1 vs. 0 / "on" vs. "off") and since the base category has a mean of 10, the intercept term equals 10 and for gender = 1, the difference to the base category is identified (here 20) because the mean of category gender = 1 is 30. (Remember that a regression with only an intercept or with dummies simply gives the arithmetic mean).
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  10.0000     0.9129   10.95 0.000394 ***
gender       20.0000     1.2910   15.49 0.000101 ***

Case 2:
Without a constant only gender = 1 is considered, since for gender = 0 we have $0 + 0 * \beta$ so that gender = 0 is dropped. The coefficient now is the mean of gender = 1.
Coefficients:
       Estimate Std. Error t value Pr(>|t|)
gender   30.000      4.546   6.599   0.0012 **

Case 3:
Including a dummy for both groups (denote gender = 0 from above as gender2) without adding an intercept now gives the mean for each group directly. Note that the interpretation of the coefficients is different here compared to case 1.
Coefficients:
        Estimate Std. Error t value Pr(>|t|)
gender   30.0000     0.9129   32.86 5.11e-06 ***
gender2  10.0000     0.9129   10.95 0.000394 ***

The interesting bit is when you add some additional $x$:
Some new data, now including $x$:
df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1), x=c(20,22,25,28,30,29))

Regression with both dummies, no intercept:
summary(lm(y~gender2+gender-1+x,data=df))

Coefficients:
        Estimate Std. Error t value Pr(>|t|)
gender2  19.8864    12.6285   1.575   0.2134
gender   37.6136     9.7446   3.860   0.0307 *
x        -0.3409     0.4342  -0.785   0.4897

is the same as...
Regression with one dummy and intercept (apart of the dummy interpretation discused above):
summary(lm(y~gender+x,data=df))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  19.8864    12.6285   1.575   0.2134
gender       17.7273     3.1973   5.544   0.0116 *
x            -0.3409     0.4342  -0.785   0.4897

... so the marginal effect of $x$ is the same. This is in contrast to...
Regression with one dummy, no intercept:
summary(lm(y~gender+x-1,data=df))

Coefficients:
       Estimate Std. Error t value Pr(>|t|)
gender 22.38736    1.41677  15.802 9.37e-05 ***
x       0.34086    0.03864   8.822 0.000911 ***

Here the marginal effect of $x$ is entirely different.
Why is this?
When you fit some new data, you will see that the fitted line for $x$ goes through (0,0) "no intercept in the model".
newdata = data.frame(gender=c(0,0,0,0,0,0), x=c(-1,0,1,2,3,4))
predict(lm(y~gender+x-1,data=df), newdata=newdata)

         1          2          3          4          5          6
-0.3408643  0.0000000  0.3408643  0.6817286  1.0225929  1.3634572

This happens because there are cases in which you have $0 + \beta x$ (which is 0 for $x=0$). Or as $x$-matrix (first row would be the intercept, for illustration = 0):
\begin{pmatrix}
0 & 1 & x_1\\
0 & 1 & x_2\\
0 & 0 & x_3\\
0 & 0 & x_4
\end{pmatrix}
However, when you have the two dummies included, you have:
\begin{pmatrix}
0 & 1 & x_1\\
0 & 1 & x_2\\
1 & 0 & x_3\\
1 & 0 & x_4
\end{pmatrix}
So there is no case in which you force $\beta x$ to be zero.
See this post for further discussion on regression without constant term.


One possible solution is the following:

import re
import pandas as pd

pattern = re.compile('blood', re.IGNORECASE)

def change(text):
    if pattern.findall(text):
        return 'Blood test'
    else:
        return text

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]
labels = ['title', 'description']

# Create data frame
labtest = pd.DataFrame.from_records(values, columns=labels)
labtest['title'] = labtest['title'].apply(change)

print labtest

The output was:

        title description
0  Blood test          AB
1  Blood test          AB
2  Blood test          AB
3  Blood test          AB

The first line import the regex (regular expression) module of Python. The line:

pattern = re.compile('blood', re.IGNORECASE)

creates a regex that finds the word blood ignoring case. The function change, replace the input text with 'Blood test' in case the string 'blood' was found. Finally you used the apply method from pandas DataFrame to transform the column. Finally the apply method, as the name suggests, 'applies' the function change to every value in the 'title' column.

More info on regular expressions using Python and pandas apply method can be found here and here. If you want to know more about text processing in Python I would recommend you take a look at the pointers in this question.


Improve test accuracy for TensorFlow CNN

This is clearly a case of overfitting, since your validation error is way bigger than your training error. Having understood that overfitting is your problem, there are many ways you can address this problem:

Dropout (keras)
Weight decay (L2 regularization)
Early stopping (keras)

I can see that you are already doing dropout, so setting the dropout higher should help. Moreover, you can try using dropout in more layers, not just the two last ones. If setting the dropout rate higher and setting more layers with dropout does not work, then this probably means that your training and validation data do not come from the same distribution. You can study statistical properties of your training data and validation and study if they come from the same distribution.

Extra comment: if there are just two classes, why don't you use a sigmoid activation function in the last layer?


How to deal with skewed imbalanced image dataset to work with CNN?

I think that you first need to consider what are the class proportions in the data that you are going to eventually use the model for. It is important to have a test set that comes from the same data generating distribution as the real data. Now, in case the proportions in this data are the same as you wrote than it is actually a good thing to have more examples for the classes that are more common. It allows your model to give those classes a bigger prior and focus more on those classes, extract more features relevant to those classes particularly.

In case you'd like to reduce this skew (for example because the proportions in the real data are different or the imbalance is so big that your model learns to assign always just the most common class and ignores others) you can use resampling: define what class proportions would suit you and randomly sample examples using those ratios.


Generate series of values using Keras GAN architecture

In case it will help someone in the future.
This is what solved it...
def create_G(num=100):
    G_in = Input(shape=noise_size)
    x = Reshape((-1,1))(G_in)
    x = Conv1D(2,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(4,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(8,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(16,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(32,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(64,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(100,3,activation='tanh')(x)
    x = Flatten()(x)
    G = Model(G_in,x,name='Generator')
    G.compile(loss=wasserstein_loss,optimizer=G_rmsprop_optimizer)
    return G

def create_D(num=100):
    D_in = Input(shape=num)
    x = Reshape((-1,1))(D_in)
    x = Conv1D(num/2,3,activation=LeakyReLU())(x)
    x = Conv1D(num/10,3,activation=LeakyReLU())(x)
    x = Conv1D(num/50,3,activation=LeakyReLU())(x)
    x = Flatten()(x)
    x = Dense(128,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(64,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(32,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(8,activation=LeakyReLU())(x)
    x = Dense(2,activation='sigmoid')(x)
    D = Model(D_in,x,name='Discriminator')
    D.compile(loss=wasserstein_loss,optimizer=D_rmsprop_optimizer)
    return D

Cheers :)