tree.DecisionTree.feature_importances_ Numbers correspond to how features?

You can take the column names from X and tie it up with the feature_importances_ to understand them better. Here is an example -

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
import pandas as pd

clf = DecisionTreeClassifier(random_state=0)
iris = load_iris()
iris_pd = pd.DataFrame(iris.data, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
clf = clf.fit(iris_pd, iris.target)

I am taking the iris example, converting to a pandas.DataFrame() and fitting a simple DecisionTreeClassifier. Once the training is done, you can take the columns attribute of a pandas df and make a dict with the feature_importances_ output.

print(dict(zip(iris_pd.columns, clf.feature_importances_)))

This will give you what you want -

{'sepal_length': 0.0, 'sepal_width': 0.013333333333333329, 'petal_length': 0.064055958132045052, 'petal_width': 0.92261070853462157}

Hope this helps!


Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the process, so you will have to approximate with a flexible enough model. If the model is not flexible enough you will not need to regularize but you won't approximate well anyway. If you use a more flexible model, you will get closer on average (low bias) but you will have more variance, thus the need for increased regularization. In other words, the amount of regularization you need depends on the model. This is related to the bias-variance trade-off.

Welcome to DataScience.SE.


Scikit Learn Logistic Regression Memory Leak

Ok, this ended up being an RTFM situation, although in this case it was RTF error message.

While running this, I kept getting the following error:

DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().

I assumed that, since this had to do with the target vector, and since it was a warning only, that it would just silently change my target vector to 1-D.

However, when I explicitly converted my target vector to 1-D, my memory problems went away. Apparently having the target vector in an incorrect form caused it to convert my input vectors into dense vectors from sparse vectors.

Lesson learned: follow the recommendations when sklearn 'suggests' you do something.


Is there any reason not to use classification instead of regression when looking for ranges?

The general answer is how the model will be used deal. Either way may be optimal for the case.
For example - If the model groups applicants into good credit risk and bad credit risk, that might be fine to say model score > x = good risk and model score <= x = bad risk. But maybe there will be differential action based on the model score - like giving a different interest rate or a bigger loan.
In the original example, in regression actual = 191, predicted = 189 you can calculate the loss.
In classification, if actual = 191 and P(>190)=0.35, P(170-190)=0.40, P(150-170)=0.25, then you just know the wrong class. Is that enough for the model usage?
There is also the assumption that a "closer" class will be chosen but that might not be true, e.g actual=191, P(>190)=0.25, P(170-190)=0.25, P(150-170)=0.5. The regression could come up with 160 also but you can measure that loss if the model usage requires it. Many classification algorithms do not know if classes are "close" - Confusion matrix. "How close I am to the diagonal?". Is there such metric?
You can also look at Ordinal Regression https://en.wikipedia.org/wiki/Ordinal_regression. In this case there is an implicit ranking in the "class".
Choose based on how the model will be used. Always important to know the usage and the problem being solved, then work backwards to the model.
Hope that helps.


How to barplot output of pandas.describe() from multiple datasets

You need to get used to so called wide and long table format, from there you should get the trick rapidly. By then, I would use unstack in order to have three columns and do something like this:

unstacked_data = describedWidth.unstack()
unstacked_data = unstacked_data.reset_index()
unstacked_data.columns = ['metric', 'run', 'y']
sns.barplot(data=unstacked_data, x="metric", y="y", hue="run" )

Which should give a table that has the following format:

And final result:


Equivalence of Tidy Data and Third Normal Form

I believe it goes a like this, based on thinking about this myself:

Hadley's rule 1 & 2 (call it TR1 & TR2) is equivalent to 1NF with the exception that Hadley's rule 1 would permit repetition of columns. All we are saying in TR1 & TR2 is that we have a single value in each cell of our table, although I think that is dependent on how you interpret observation. If an observation is an instance of a specific object then this is a stronger statement, although TR3 suggests that is not the case I think.

The combination of Hadley's rule 1, 2 and 3 is equivalent to 3NF. Using this link which also explains transitive dependence:

The table with UnitCode, UnitName, CourseCode, CourseName is in 2NF but violates TR3 since course and unit are different objects. The tables in 3NF where course and unit are separated and linked by foreign key comply with TR1, TR2 and TR3.

Let's say we want to prove implication and assume we have TR1, TR2 and TR3. Since those say nothing about repeating columns I think we have to either assume no repetition of columns or accept that these are not equivalent. Let's assume no repetition of columns. From the definitions TR1 & TR2 => 1NF. If we have a table that is not in 2NF then TR3 is violated using the logic in the example above. Therefore we have 2NF. Similarly if we have a table that is not in 3NF then TR3 is violated, again using the logic above. Therefore TR1 & TR2 & TR3 => 3NF.

I think someone could probably do a more thorough proof with more time.

HTH


Gini coefficient vs Gini impurity - decision trees

No, despite their names they are not equivalent or even that similar.

Gini impurity is a measure of misclassification, which applies in a multiclass classifier context.
Gini coefficient applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class.

Both could be applied in some cases, but they are different measures for different things. Impurity is what is commonly used in decision trees.


how to save deep learning model and test it after training?

You can find the details in this tutorial: Save CNN model

To summarize:

Tensorflow variables are only alive inside a session. So, you have to save the model inside a session by calling save method on saver object.

import tensorflow as tf
saver = tf.train.Saver()
sess = tf.Session()
sess.run(tf.global_variables_initializer())
saver.save(sess, 'my_test_model')

For saving the model after 1000 iterations, call save by passing the step count:

saver.save(sess, 'my_test_model',global_step=1000)

To use  pre-trained model for fine-tuning:

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('my-model-1000.meta')
    saver.restore(sess,tf.train.latest_checkpoint('./'))
    print(sess.run('w1:0'))
##Model has been restored. Above statement will print the saved value of w1.

To add more operations to the graph by adding more layers and then train it.

sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))

# Now, let's access and create placeholders variables and
# create feed-dict to feed new data

graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

#Add more to the current graph
add_on_op = tf.multiply(op_to_restore,2)

print sess.run(add_on_op,feed_dict)
#This will print 120.


Why am a getting wrong prediction when combining two list of samples, which individually gives correct prediction?

You have a mistake somewhere in either the data you provide as training data or the model you use for predicting.

If it was really the same model trained on the full data in all the cases, then any given instance in set2 would always be predicted with the same class (right or wrong), independently from the other instances in the test set. This implies that it's not the same model being applied when you predict individual sets or both merged.

As a test, save your model somewhere first then apply it to your different cases (the model that you saved, don't train it again).

Ok so you are trying to do 3 experiments:

Training on full training set produces modelA, then apply modelA on set1
Training on full training set produces modelB, then apply modelB on set2
Training on full training set produces modelC, then apply modelC on both set1 and set2

But if this was really what you are doing the three models would be the same: modelA = modelB = modelC. Basically you could do this instead:

Training on full training set produces modelD
apply modelD on any set

Based on your description what probably happens is something like this:

Training on instances of class A (error) produces modelA, then apply modelA on set1 -> perfect results
Training on instances of class B (error) produces modelB, then apply modelB on set2 -> perfect results
Training on instances of class A (error) produces modelC = modelA, then apply modelC on both set1 and set2: perfect for class A, completely wrong for class B

You need to check which instances you give as training set in each case.


How To Merge Features in the Dataset Forest Cover Type Classification Problem?

On Some Analysis I found out the columns were mutually Exclusive and therefore created one column named Soil_Type(0..40 possible values) replacing 40 binary columns and Wilderness_Area(0..4 possible values) replacing 4 binary columns. This did improve performance of the classifiers (Random Forest, Decision Tree ) with respect to time and prediction accuracy as well.


Hill Climbing Algorithm - Optimum Step Size

Usually it worked for me that if the search space was know then annealing rate (divide the size size with number of iteration)helped to decrease/increase the step size gradually to get to local max/min but the draw back is it might get stuck in local and might need some "momentum" to go on, another draw back it it might be very slow.however it doesn't seems like the case with your question that you bother about it.

Please refer-
https://courses.cs.washington.edu/courses/csep573/11wi/lectures/04-lsearch.pdf

This paper proposes self adaptive step size search-
https://link.springer.com/chapter/10.1007%2F3-540-34783-6_56


Help improving my "read_excel" execution time in python. My code reads slowly

Well yes it would be slow because you are opening and closing the file for every iteration of the for loop. A general rule in programming is that if the file is not constantly changing, then only open and read it a single time. Also, there are large sections of your code that can be shaved off if you simply use list comprehension

Here, I have rewritten your code to only open the file and read it once, then it creates the two lists using list comprehension and slicing.

from pandas import read_excel

# excel relevant info
my_sheet = 'Junaid'
file_name = '../Documents/Junaid1.xlsx'

df = read_excel(file_name, sheet_name = my_sheet, index_col=0, header=None)

training_input = [df.index[i:i+5].tolist() for i in range(len(df)-5)]
training_output = [df.index[i].tolist() for i in range(5, len(df))]

Also, there seems to be a bug in your code since the excel file you described in your question does not have a header (i.e. the very first row contains data), thus your code skips the very first row of values. To fix that you should pass the parameter "header=None" to the pandas function to tell it that there is no header index. You can read more about that here.


How to deal with situation where LSTM fails to learn (constantly makes the same incorrect prediction)

Try using a recurrent neural network and use a RProp minus trainer without weight backtracking (aka RProp-, cf. [Igel&Huesken, Neurocomputing 50, 2003]) and without ponderation, ie. all training samples have the same weight.


Here's a few fragments of information relevant to your question.

For generalisability testing, it is most common to split the dataset on which you aim to apply a Machine Learning model into a training, validation and testing set. K-Fold cross validation is also common: splitting the dataset into training and validation sets multiple times, randomly, to ensure a good performance on the validation set isn't just pure luck.
Likewise, to ensure your algorithm is not biased and fits alleged performances, there is no better way than to benchmark it on test datasets that the model has not seen during training time. Some deep learning frameworks come with tools that generate reports regarding your model's performance, such as Tensorflow's Tensorboard.
To find out what attributes influence an algorithm, there are various approaches you can take. Interpretability tools such as LIME will help you see what characteristics of the data you trained your model on affect your model's predictions most.

This article on interpretability continues the discussion, goes more in-depth, presents a few frameworks as well as a tutorial you can follow.


I'd go out on a limb to say this is a difficult problem without incorporating other data. Models just try to explain future action with past information. Traders act differently in different scenarios.

Without data on their analysis or financial data on the stocks they are analyzing, a model on the previous transaction history of traders would not be effective. Unless, if the "traders" you describe are more like etf fund managers that operate on a predefined set of rules.


Multi-Source Time Series Data Prediction

I'm only aware (and using) a RNN which gets multiple time-series in it's first layer and then mixes those in the following layers. Let me know if you have a question on this approach.


Convergence of vanilla or natural policy gradients (e.g. REINFORCE)

What is the proof of this? Can someone point me to a reference?

It is called "The Policy Gradient Theorem", and a good reference would be Sutton & Barto Reinforcement Learning: An Introduction In the second edition, the theory behind REINFORCE and Actor-Critic is examined in Chapter 13.

In brief, the proof is focused on showing that a sample of a particular expression based on reward seen and the parameters of the policy function, is a sample of the gradient of the policy functions with respect to the parameters. It thus follows that a step in the direction of this sampled value, will in expectation be a step in the direction that increases expected sum of discounted reward (or the average expected reward) associated with that policy.

  What is the best convergence rate out there that's known (if any are proven)?

This is not something that can be proven analytically, except in situations where you already know the loss function expressed in terms of the policy parameters (and nothing else that depends on them). Even for simple toy environments this is not really possible. The usual approach is to compare different algorithms experimentally, plotting learning curves. There is a lot of variance in practice, so learning curves are often plotted as averages of multiple training scenarios.

As far as I know, there is no single clear winner in a general sense between different policy gradient methods. However, Actor-Critic and similar methods that combine TD learning with policy gradients are usually preferred over the basic approach of REINFORCE, as the agents that estimate a value function allow "bootstrapped" updates to be made on every step. Although this adds initial bias (as value function estimator parameters initially have no relation to the true value function), it reduces variance and allows for more updates, thus often faster convergence.

  Is there any formulation that works for the mean reward of a rollout?

The base theory of policy gradients applies to episodic, continuous discounted, and continuous average reward formulations with minor changes in the proof.

The algorithm REINFORCE cannot be applied to a continuous environment, as updates are made only at the end of episodes. Actor-Critic approaches can be adapted to average reward formulation - for examples see this survey.


Standardization After PCA for Kmean clustering

Usually PCA already returns standardized components.

Did you compute the variance of each component? Usually, it will be 1.

The more tricky question is whether to use standardization before doing PCA. I don't think there is a general answer for that.


Keras SimpleRNN is a fully connected RNN so indeed each unit connected with all other units. So the equation becomes:

(input_feature +1) x units + units x units
2x50 +2500
+1 comes from the bias


Short answer - No, CNNs are not really translation invariant.

I specifically mean the style of image classification network in the paper you mentioned (i.e., (input) > (conv layer) > ...  > (conv layer) > (fully conn layer) ... > (fully conn layer) > (output)).

This is partly because of the difference between translation invariance and translation equivariance (an important distinction, imo). See this question and the answers

Why

The last few layers in the network are fully connected (FC).  FC layers are definitely not translation invariant (they don't give consideration to the spatial relationship between the inputs).  But, the whole network could still be translation invariant if everything before the first FC layer is translation invariant.

We have convolution layers before the FC layers, but convolution layers are translation equivariant, not translation invariant (see below, it'll make sense).  Therefore the whole network is not (really) translation invariant.

Adding pooling layers after convolution makes the network invariant to small translation motions.

Convolution operations are translation equivariant

Translation equivariance of a function means that it's output for a translated version of the input is a translated version of the output.

if $$f(x(i)) = y(i)$$
then $$f(x(i-t)) = y(i-t)$$
where $i$ is the spatial index

Compare to translation invariance

Translation invariance means that the output of a translated version of the intput, is exactly the same as the output for the original input.

if $$f(x(i)) = y(i)$$
then $$f(x(i-t)) = y(i)$$
where $i$ is the spatial index


Given that the data model you receive is different, it makes best sense for you to use a NoSql store like mongodb.

Here is the tech stack I would use ( python ) :

To receive requests with csv file, I would use flask microweb framework.
I would run flask with gunicorn and gevent.
For every csv file received, I would parse the rows based on varied columns and store them in a mongodb.
I would write queries on the collection in mongodb to extract information.

I would be more than happy to elaborate more if required .


You are in a regression setting, where accuracy is meaningless (it is meaningful only in classification settings).

Simply remove the metrics=['accuracy'] argument from your model compilation, so that model.evaluate returns the loss only (discard also val_acc).

See the SO thread What function defines accuracy in Keras when the loss is mean squared error (MSE)? for more details.


What model to fit pseudo-logarithmic trend?

One option is to take a curve fitting approach. For example, scipy.optimize.curve_fit.
Something like this:
import numpy as np
from scipy.optimize import curve_fit

def f(x, a, b, c):
    "Rough functional form of the data"
    return a + b * (1 - np.exp(-c/x))

solution = curve_fit(f, c, y, p0=(3_500, 3_500, 200))


One multilabel classifier or one for each type of label?

Both ways are valid and both are commonly used. Sometimes, a classifier that claims to be multilabel may just be separating the labels into multiple OneVsRest classifiers under-the-hood and conveniently joining the results together at the end.

However, there are cases where the methods are fundamentally different. For instance, in training a neural net with multiple targets (labels), you can setup the structure of the network such that there is shared structure. The shared nodes will end up learning features that are useful for all the targets, which could be very useful.

For example, if you're classes (labels) are "cat-pet", "cat-big", and "dog", you may want an algorithm that first learns to distinguish between any cat and any dog, and then in a later step learns to separate cats that are pets from cats that are big (like a lion!). This is called hierarchy, and if your classifier can exploit hierarchy you may gain better accuracy. If your classes are completely independent however, it may not make any difference.

I suggest you start with the method that is easiest (i.e. OneVsRest), and see if the performance is suitable to your needs, then move to more complicated methods (multilabel, hierarchical methods, etc) only once you need better performance.


Finding the inverse of a matrix with neural networks

In general this problem falls under the umbrella of "structured prediction" since you are trying to estimate a number of things that are related by virtue of being embedded in a PSD matrix.

Instead of estimating the inverse in one swoop, I'd pick an appropriate algorithm, say inversion by eigendecomposition, estimate the components (eigenvalues and eigenvectors), then piece them together. Here's one paper that shows how: Neural networks based approach for computing eigenvectors and eigenvalues of symmetric matrix.

Alternatively, you can investigate approximation algorithms such as the one elaborated in Approximating the inverse of a symmetric positive definite matrix

Welcome to DataScience.SE


Historical weather data with machine learning?

For time-series modelling, one would typically look to LSTMs (or their more recent improvements). Such models are known for their long-time-memory, which is necessary here because of long term seasonality and similar effects.

As for production, there are several methods to serve neural nets, for example Tensorflow Serve.

Specifically about training once in production - you could choose to simply not do it. If your dataset has 10 years of data, and your model will be in production for more than a year, you could think about retraining it (with all the newly collected data + old) periodically, every year or so. It could be completely automated too.

There are other methods (look for "online training and inference" approaches, which are traditionally in the RL domain). But based upon your description, the extra time you would spend to build such capability doesn't seem to be worth the gain.


Does ensemble (bagging, boosting, stacking, etc) always at least increase performance?

The short answer is no.

I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves.  In some cases the precision and recall was better with the ensemble, but more often, it was not.  That's not to say it's not worth investigating.  But sometimes, there is one model that does a reasonable job of classifying data, but it can get drowned out in an ensemble.  Perhaps a weighted ensemble might improve the results, but its not a clear cut approach to improving the performance.

In practice, I would try several models and then try an ensemble of the models.  If the ensemble is the best, however you define best, go with it.  But sometimes it is easier to just pick the best base model and then figure out how to tune that model.


Indeed you can introduce some "unvariant" features to your LSTM network using Conditional RNN that use these features to create the initial hidden state:
https://github.com/philipperemy/cond_rnn
I hope this is what you are looking for.


For linear regression you want to use R's lm() function, like this:

my.model <- lm(response.variable ~ predictor1 + predictor2, data = my.data)

Look at the model using:

summary(my.model)

You can apply this model to a "test" dataset (your 20% split) by using predict(), like this:

predict(my.model, test.data)


How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm?

Hi David and welcome to the community! I think that the [0] is for accessing the array of actions as it is in double brackets. After this you need the action index to update appropriately.

The Q network would have as many outputs as the actions available (in your case 2). Then you want to update the weights of the network in the output layer that is responsible for estimating the Q that corresponds to the action selected.  So it seems that you have received a $r(s,action)$ and thus you assign it as target in order to use it for the MSE($Q(s,action)$, $r(s,action)$) between estimation and real reward.

The weights of the other outputs cannot be updated by using the collected $r(s,action)$ as the Q-learning update equation would be wrong so the target remains the same as the prediction (resulting a MSE=0). Think that each of the network's outputs is responsible for estimating the return by performing a particular action, at a particular state. For example the first output is responsible for estimating the return (given input state) for action=left ($Q(s,left)$) and the other for action=right ($Q(s,right)$). You train each network head (output layer) with MSE between real reward and estimation. Each time the reward that is sampled from the environment would be the result of a selected action. Thus you update only the corresponding head by assigning as target the reward sample.


Fastest batch perspective transform for image augmentation

Image augmentations heavily relies on your DataGenerator and DataLoader design, mostly along with the hardware resources that you are using. Apart from that, here is a quick comparison chart to help you with the transformations along with the libraries links.
The numbers represent the number of images it processes per second.

Here are the links to all the required libraries:

Albumentations
Imgaug
Augmentor
SOLT

Ref: Albumentations benchmark results


Making sense of loss and accuracy curves

Just a couple of points below

Red box: In strict terms, overfitting takes place when the model is doing better in the training than the validation set. And so, I would say here it starts earlier than what is denoted in red and it probably does relate to the model finding better parameters that only apply to the training set. Of note, no significant train loss decrease.
Green box: In training, significant drop in the loss. This discovery of a significantly better minima means it may also apply in the validation set and it is actually reflected in the val loss curve. Despite that, the loss is no better than the checkpoint at the beginning of the red box.
Blue box: Loss and accuracy are not co-dependent metrics. Whereas accuracy is a measure of the number of correct guesses (how many), the loss is a measure of distance from correct given a data point (how far off).

Generally, smaller LR means the model has less "freedom" to hop very far around in the feature space. And so there are fewer chances to see significant and fast loss decrease. As you suggest, there is a slight upward overfitting trend in your val loss but is as significant or fast as the smaller LR allows in training.
Also, note that your train loss is higher than val loss for a good amount of your training (~55th epoch). You may want to investigate how your model is regularised as this may affect your learning curves in ways that may help you infer diagnostics better. E.g. in PyTorch, train loss between model states (model.train() and model.eval()) differ significantly.


Reinforcement Learning on data only (NO emulators)

Q-learning does not in fact need to be online or need an emulator, it can learn exclusively from experience replay. If you put all your history into a table or state, action, reward, next state and then sample from it, it should be possible to train your agent that way.
To do this, you will need to skip the algorithm steps that take actions and store results. The algorithm will then learn from the data you have. It will just not be possible to collect more. Depending on the problem you are trying to solve, this could be OK, or it may inhibit learning.
RL algorithms learning optimal control in complex environments benefit from sampling near to their current policy, so it is possible in your case that your agent will reach a limit on what it can learn from historic data. It may end up still quite far from optimal behaviour, although it should stand a reasonable chance of improving on the best that the historic data shows.
If you need to use function approximation (e.g. a neural network) due to size of the state, action space, then take extra care, because it will be hard to detect whether the action values have converged correctly. This is because you are learning the optimal Q values, and you will have no test data that demonstrates what those should be (to collect that data, you need to follow the optimal policy and measure the total reward).
Here is roughly what an experience-replay-only Q-learning algorithm would look like:

Input: History $H$, consisting of rows of $S,A,R,S'$
Initialise the NN for calculating $\hat{q}(s,a)$
Repeat until NN converges:
$\qquad$ Sample $S,A,R,S'$ from $H$
$\qquad$ $tdtarget \leftarrow R + \gamma \text{max}_{a'}[\hat{q}(S',a')]$
$\qquad$ Train NN single step, $\hat{q}(S,A) \rightarrow tdtarget$

You can make use of mini-batch processing to generate multiple $tdtarget$ values at once and train on them. A worthwhile improvement for stability is to use a frozen copy of the neural network when calculating the $tdtarget$ value, and update it only every N steps to be a copy of most recent network, with N maybe 1000 steps.


Problem with converting string to dummy variables

You can use label encoders for your product and market. however one hot encoding will be the best one to suit for converting string to numbers to feed into an algorithm  . I suggest looking at this link for further details :- https://towardsdatascience.com/choosing-the-right-encoding-method-label-vs-onehot-encoder-a4434493149b


Where is the Backward function defined in PyTorch?

Backward function is same for all type of layers.
Look at 155 number line here.
https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/tensor.py
and it will forward you to here
https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/autograd/__init__.py


Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all it's work in memory, so you are basically limited to the amount of RAM you have available to you.

A mature project for R and Hadoop is RHadoop

RHadoop has been divided into several sub-projects, rhdfs, rhbase, rmr2, plyrmr, and quickcheck (wiki).


Best way to implement 10-fold cross validation for CNN model

In my point of view, it should be Option-I
 Primary logic is to test your Model on unseen data.
Reason for that is,
When you executed your K-Fold and let's say got a score ~87% and then you tweaked your Hyperparameters.
In this way, we actually leak the test data(K-Fold test set) information to the learning process, and eventually, the process will overfit to the test data too if this hit-try happens multiple times.
So, the Option-I score should be closer to the score on future new data.


How do I prepare data in which each output row depends on multiple input rows?

A simple, practical approach would be to aggregate your data on each customer. The idea is that the repartition of credit usage / credit limit might not really matter for the overall bankruptcy. You might then want to build new features to avoid loss of information : the number of maxed out credit card, average interest on credit card. This is the general idea :
Table 1: Customer credit usage
customerId | nb_cc  | maxed-out_cc| Overal Credit limit| Overal Credit used | bankruptcy
1            2        1             14                   13                   1
2            1        0             10                   1                    0

The main alternative would be to try to guess on which credit card clients will default. Depending on the law and procedure you might want to build a target at credit card level (when someone goes bankrupt, do you lose the overall credit used or simply the credit used on the cc he failed to reimburse ?). Contrary to bankruptcy this is not a well defined (ie law defined and enforced target). This would be the general idea :
Table 2: Credit cards
cc_id | Limit | Used | nb_other_cc | other_limit | other_used |  bankruptcy (Y1) | cc_default (Y2)
1-1     5       5      1             9             8             1                 1
1-2     9       8      1             5             5             1                 0
2-1     10      1      0             0             0             0                 0

In general I don't think using Y1 here would really be beneficial compared to using it in the first approach. (It might even add some modelling problem as it would overweigth people with multiple credit cards)
Defining and mesuring Y2 might be difficult overall and it might require further modelling to aggregate predictions at a client level (if #1 default on cc #1 what does that mean for cc #2 ?).
Note : as you didn't mention it in your initial question I didn't deal with it, but generally speaking you'll have to deal with a time horizon. Basically you might want to have multiple observations over the life of your products (say each year) and a time horizon for your prediction (say bankruptcy over the next year). Even if that introduce some modelling problems it might help you have better business oriented metrics (how much will we lose next year ?).


xgboost GridSearchCV take too long or does not goes to the next step

Based on the combinations of learning parameters, learning rate(2), max_depth(2) and n_estimators(2), it seems the algorithm is exactly doing what it's supposed to do. with cross validation set to 5 it's performing 40 fits (2*2*2*5).
Can you add some more details to clarify like what configuration you are using for doing this task?


How is the E(X) of a Poisson distribution lambda?

The probability mass function for a Poisson-distributed random variable $X$ is $f_X(x)=\dfrac{\lambda^xe^{-\lambda}}{x!}$. The expected value is a sum over all possible values of $x$, the natural numbers (non-negative integers).

$$\mathbb{E}[X]=\sum_{x=0}^{\infty} x f_X(x)$$

When you evaluate that sum, the result is $\lambda$.

Yes, $\lambda$ often means a wavelength in physics. That’s just notation, however, and easily could be $\xi$ or $\beta$ or $\gamma$. Your physics knowledge will also tell you that $\theta$ is an angle, yet statisticians often use $\theta$ as a parameter. For the Poisson distribution, $\lambda$ denotes a rate, and this is how you get five cars per hour as the average rate of cars passing through a remote intersection (or whatever they’re doing).

EDIT

I like JBStatistics. He derives the expected value of a Poisson distribution, if you’re interested in seeing how that series calculation works: https://youtube.com/watch?v=65n_v92JZeE.


While downsampling training data should we also downsample the validation data or retain validation split as it is?

I would recommend not to downsample the validation set. In the end you care about performance on the test set with the skewed class distribution. Therefore your validation set (used for hyperparameter selection, early stopping etc.) should have the same distribution in my opinion.

Have you considered upsampling the minority class? By downsampling you loose training data, which might contain valuable information and therefore might harm the learning process.


How can I detect if an image was photoshopped?

Error Level Analysis as described Error Level Analysis, found at https://github.com/afsalashyana/FakeImageDetection seems to be one seems to be one way:

You exploit that local compression ratios might be different. And it seems to be possible to train neural networks on it.

I didn't find a paper which says how well this works so far


Need some help on data mining platform recommendation

As @oW_ mentioned, Orange seems to be a good choice.

Data Source

Orange3-spark provides Hive/Spark support, cool!

Interface

A little laggy, but acceptable.

Algorithms

Full algorithm supports (based on scikit-learn). I can implement deep learning stuff myself.

Customization

Highly supported. Widgets & addons can be developed.

Visualization

Not bad.


How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

As far i know u can't saything about hidden class, hidden class value at time t is 'some intermediate values of weighted values of all hidden classes'.

see the point-  2) hidden state sequence in blog-
https://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html

so ur statement-

"Hence let in my HMM, the hidden state Zi denote the true class of the i-th sample Ii, and Xi be the predicted class by C- is incorrect.

How can u compare hidden class value with actual, u could have compared emission value and actual.

You can try ensembling of HMM and Cs. I wonder how exactly you trying to use parameters from classification problem to time series/sequence (HMM).


What is the purpose of setting an initial weight on deep learning model?

This is greatly addressed in the Stanford CS class CS231n:

  Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

There are several weight initialization strategies; each one is best suited for a type of activation function. For instance, Glorot's initialization aims at not saturating sigmoid activations, while He's initialization is meant for Rectified Linear Units (ReLUs).


How much data are sufficient to train my machine learning model?

The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data.

A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below:

The learning curve represents the evolution of the training and test errors as you increase the size of your training set.

The training error increases as you increase the size of your dataset, because it becomes harder to fit a model that accounts for the increasing complexity/variability of your training set.
The test error decreases as you increase the size of your dataset, because the model is able to generalise better from a higher amount of information.

As you can see on the rightmost part of the plot, the two lines in the plot tend to reach and asymptote. Therefore, you eventually will reach a point in which increasing the size of your dataset will not have an impact on your trained model.

The distance between the test error and training error asymptotes is a representation of your model's overfitting. But more importantly, this plot is saying whether you need more data. Basically, if you represent test and training error for increasing larger subsets of your training data, and the lines do not seem to be reaching an asymptote, you should keep collecting more data.


Adding recommendations to the output of a classification model

Decision tress can be plotted in python, this is the most visualizing machine learning algorithm, you can see a link here: https://medium.com/@rnbrown/creating-and-visualizing-decision-trees-with-python-f8e8fa394176.

For everything else you need to do data exploration again. Try to visualize in bar plots, for example you can show, if you have class 'X' and 'Y', what your confidence intervall of your future 1 if it is class X and what if it is class 'Y'.


It's a quite complex problem and there might be better options for the design (I'm thinking maybe something more specific to times series)...

However before that there's a more obvious problem to solve: it seems that you are calculating accuracy on the "Glucose_t" numeric value, right? If yes this is incorrect and that would explain your terrible results:

Accuracy is an evaluation measure for classification tasks, not for regression tasks: accuracy simply checks for each instance whether the predicted value (which is supposed to be a categorical value) is the same as the gold true value, and then divides the number of correct cases by the total. Naturally it's very hard to predict the exact true value in the case of numbers: if the true value is 183 but the algorithm predicts 184, then accuracy would count that as incorrect even though it's very close. That would explain why your accuracy is very low.
Typical regression evaluation measures are mean absolute error and mean squared errors. These measures (and their variants) are designed to calculate how far the predicted numerical value is from the true value. If you use these keep in mind that it's an error score, so the lower the value the better.
If you want to use accuracy, it would make sense to use it on your "event" value (which is derived from your "glucose_t" prediction I assume?): in this case you have 3 classes (categorical values), and I bet the accuracy will be much better. Note that accuracy can be biased by class imbalance, so don't forget to check the confusion matrix.


Confidence intervals in multivariate linear regression

Looking at https://en.wikipedia.org/wiki/Simple_linear_regression :

This t-value has a Student's t-distribution with $n-2$ degrees of freedom. Using it we can construct a confidence interval for $\beta$:

$$ \beta \in \left[\widehat\beta - s_{\widehat\beta} t^*_{n - 2},\ \widehat\beta + s_{\widehat\beta} t^*_{n - 2}\right] $$

at confidence level $1-\gamma$, where $t^*_{n - 2}$ is the $(1-\frac{\gamma}{2})$-th quantile of the $t_{n−2}$ distribution.


How different should discriminator be from generator in GAN

Generator cant learn if the discriminator error is too small. But generator should always be "ahead" of generator in order to learn.

There is a paper explaining why the generator's gradient vanishes if the discriminator gets too strong. And how does this proportion look mathemactically optimal

TL;DR Dont make it too strong, but make sure D is ahead to ensure optimal learning. GANS are notoriously unstable (expensive also)


Is there a C library for machine learning algorithms?

Here's a detailed table about different machine learning libraries on different languages:

https://github.com/josephmisiti/awesome-machine-learning

Checkout C version here and C++ version here.

Personally speaking, try OpenCV ! OpenCV provides multiple machine learning implementations including KMeans, kNN, SVM, etc.


Isn't computing the "tractable error" in Restricted Boltzmann Machines (RBM) intractable?

Yes, calculating $\sum_{h} e^{-E(v, h)}$ does include summing over $2^N$ possible configurations of the hidden vector $h$. However you don't need to actually calculate it:

If we have the negative log likelihood function
$$ \mathcal{L}(W, c, b) = - \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right) $$
where $p(v^{(t)}) = \frac 1Z \sum_h e^{-E(v^{(t)},h)}$,
we don't need to calculate $\mathcal{L}(W, c, b)$ directly, instead we want to minimize it using gradient descent. Using $\theta$ as the full parameter vector instead of $W, c, b$ for simplicity,  we get:
$$\begin{array}{ll}
    \nabla\mathcal{L}(\theta)
    &\displaystyle = \frac{\partial}{\partial\theta} \left(- \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right)\right) \\
    &\displaystyle= \frac 1T \sum_{t=1}^T \left(
        \sum_h p(h \mid v^{(t)}) \frac{\partial E(v^{(t)},h)}{\partial\theta} - \sum_h\sum_vp(v,h)\frac{\partial E(v,h)}{\partial\theta}
    \right)
\end{array}$$
The first term of the difference above is the expected value of $\frac{\partial E(v^{(t)},h)}{\partial\theta}$, given a specific visible vector (i.e. a training example) $v^{(t)}$. This can easily be calculated for each parameter $W, c, b$.

The second term is the expected value of the energy, calculated over the full distribution of the RBM. Unfortunately, this is again intractable. Here, the contrastive divergence algorithm by Hinton comes in useful: we run Gibbs sampling for a number of steps and estimate the second term using the resulting states $\tilde{v}$.

tl;dr: yes, the problem is intractable, which is why we use the contrastive divergence algorithm by Hinton, which is only an approximation but works well in practice.


Counting of events is probably best done as a separate software component that takes the output of your event classifier. The event classifier model would output for each time-step two probabilities: one for started and one for stopped.

The easiest way to count the events to have a state machine that compares the probabilities against a fixed threshold to create discrete events. And once you have discrete events, counting is trivial. Pseudo-code below:

# returns a tuple with (new_state, event_name)
# event_name can be None if no event occurred
def next_event(state, start_prob, stop_prob):
    start_threshold = 0.5
    stop_threshold = 0.5

    if state == 'running':
        if stop_prob > stop_threshold:
           return ("parked", "stop")

    if state == 'stopped' or state == 'unknown':
        if start_prob > start_threshold:
           return ("running", "start")

    return (state, None) # no change

...
# initialization
start_counts = 0
state = 'unknown'

...
# for every new time window X
   event_prob = model.predict(X)

   state, event = next_event(state, event_prob[0], event_prob[1])
   if event is not None and event == 'start':
        start_counts += 1
   ....


How is loss computed for multiclass CNN with an output layer larger than the number of classes?

There are many great tutorials covering how exactly cross entropy loss works. The key thing is : the classifier outputs probability for each class and not a single label.
Taking your example, if there are 3 classes then the network will output something like [0.5, 0.25, 0.25] giving probabilities for each class. You can treat expected output as [1, 0, 0] if the output is, say, label 0. Now the dimensions agree and you can calculate the loss.


Tensorflow RNN not learning when output included in training variables

Strangely, once the columns of each block of 5 data entries were normalized, with null rows included, the accuracy jumped to >99%. Again the outcome is included in the data set but this seemed to correct the issue with this sanity check.


Momentum in neural networks is a variant of the stochastic gradient descent. It replaces the gradient with a momentum which is an aggregate of gradients as very well explained here.
It is also the common name given to the momentum factor, as in your case.
Maths
The momentum factor is a coefficient that is applied to an extra term in the weights update:

Note: image from visual studio magazine post
Advantages
Beside others, momentum is known to speed up learning and to help not getting stuck in local minima.
Intuition behind
As it is really nicely explained in this quora post, the momentum comes from physics:

Momentum is a physical property that enables a particular object with
mass to continue in it's trajectory even when an external opposing
force is applied, this means overshoot. For example, one speeds up a
car and then suddenly hits the brakes, the car will skid and stop
after a short distance overshooting the mark on the ground.
The same
concept applies to neural networks, during training the update
direction tends to resist change when momentum is added to the update
scheme. When the neural net approaches a shallow local minimum it's
like applying brakes but not sufficient to instantly affect the update
direction and magnitude. Hence the neural nets trained this way will
overshoot past smaller local minima points and only stop in a deeper
global minimum.
Thus momentum in neural nets helps them get out of
local minima points so that a more important global minimum is found.
Too much of momentum may create issues as well as systems that are not
stable may create oscillations that grow in magnitude, in such cases
one needs to add decay terms and so on. It's just physics applied to
neural net training or numerical optimizations.

In video
This video shows a backpropagation for different momentum values.
Other interesting posts
How does the momentum term for backpropagation algorithm work?
Hope it helps.


Grouping profiles strings having the same words, but occurring out of order

"fuzzy match" is a vague term for many different kind of string similarity measures. There are two main categories:

Sequence-based methods, such as Levenshtein edit distance (and variants), Jaro-Winkler
Bag-of-something methods, such as Cosine-TFIDF, Overlap, Jaccard, ...

Traditionally the first category is used at the level of characters (e.g. to match "abcdefg" with "abdfeg") and the second at the level of words (e.g. to match "the management of the team" with "the team management"), however there is no technical restrictions to keep it this way. The main difference between the two categories is whether or not they take the order of the units (e.g. words or characters) into account. In fact there are also some hybrid measures such as SoftTFIDF which combine both approaches.
Conclusion:

For the problem of word order, a simple bag-of-words measure such as cosine or Jaccard would perfectly match two profiles.
If more flexibility is needed, Soft-TFIDF or other variants would work.

The last part of the question is about selecting a standard string. In general there's no way to know what should be the standard string from the different variants themselves. A simple approach would be to pick the most common variant.


Is it possible for a DCGAN to do regression? What are some examples of this?

The answer is yes.  There is a paper Intra-class Variation Isolation in Conditional GANs (https://arxiv.org/pdf/1811.11296.pdf) discussing it.

They propose a "C-GAN that is able to learn realistic models with continuous, semantically meaningful input parameters".  They actually cover generating images of people at different ages as well.


It seems "stovepiping" is a term that legendary author on the subject of data warehousing Ralph Kimball adopted, and it looks that it actually originates from intelligence community. In its original (intelligence community) meaning, it is mostly related to the usage of raw data, instead of processed data.

From Wikipedia:

  Stovepiping (also stove piping) is a metaphorical term which recalls a stovepipe's function as an isolated vertical conduit, and has been used, in the context of intelligence, to describe several ways in which raw intelligence information may be presented without proper context. It is a system created to solve a specific problem. The lack of context may be due to the specialized nature, or security requirements, of a particular intelligence collection technology. It also has limited focus and data within is not easily shared.

From RationalWiki:

  Stovepiping is a term originating in the intelligence community to describe a process by which raw data is funneled directly to high-ranking officials or the media.

Also, there is an interesting article in PhraseFinder.

Even better explanation from Relational Solutions Blog:

  Most data marts are designed as one off reporting solutions, designed to solve an immediate problem that the business users need to solve. When designed as a stand-alone, they are often referred to as “stove pipes” or “silos” of information.

  Analysts new to the space think these are new terms, but Data warehousing consultants have used these terms since the 90’s. They’re used to describe stand-alone reporting solutions. Typically these stand-alone solutions are developed by individual teams or departments.

  These groups develop “silo’s” or “stove pipe” reporting databases to achieve a specific goal that they were unable to get financial approval for. If they have a need for something that you can't get approval for, you resort to building something on your own. It happens in every company and every department.


How to automate left join of multiple data frames with single data frame one by one in R

If your data resides in clean form inside a database already:

write an sql statement that includes a series of UNION statements for your 206 country tables. You can use R or any other programming language to create this string.

save this series of 206 UNIONs as a view (maybe as an indexed view / materialized view), or as  a new table, or as a stored-procedure that creates a temp-table on your database server

perform one SQL join on the DB server

fetch the data batch-wise


What do the hidden layers of a neural network try to learn?

Each hidden layer represents a nonlinear application of a function on the inputs of the previous layer.

The first layers perform a function on our data-inputs and the last layer is responsible for either using softmax for classification purposes or be an affine layer for regression.

The in-between layers are necessary in order to stack complexity and therefore avoid having to create a layer of very high complexity.

In fact, in theory, two level deep neural network can represent any function. But how complex would this function need to be?
A layer of a very high complexity would be able to map our raw inputs into our final classification or regression task.

However what would be simpler to do is to let the intermediate hidden layers learn useful features regarding the given task and then the next hidden layer could exploit these features in its own way. And then the next hidden layer could use these newly transformed features in its own way and so on. So stacking nonlinear functions one on top of the other is more clever than requiring a single nonlinear function to do all the hard work.

There is a limit on how many layers you can stack together for the main reason that these hidden layers get trained by backpropagation which uses the chain rule of derivation in order to pass the deltas of the error back to the weights of the layers and tune them but a long series of these operations could end-up with exploding or vanishing gradients which means really poor performance or a model that cannot be trained.


What is Coarse-to-Fine in the context of neural networks?

"Coarse to Fine" usually refers to the hyperparameter optimization of a neural network during which you would like to try out different combinations of the hyperparameters and evaluate the performance of the network.

However, due to the large number of parameters AND the big range of their values, it is almost impossible to check all the available combinations. For that reason, you usually discretize the available value range of each parameter into a "coarse" grid of values (i.e. val = 5,6,7,8,9) to estimate the effect of increasing or decreasing the value of that parameter. After selecting the value that seems most promising/meaningful (i.e. val = 6), you perform a "finer" search around it (i.e. val = 5.8, 5.9, 6.0, 6.1, 6.2) to optimize even further.


SVM using scikit learn runs endlessly and never completes execution

Kernelized SVMs require the computation of a distance function between each point in the dataset, which is the dominating cost of $\mathcal{O}(n_\text{features} \times n_\text{observations}^2)$. The storage of the distances is a burden on memory, so they're recomputed on the fly. Thankfully, only the points nearest the decision boundary are needed most of the time. Frequently computed distances are stored in a cache. If the cache is getting thrashed then the running time blows up to $\mathcal{O}(n_\text{features} \times n_\text{observations}^3)$.

You can increase this cache by invoking SVR as

model = SVR(cache_size=7000)

In general, this is not going to work. But all is not lost. You can subsample the data and use the rest as a validation set, or you can pick a different model. Above the 200,000 observation range, it's wise to choose linear learners.

Kernel SVM can be approximated, by approximating the kernel matrix and feeding it to a linear SVM. This allows you to trade off between accuracy and performance in linear time.

A popular means of achieving this is to use 100 or so cluster centers found by kmeans/kmeans++ as the basis of your kernel function. The new derived features are then fed into a linear model. This works very well in practice. Tools like sophia-ml and vowpal wabbit are how Google, Yahoo and Microsoft do this. Input/output becomes the dominating cost for simple linear learners.

In the abundance of data, nonparametric models perform roughly the same for most problems. The exceptions being structured inputs, like text, images, time series, audio.

Further reading

How to implement this.
How to train an ngram neural network with dropout that scales linearly
Kernel Approximations
A formal paper on using kmeans to approximate kernel machines


How to test/train a model for realtime data with new data points and classes in a ML pipeline

In both cases you need to know that the already trained model is not written on the stone. Specially the second question "needs" change of model.

To be more detailed, you are talking about Online Learning in general. Many ML algorithms have online versions. Here, for example, you see the online version of K-means for second question. But in general, online learning is nothing more than adapting a model to the situation in which data comes one sample after the other (like your question). So for basic usage, you can simply adapt your model respectively in a regular manner. this can not be fully online but with a time gap you can retrain the model with new data points which are not assigned to any cluster yet. For fully online-algorithms you can just refer to the link above and start researching.

For the first question I have a meta-question. How do you know that you have a totally new class of texts? So first you need to detect the change and then react correspondingly. To detect the presence of a new class, I propose looking at the logit of the ML algorithm. For instance, if it is a probabilistic approach, you may look at probability of assignment to any present class and say if all less than a threshold and the number of out-of-vocabulary words of new document according to current vocabulary is above a threshold, I create a new class. Not state-of-the-art but for basic usages it will work I assume. Not much to do with preprocessing in a very online manner if it needs a big change. Preprocessing should either be simple and general or very specific according to the class. The first does not need a change and the second needs a bunch of samples from new class to be decided. leave it simple and general for all (like removing strange punctuations, lower-casing, etc.)


The accuracy given by Keras is the training accuracy. This is not a proper measure of the performance of your classifier, as it is not fair to measure accuracy with the data that has been fed to the NN. On the other hand, the test accuracy is a more fair measure of the real performance. As there is a big gap between them, you are overfitting very badly, and you should regularize your model. Consider using dropout or weight decay.


How can you adjust a prediction based on features in the future being different than predicted?

Try these two approaches:

First:

Make a model, using any ML algorithm and divide your data into train and test.
Now using the previous features, check the train and test accuracy.

Now add the new features to the previous ones, again divide data into train and test.  Check the train and test accuracy.

If the new features help improve the test accuracy then only use them otherwise just skip them.

Second:

Add the new features to the previous dataset and set their values as null/0. So while predicting the results before wont be affected by their values. And later when these features do get values, they will help improve the predictions.

I hope this helps!


Can anyone verify my NN diagram if it is properly drawn?

I do not think there is a special kind of format that needs to be followed as long as the image is clear and readable, which (imho) it is for your case. Regarding the last 2/3 layers, the final layer is the output with 1 unit, so you pictured it correctly, along as the article mentions the output shape (that is not a multi-output situation).
Good luck with publication!


Generative Adversarial Text to Image Synthesis

There are several important concepts in this sentence so let's break them down.
"Data manifold"
This is a usual model we use in machine learning problem, that is to consider real data as a manifold.
I suggest to you this article by Christopher Olah on the topic.
"Interpolation between embedding pairs"
One way of generating new data is to sample the embedding space learned by the neural network. For example, you can take two real data samples, compute their embedding, interpolate them to obtain an intermediate embedding, then see the output of your neural network when fed with this intermediate embedding.
"Interpolations between embedding pairs tend to be near the data manifold"
Following the previous step, the output of your neural network should be realistic. In mathematical terms, it should be near the real data manifold.
This is the main point of generative models such as generative adversarial networks or variational autoencoders. They learn to fit some random distribution, usually a gaussian, to the real data distribution, and they learn to "convert" noise to real data and vice versa.
This is often referred to as disentanglement. As explained in the first paper cited by the article after the sentence,

deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation.

In other words, we can obtain deep embeddings that isolate the factors of variation of real data. Ideally, embeddings of human faces can isolate the axis that controls color of the hair, expression of the mouth, etc. (like TL-GAN for example). But these factors of variations are not always as easily explainable.


What is weight and bias in deep learning?

Mathematically speaking, Imagine you are a model (No! not that kind, figure 8 ones).
Bias is simply how biased you are. If you are a Nigerian, and you are asked "Which nationality have the most beautiful women" you might say Nigerian Ladies. We can say it's because you are biased. So your formula is $Y = WX + nigerian$.
So what do you understand? Biased is that pre-assumption in a model like you have.
As for weight, logically speaking, Weight is your Gradient (as in linear algebra).
What is Gradient? it's the steepness of the Linear function.
What makes the linear gradient very steep (High positive value)?
It's because little changes in X(input) causes Large differences in Y axis(output). So you (Not as a Model anymore, but a brilliant Mathematician (your alter ego)) or your Computer tries to find this gradient, which you can call weight. The difference is that you use a pencil and graph book to find this, but the black box does its electronic Magic with registers.
In the Machine Learning Process, computers, or you, try to draw many straight lines or Linear functions across the data points.
Why do you try to draw many straight lines?
Because in your graph book/Computer memory, you are trying the see the line that fit appropriately.
How do I or Computer know the line that fits appropriately?
In my secondary school, I was taught to draw a line across the data points, visually checking the line that cuts through perfectly in the middle of all the data point.(Forget those A.I hype, our brains can calculate by just staring at things). But as for the computer, it tries the standard deviation and variance of each line towards the data points. The line with the least deviation (sometimes we call it the error function) is chosen.
Cool! so and what happens
The gradient of that line is calculated, let's say the Weight of the Learning problem is Calculated.
That's Machine Learning at its basic understand and a high school student plotting graph in his/her Graphbook.


First question - predicting user score:
One approach addressing this problem is using collaborative filtering.
Where the basic idea is 'placing' each entity type(users, restaurants) in a vector space, where a distance function can be utilized.
This way you can learn that if user1 is similar(short distance) to user2 and restaurant1 is similar to restaurant2 then its probable that both users will give similar score to both restaurants.
 One popular algorithm that uses collaborative filtering is the 'Matrix Factorization' - the matrix factorization example is in the bottom of the article.

Second question - handling categorical variables:
Using one-hot-encoding for handling categorical data is a common methodology.
I personally love using pandas.get_dummies function.


What are some resources to test your data science skills?

If you want to move from theory to application then I suggest to do exactly that: get your handy "dirty"!

UCI Machine Learning Repository has some easier datasets to get started. Kaggle is great too but before going for any competition look for an easier dataset from their repository.

If you prefer something with more guidance the book "Introduction to
Machine Learning with Python" by Mueller and Guido contains many examples starting with the classic Iris dataset.

Finally, you could consider an online course. The Udacity Nanodegrees include projects which you need to submit so it gives some basic hands-on experience. Though, in your specific case, I would only go this route if you think it might help your CV.


In this example, you have 2 classes which are separated with a line with negative slope (if you don't know why please comment). So if you set the slope as +1 then you can not separate classes. The whole point is to learn the line and line is defined with its coefficients (including threshold but threshold is JUST ONE OF THEM!).

PS: If you later want to upgrade to multilayer perceptron and use backpropagation algorithm, then you need to choose randomly different initial weights for mathematical reasons according to the backpropagating error of layers


Can I conduct independent t-test when data is infested with outliers ? and how to interpret the t-statistics?

It's fine to do a t-test on unequal sample size, however, the power wouldn't be as good as equal sample size.

1:) Yes or no. Impossible to say without plotting the outliers. What's more important, can you assume your data be normally distributed? Have you checked the QQ-plot? Have you checked the histogram? Do they look like close to a normal distribution? While the t-test is robust against non-normal data as long as the sample size is sufficient large, your data shouldn't behave too far away from a normal.

When you think about outliers, ask yourself the following questions:

How many outliers? If you have many, t-test is probably not appropriate.
Why the outliers? If it's a random error (you're just unlucky), you could include it in the t-test. If it's a systematic error, stop the test, go back and check your data.
How do you define the outliers?
Do those outliers look symmetry? If so, you might assume your sample come from a normal population. You can check the skewness of your data.

You have to try to understand those outliers to come with up a decision.

2:) You can just explain like "the probability of the difference in means is (or isn't) significant".

3:) You should draw a box-plot for each group.


How to set different weights for different training samples?

In scikit-learn, most algorithms (SVM, Decision Trees, SGD, etc.) have a sample_weight argument that you can pass when fitting. In your case, you could provide a different weight based on which of the 3 datasets the data point comes from.
If the algorithm you want to use doesn't provide the sample_weight argument, you can always sample with replacement. Simply put, you give each sample weight, and then you create your dataset by sampling them with replacement. This means that instances with higher weights may appear multiple times in your dataset.


Training CNN: Understanding number data generated while training the model

95 is not number of samples, it equals to testset divided by batch_size: testset // batch_size in other words steps in per epoch


Use serialized model without installing dependencies

You may be able to convert the entire model pipeline to a standardized format. PMML is such a format, and there are tools (e.g. jpmml) to convert all your named modeling package objects to PMML, though perhaps you've used something else that isn't already easily-converted.

Otherwise, just force installation of dependencies (and make it easy), through a virtual environment or docker image or ...


Evaluation of semantic segmentation network with mAP

Here is a great post which explains the evaluation methods for semantic segmentation: https://www.jeremyjordan.me/evaluating-image-segmentation-models/

It also describes the mean average precision at the end.


Establish relationship between two sets of data

I think you may be able to shed light on relationships between features and parts using association rule learning.  You can treat the parts and features similarly to items in a market basket.

Updated:

A = {('P1', 'F1', 'F2', 'F3'),
     ('P2', 'F2', 'F4', 'F6'),
     ('P3', 'F1', 'F6', 'F8')}

B = {('P1', 'M1', 'M2', 'M3'),
    ('P2', 'M4', 'M7'),
    ('P3', 'M1', 'M5', 'M7', 'M9', 'M10')}

parts = {'P1','P2','P3'}

basket = {}
for k in parts:
    temp = []
    for a in A:
        if k in a:
            temp += list(a)
    for b in B:
        if k in b:
            temp += list(b)
    temp.remove(k)
    basket[k] = temp

Then use each value in the basket dictionary as a basket.


What should I do to test the confidence of my deep learning model?

For binary predictions, it is standard to evaluate models based on their ROC and PRC curves. Some metrics are also useful, namely MCC, which is probably the most holistic scalar metric.
Using these metrics, you should evaluate the model via cross-validation. For deep models that take significant time to train, k-fold cross-validation is often sufficient. If you want, you can also do repeated k-fold cross validation, if time permits.
Lastly, while not always possible, many consider the use of different datasets to be one of the best indicators of reliability. Splitting a single dataset still risks "leaking" a common bias in both the training and testing sets. When using two or more datasets, biases that the model learns in the training dataset probably wouldn't manifest in a completely different dataset, hence a more objective evaluation methodology that mimics a production environment. Such biases include, among many things, data acquisition methods, preprocessing/data cleaning methods, etc..


ValueError: X has 3 features per sample; expecting 1500

You should not so a fit transform while prediction. You should save the fit model in your preprocessing code for the tfidfvectorizer and use the same vectorizer while predicting.

Otherwise your vectorizer is learning new word representations and whose dimensions might not be 1500 what you have specified in your preprocessing code.


When is the sum of models the model of the sum?

The two models are not equivalent in general, either it might happen they provides similar results. There are multiple issues here. What you have actually in $Y$ is something like:
$$Y = \tilde{Y} + \epsilon = \tilde{Y_1} + \epsilon_1 + \tilde{Y_2} + \epsilon_2 + \tilde{Y_3} + \epsilon_3$$
where $\tilde{Y}$ and $\tilde{Y_i}$ are the true functions which you want to approximate and $\epsilon$ and $\epsilon_i$ are errors/noise.
One issue is how do you split output $Y$ into components. Supposing you have knowledge about components of $Y$ and you split it into simpler functions which are easier to fit independently. However in practice you have a sample and each observation has errors, you would have to split also the error components, which is not easy since most of the time the assumption used is that the errors are at least independent, if not identical distributed also.
The second one is the variance of the models. In the analogy with linear equations you stated
$$Y = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$$
Well, this is actually true only on expectation, aka:
$$EY = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$$
But your predictions have also variances, which can put you away from the true functions. If we assume the noise for each component is mutually independent, than the variances accumulates, giving an inflated variance for the summed model $$Var(\epsilon)\leq Var(\epsilon_1) + Var(\epsilon_2) + Var(\epsilon_3)$$
If the component errors are not independent than you can also add the covariances to the issue and you can easily go into trouble.
As a conclusion if you have three sets of observations for each $Y_i$ than it should be usually better to fit them separately rather than summed. But if you have the summed data set it is usually much harder to split data and the results have better chances to be more unstable, even if on expectation you have the same target.


How to combine rows after Pandas Groupby function

You should use the function drop_duplicates :

Define the columns you take into account:

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")

But it seems to me that the column Employee Id is enough, as you have only one person per Employee Id so you can define :

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")

Then you can write:

import pandas as pd

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")
df = df.drop_duplicates(subset=unique_columns)


How can I use all possible spelling correction of documents before clustering those documents?

The problem is that you cannot be a 100% sure that the predictions of the corrections of your algorithm will be true. Such an algorithm does not exist yet.

So you have 2 options :

Either you correct the algorithm errors by yourself, but it will be very long, and if your aim is to study the performance of a method it will bias the results. I would not recommand that.
Or you can just apply your algorithm and either set a probability threshold, i.e. you let the process correct the words if the probability is higher than an arbitrary value, say 75%, or correct all the words with the highest probability correction. You can then apply the classification model, and even if some corrections made were false, it should globally enhance the result, so it should be better than not correcting.


Affinity propagation fit, fit_predict and predict methods

It is correct to use as you indicated.

For example, in below , I use levenshtein for make similarity matrix. then use this matrix for clustring by affinity algorithm.

lev_similarity = -1*np.array([[distance.levenshtein(w1,w2) for w1 in words]
                              for w2 in words])

af = affprop.fit(lev_similarity)


Why we use an activation function for introducing nonlinearity instead of a polynomial Perceptron implementation?

Well, thanks to the universal approximation theorem from a purely theoretical point of view, absolutely nothing.
The main issue is with computation. You can find more information here. Mainly, you want functions easy to calculate (polynomials are ok) but with specific regions where derivatives are monotonic (here polynomials are not good) and approximating the identity near the origin (again, polynomials not good).


Clustering of news combining headline and main article

Put more weight on the title.

For example if you are using tf-idf, you first compute the tf-idf vectors of headline and body separately. Then you combine them into a single vector by simply adding both vectors and rescaling to unit length. That would put 50% of the weight onto the headline even though it is much shorter.


How do I write a SQL (or dwSQL) query across these two different datasets?

Have you tried something like this in SAS?

PROC SQL;
 CREATE TABLE TRUMP_FB_TWEETS AS
 SELECT *
 FROM TRUMP_FB
 UNION CORRESPONDING ALL
 SELECT *
 FROM TRUMP_TWEETS;
QUIT;

Source:http://www2.sas.com/proceedings/sugi25/25/cc/25p109.pdf


Pandas - Sum of multiple specific columns

df.loc['Total'] = pd.Series([df['Commission'].sum(),df['Profit'].sum(),df['Net profit'].sum()], index = ['Commission','Profit','Net profit'])


Converting a string to dummy encoded variables

It is simple with python:

from pandas import DataFrame
data = [('2GRG822L9',"barbarians8,valkyrie5,chr_balloon3,fire_spirits8,minion8,firespirit_hut6,rage4,skeleton_horde3",1), ('8PYPY0LLQ',"valkyrie5,chr_witch4,hog_rider5,zapMachine1,mega_minion3,baby_dragon2,bomber7,skeleton_horde1",0)]
df = DataFrame.from_records(data,columns=('PlayerID', 'Characters', 'Result'))
df = df.drop('Characters', 1).join(df.Characters.str.get_dummies(','))

The result:

    PlayerID  Result  baby_dragon2  barbarians8  bomber7  chr_balloon3  \
0  2GRG822L9       1             0            1        0             1
1  8PYPY0LLQ       0             1            0        1             0

   chr_witch4  fire_spirits8  firespirit_hut6  hog_rider5  mega_minion3  \
0           0              1                1           0             0
1           1              0                0           1             1

   minion8  rage4  skeleton_horde1  skeleton_horde3  valkyrie5  zapMachine1
0        1      1                0                1          1            0
1        0      0                1                0          1            1


scoring argument in scikit-learn LassoCV, LassoLarsCV, ElasticNetCV

RidgeCV implements CV by GridSearchCV, which supports custom scorer.

However, LassoCV and ElasticNetCV implement CV by LinearModelCV, with MSE as scoring hard-coded. LassoLarsCV impletments CV by LarsCV, with MSE as scoring hard-coded too.

If you want to use other scorers, maybe you can use GridSearchCV directly.

References

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/ridge.py#L1092

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/coordinate_descent.py#L1181

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/least_angle.py#L1143


SVM radial basis generate equation for hyperplane

I'm not sure if I've fully understood you. Radial basis kernel assumes that you transform your features into an infinite space and the dot product of your transformed vectors is exactly the radial basis kernel.

$k(x,y)=\phi(x)\cdot \phi(y)$

$\phi(x)$ - mapping

The main reason for using a kernel trick is the ability to transform features into higher dimensions without knowing the map function explicitly. Your hyperplane has infinite number of coefficients. You can always expend the radial basis kernel into Taylor series and get some of the initial coefficients.


Build a binary classifier with only positive and unlabeled data

My suggestion would be to attempt to build some kind of clustering on your unlabeled data that somewhat approximates a labelled dataset. The rationale is more or less as follows:

You have some feature vector for representing your documents
Based on that feature vector, you can come up with a number of different clusterings, with either fuzzy, rough, or class-based clustering methods
Knowing what a positive example looks like, you can quickly evaluate the overall similarity of a cluster to your positive cluster
Knowing that there should really only be two clusters, you can adjust the hyperparameters on your clustering method so that the above two metrics are closer and closer to satisfaction
With the two clusters, you have what is likely a close approximation of a labelled dataset, which you can then use as a silver-standard corpus of sorts to actually train your model

Hope that makes sense, if you're specifically looking for clustering algorithms, a few that I personally enjoy that might be good in this scenario are FLAME and tsne. Alternately, looking at the spectacular gensim library in python will get you a long way toward the clustering you're looking for.

Hope that helps and makes sense, leave a comment if you've got any questions.


unsupervised learning in medical systems and intelligent systems?

Expert systems rely on knowledge "extracted" from human subject matter experts (e.g. a team of physicians). They then use this knowledge to make prediction about the data (e.g. if blood pressure > x then patient).

Your approach used properties of the dataset you worked on. The knowledge was in fact extracted from the data themselves.

The second approach may or may not be compatible with expert advice. If your algorithm found that blood pressure was a good feature to split healthy people and patient, you will probably find experts agreeing. However, if your main feature comes from a combination of many variables (e.g. PCA) you may have trouble finding an expert validating your approach.

So the way you could use your approach would be to team up with subject matter experts and make sure that you use only features that have a meaning for them (i.e. don't over-engineer data).


Should output data scaling correspond to the activation function's output?

What you read hold true for the neurons of the output layers and not for the hidden layers!

Hence, its true that if you are using tanh in output layers then you need the data labels to be within [-1, 1] where as between [0, 1] for sigmoid.

As for your concern with relu, use it on output layers if you know that the range of the labels of the data is positive only. If you are using relu in hidden layers then the scaling doesn't depend on relu but rather on the type of activation function used in output layers.


Training set Distribution and Activation function/Loss function correlation

The probability distribution of the training set has normally nothing to do with the activation function/loss function. Instead, the activation function of the last layer and the loss function are directly defined by what you are trying to predict.
For instance:

If you have a regression problem where the output values are not bounded, you would probably use no activation in the last layers and a mean squared error as loss function.
If you want your network to perform binary classification, you would use a sigmoid activation in the last layer (which outputs a value between 0 and 1, associated with the probability of belonging to one class or the other), with a binary cross-entropy as loss function.
If you want your network to select among N elements (e.g. multiclass classification), you would probably want your network to predict a probability distribution over those N discrete elements. This is precisely a categorical/multinoulli distribution over a discrete output space (i.e. only one element is selected from N possible discrete alternatives). In this case, the activation of the last layer of your network should be a softmax, and the loss must be the categorical cross-entropy, also referred to as negative log-likelihood. Be careful, however, with frameword-specific details, because it is frequent that this pair of activation and loss is implemented with numerical stability considerations and you need to select the appropriate implementations. For instance, Pytorch, when you use NLLLoss, the last layer activation must be a LogSoftmax, not a normal Softmax. Alternatively, you can use CrossEntropyLoss which combines NLLLoss and LogSoftmax into a single class. These aspects are normally described in the framework documentation.


Can someone spot anything wrong with my LSTM forex model?

What you are up against is this fundamental property of most tradable, liquid financial price series, and that is, they are Brownian Motion. In discrete time, it's also known as random walk.

The most important property of Brownian Motion is that it's memoryless, whose mathematical expression is

$E[p_{t+1} | p_{-{\infty} : t}] = p_t$

Recurrent neural net, particularly the LSTM flavor, is very powerful in capturing and modeling a long memory process. In fact, it was invented to deal with state that depends on itself many time steps ago (that famous LSTM paper was dated 20 yrs ago[1]).

What you've demonstrated using RNN (assuming your code and training are free of bug) is precisely this property. Put it another way, there is no way to beat a fair coin in predicting tomorrow's FX price.

Another perspective on your attempt. If such naive model could accurately predict FX time series, it would have been exploited by the whales in the hedge fund industry long ago and the opportunity would cease to exist.

[1] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.


What do the parameters used in crop mean?

Lets say my image is called frame and x, y, w and h are xmin, ymin, xmax and ymax

You're confusing $w$ with $xmax$ and $h$ with $ymax$: Usually $w$ is the width of the crop whereas $xmax$ is the horizontal position of the end of the crop. Similarly $h$ is the height and $ymax$ is the vertical position of the end of the crop.
Logically since $x$ is the (horizontal) start of the crop and $w$ is the width, we can obtain $xmax$ like this: $xmax=x+w$.
Example: in a 100x100 image, let's say we want to crop a 20x20 square in the centre: $x=40, y=40, w=20, h=20, xmax=60, ymax=60$.
In the following code:

frame = frame[y:y + h, x:x + w]

the operator : is used to represent a sequence (for instance 3:7 means 3,4,5,6) so y:y + h represents the sequence from y to y+h, i.e. from $y$ to $ymax$. Same for x+w, so this line would select the part of the array corresponding to the crop.
Your second example is wrong due to the same confusion, the actual code is:
face = frame[startY:endY, startX:endX]

In this case the author is directly using the end coordinate endY (same as $ymax$) instead of calculating it as startY+h.


Distribution of Regression Residuals: Is this a normal distribution?

You can perform a statistical test to confirm your data is normally distributed
Try:
from scipy import stats

np.random.seed(42)
x = np.random.normal(2, 1, size=1000)
k2, p = stats.normaltest(x)
alpha = 0.001
print("p = {:g}".format(p))

if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")

This function tests the null hypothesis that a sample comes from a normal distribution. It is based on D’Agostino and Pearson’s test that combines skew and kurtosis to produce an omnibus test of normality.


Training data set for food image recognition

You could use the Food 101 dataset or
UEC Food 256, both contains real-world food images.


No, there is no such method, I have found out.

The reason is, plotting libraries run on a single machine and expect a rather sample dataset. Data on Spark is distributed among its clusters and hence needs to be brought to a local session first, from where it can be plotted.

That is why methods such as collect(), toPandas() are needed.


Why Keras Dense layer is expanding number of tensors in each layer

The number of parameters is the number of weights connecting the layer to the previous one, so for the layer $i$ it depends on the number of neurons in the layer $i$ and the previous one $i-1$.

The exact formula for a fully connected neural network is: $$n_i(n_{i-1}+1),$$ where $n_i$ is the number of neurons in the layer $i$, $n_{i+1}$ the number of neurons in the layer $i+1$, and the $+1$ term takes into account the bias.

So for dense_2477 you indeed get $$n_{\mathrm{params}}=8(16+1) = 136,$$ and
for dense_2482, $$n_{\mathrm{params}}=16*(8+1) = 144,$$ as expected.


This is likely the Exponential Moving Average function.
In simplest form it is
ma_new = alpha * new_sample + (1-alpha) * ma_old

Where alpha is the parameter that controls rate of decay, between 0.0-1.0.


To my eye, the second half of 2018 is substantially higher than the prior two years, so the trend that stl() is giving seems not unreasonable.

Trying to fit a time series model to only three years of data is going to require you to take a pretty strong hand guiding the model; I wouldn't expect a turn-key solution to give satisfactory results.

One obvious problem with the decomposition from stl() is that the seasonal figure is wildly overfit.  I would try using the Fourier series approach to fitting the seasonal figure that Rob Hyndman describes in this blog post.  I have applied that technique in exactly your situation and gotten decent results.

The closest to a turn-key solution is probably going to Facebook's prophet library.  But with only three years of data, I still suspect you'll overfit the seasonal component if you call it with the defaults.

If you post your actual data and describe your goals more completely, folks might be able to help more.


Extracting structure and content from invoices

I think extracting relevant details from an invoice in commercial applications certainly involves a lot of high spec algorithms. Maybe you are right that they identify relevant parts first and extract the details afterwards.

However, my first starting point would be to get all the text from an invoice (e.g. via tesseract). If you have a decent photo, tesseract will be able to OCR the content. The next step would be to identify relevant content, such as payment amount, names, and bank account numbers. This may be possible by hardcoded rules to some extent. Alternatively, one could use NLP-like models to detect certain sequences. With some effort, this should work out well since invoices are relatively structured documents.

https://pypi.org/project/pytesseract/

https://github.com/tesseract-ocr/tesseract/wiki


What algorithm is used to extract keywords from unstructured texts?

The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., this post).

The approach I would suggest is a key keyword approach advocated by Mike Scott (author of WordSmith Tools) and Chris Tribble (a computational linguist). As discussed here, the basic approach is to create two corpora, your target corpus which is composed of texts sampled from the texts you are interested in, and a reference corpus, which is typically a much larger corpus (which is often more general in content).

The procedure begins by computing word (or n-gram) frequencies for both corpora. During this process, if a word’s frequency in the target corpus is found to be statistically probable in comparison to the reference corpus (as computed by a chi-squared test and a user defined p-value), it is considered a keyword (Baker, 2004). According to Scott (2006), the process typically identifies three types of words as key:
proper nouns, words that characterize a text’s “aboutness,” and high frequency words that are indicators of style or genre. I discuss the approach in greater length in this 2007 article where I use the method to extract the salient features of academic discourse.

To illustrate with a concrete example, imagine your are interested in identifying the key topics surrounding a target brand (say, Pepsi Cola) as expressed in social media (e.g., twitter). Create two corpora: for your target corpora you create a search for "pepsi" and for the reference corpora you could search for competitor brands of soft drinks ("coke", "mountain dew", "dr. pepper", etc,). When the keyword process has terminated you will be left with all the keywords/topics that differentiate pepsi from other soft drink brands (and as a bonus you will also identify negative keywords... words that occur statistically less frequent in the target corpus).

As you might surmise, the results you get depend upon how the reference corpus is constructed. This, in my opinion, is a feature - as it gives the researcher much greater flexibility in hypothesis testing and data exploration.


What is the main goal of using activation function in CNN?

The idea of convolutional layers is that we need same weights to be applied to different regions of the input. It lets you identify same patterns that occur in different regions of the input. You can consider each convolution operation which is carried out by each window as a single neuron which just transforms a local region with a non-linear transformation. The linear and non-linear operations that are done by these so-called local neurons help the convolutional layers learn non-linear and complex features. Another aspect of this operation is that it reduces the number of weights significantly. The final interpretation is that if you don't use non-linear activations, you would have linear local features extracted from the input. Those linear features can easily be found with fewer number of parameters.


You have a nice free course about it on DataCamp

  Data Analysis in R, the data.table Way

The author of the course is Matt Dowle, the guy that actually wrote the data.table package.

Hope that helps.


How to improve precision under imbalanced classification

Introducing a Strong Feature would definitely help as it is "Strong" :-) .. if you do not have a sure-success feature then you may try changing penalty of miss-classification to start with.

You may try synthetic (i.e. SMOTE) or non-synthetic (domain based) approaches to bulk up the lower class.

Also, if this is a very very rare class the repeated sampling techniques may work


How to use fine tuning of BERT when i have unlabelled dataset of text documents?

So, how should I refine the word/sentence embeddings vector given by
  the BERT model in the case when I have a set completely unlabelled set
  of documents?

What are you looking to achieve with these unlabelled documents? If you are looking to classify them, then there is no way of getting around getting labels and fine-tuning on them.

  I'm aware that the BERT model is originally trained on unlabelled
  data, so there must be some way.

If you don't need to perform any specific end task (like classification) with the data, and instead is just looking to train BERT using your own data, then there is a way. BERT is self-supervised, meaning that it takes unlabelled data and automatically generates labels that it can train on. The same methods can be used to train on any large language corpus. However, this will only train the embeddings and fine-tuning on labels will still always be needed for classification.

I think huggingfaces' blog post is a good starting point if you want train a general BERT on your own dataset.

You can read about how BERT is trained on generated labels from text here.


Error on custom RNN/LSTM with multiple inputs

I think, input_shape[0] is None. Based on my understanding, you will pass a matrix of size (Batch_size, Time steps, dim of each time step data) to an RNN cell. And while creating your architecture, input_shape[0] will give the batch size (if you mentioned what is the batch size in input layer, if not you will get None). Even if you mention batch size, it will be an integer and you can't index the integers so you are getting that error.


Assuming the server is as at least as powerful as the first local machine, you should obtain the same time or shorter duration by request. If not, there's a problem worth investigating.

Then to improve further it could be useful to design the server side architecture so that:

the model is pre-loaded in memory, not read from file every time (huge bottleneck).
there are multiple instances of the program running in parallel so as to distribute the load.


Subsequence parttern matching for time series

The real state of the art here is the Matrix Profile suite, developed by Eamonn Keogh and his team in University of California at Riverside (UCR). Here are some links to get you started:

Matrix Profile Foundation homepage
The UCR Matrix Profile Page
MPA: a novel cross-language API for time series analysis paper (2020) with links to Python, R, and Go implementations
Introducing matrixprofile-ts, a Python Library for Detecting Patterns and Anomalies in Massive Datasets
Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile (PhD thesis)
100 Time Series Data Mining Questions (with answers!) (highly recommended high-level view)

You'll find links to implementations in the above stuff.
Apart from the team at UCR, there is another (and possibly more efficient) implementation of the relevant algorithms in the STUMPY Python package:

STUMPY documentation & Github repo
STUMPY: A Powerful and Scalable Python Library for Time Series Data Mining paper (2019)