Corelation between overtime and sick leave

I don't see why you need a model at all.  This seems like a simple counting problem.  Here is SQL that finds pairs where one employee takes leave and another gets overtime:

select leave.employee_code, overtime.employee_code, count(*) as occurrences
from time_off leave
join time_off overtime
on leave.employee_id = overtime.employee_id
   and leave.period = overtime.period
where leave.leave_hours > 0
  and overtime.overtime_hours > 0
group by leave.employee_code, overtime.employee_code
order by occurrences desc

In dplyr:

time_off %>%
  filter(leave_hours > 0) %>%
  select(employee_code) %>%
  inner_join(time_off %>%
               filter(overtime_hours) %>%
               select(employee_code),
             by=c("employee_code", "period"),
             suffix=c("_leave","_overtime")) %>%
  group_by(employee_code_leave, employee_code_overtime) %>%
  summarize(n=length(employee_code_leave)) %>%
  arrange(desc(n))

Note that I haven't run this code, so it might have typos or bugs, but it should be close to correct.

To explore the relationship between leave hours and overtime hours, you don't even need the  employee pairs, you can just aggregate by period and make a scatter plot.


Predict time of dispatch for marketing campaign

I might suggest a change to your theoretical setup:

It sounds like you are trying to maximize an equation f(t), where t is the time of dispatch, and f(t) is the risk adjusted chance of purchasing (as you note it can be 0-many).

In that case, it is likely better to use a setup like this:

===========================================================
  customer campaign    campaign_time       total_responses_within_1week
-----------------------------------------------------------
1   100       a     2017-01-01 06:50:01       0
2   101       a     2017-01-01 06:50:01       3
3   101       a     2017-01-01 06:50:01       2
4   100       b     2017-01-07 06:30:21       1
5   101       b     2017-01-07 06:30:21       1
6   100       c     2017-01-14 06:43:55       1
7   101       d     2017-01-21 14:02:01       1
-----------------------------------------------------------

Unless you have specific internal data or journal research that directly indicates a relationship between speed of response and likelihood/number of purchases, I might avoid including it.

Now, if you need to estimate time of response for issues like labor or inventory, you could use a probabilistic model, but I'll avoid making specific recommendations there, as my relative strength is in regression models. You'd want the function to output a probabilistic curve over a given time period, and then sum all of those curves together (e.g. to estimate that during Weds morning, you will have 80-109 orders, so need 5 customer service reps).

If your goal is purely response frequency (within a relevant time), time of response doesn't need to be modeled, however.


Have you tried rule-based approaches?

Based on your example, I can think of two ways:

For each picture, get a list of all RGB values it contains.
The first contains only grays (which have RGB values (x, x, x)), the second one contains color (so some pixel with RGB value (x, y, z), where $x \neq y$ or $y \neq z$). Or - if that is faster - convert all pictures to gray scale and check if the converted picture is identical to the original.
Scan the image for areas (larger than one pixel) with identical colors. If your dots are large enough and the background noise is uncorrelated for neighboring pixels, it is very likely that these areas are the dots you are looking for.


If I understood correctly, you want to invent new negative samples to have a balanced dataset, because your current dataset only have positive samples.

However, the library you mentioned only re-samples a dataset where all classes are present. The sample creation process is by definition data-specific, as there can be impossible attribute combinations, etc.

If you want to blindly create combination of attribute values, you can use the stuff in module random:

To create integer attributes:
def random_integer(min_val, max_val):
    return randint.randint(0, 9))

To create real-valued attributes:
def random_float(min_val, max_val):
    return random.uniform(min_val, max_val)

To create discrete attributes:
def random_discrete(value_list):
    return random.choice(value_list)

And then, with a loop, generate your negative samples in a loop:

    def generate_negative_sample():
        return {'attr1': random_discrete(['a', 'b', 'c']),
                'attr2': random_float(-10., 10.),
                'attr3': random_integer(0, 4)}

    random_samples = [generate_negative_sample() for _ in range(1000)]


Can I make kaggle kernels read directly from my computer?

No, you cannot write a Kaggle kernel that will read your local data directly.

The best you can do is upload your data to the kernel. See https://www.kaggle.com/docs/kernels#adding-data-sources


Actor-critic architecture: How is the policy updated?

You basically have two problems on your hand:

learning the Q-function
learning the policy

The Q-functions tells you how rewarding an action $a$ would be in state $s$. Once you take the action $a$ and transition from state $s$ to $s'$, you are rewarded $r$ right away. How rewarding it was to take the action $a$ in $s$ is the sum of the received reward $r$ and the Q-value of the state that you end up in, hence you want to minimize

$$r + Q(s', a', w) - Q(s, a, w)$$

where $w$ are the parameters of the function approximator and $a'$ the action you would take in the following state $s'$. This is the TD Error and mimimizing it will learn the Q-function.

So, the TD Error is used to learn the Q-function, basically the same way as it is done in plain-vanilla (Deep) Q-Learning. But while in plain-vanilla (Deep) Q-learning you would base the actor's policy on the Q-function entirely (plus some randomness to not get stuck), in the actor-critic context the Q-function is just a means to the end of training the policy itself. For this reason, the learned Q-function then serves as an optimization objective for the policy by learning the policy parameters $u$ such that it outputs the action $a$ which maximizes $Q(s,a,w)$ (see your other question). The chain rule pops up in that context, because you are not trying to do regression by minimizing some error, but rather you are trying to maximize $Q$ with respect to one of its arguments.


You can do many things:

Forget about the labels: just use the features that are not labels and cluster along those features using the k-means algorithm (or another).
Forget about the features: this is the dummiest way of clustering. Cluster the data in 29 clusters according to the labels that they have. If you want less clusters, you can compute the centroids of the classes and use them to join clusters of different labels.
Use everything: create a categorical variable refering to the class that every example belongs to. Then, with this new variable and all the features perform a classical clustering algorithm.

The way to proceed depends on if you want to use the labels or not, and how much importance you want them to have.


With 12 or more CPU cores, you wan work with pretty much any deep learning model.  Only thing that will not work is training with 100k+ images or more than TBs of test.  For such scenarios; you can seed the model with transfer learning.

For example, training a Cat/Dog image classification model (from scratch) takes 2 hours on 6 core i5 (as compared to 30 minutes on GTX 1080ti).


I would start by looking at the distribution of bounce rates for each individual. That distribution would provide some details around what could be considered a "normal" bounce rate, and where you would draw a line for an extreme bounce rate that implies a 'bad'. Additionally, you can also look at average bounce rates (divide bounce rate by the tenure), and review these averages by tenure. This may reveal that different threshold values might be appropriate for customers with different tenure. For instance, if a customer is new (first month of activity) then even a moderate bounce rate could imply a 'bad'.

Additional metrics -- like the ratio of bounce to valid payments, time between bounce -- could be useful to look at as well.

Once a set of initial thresholds are identified, I would closely review some of the customers who are identified as 'bad' based on those thresholds to ensure that there are not too many false positives. The business point of view (from those who would be using this information to make decisions) would be great to incorporate into this review.


I am trying to figure out the stationarity of time series data

There are certainly more tests you could try out but by definition of the test, you can conclude that your data is stationary.
On the other hand I would be concerned with your test results. Getting a p of exactly zero, and a test statistic value of way below -3 or -4  I would look at your implementation again and check if it's right.


what does random seed value mean in weka?

Seed is just a value by which you can fix the Random Numbers that are being generated in your task. Also, this is a general concept and not just for weka.

So, here random numbers are being used to split the data. Now if you run the code without fixing any seed, you will get different splits on every run. But if you fix the seed to some specific value, you will get the same split every time. This is useful when you want to make your scores reproducable.


Rstudio using 2.5% of 250GB RAM. how to Increase it

You may or may not already know this, but here are a few basics about how R and Rstudio work and use resources.

Rstudio is a graphical user interface to R, not the interpeter/runtime environment.  There is a separate "R session" that actually executes your R programs and returns results for Rstudio to display.  Therefore, having Rstudio use more memory won't make any difference in the execution speed of your program.

Second, memory allocation and cleanup (a.k.a. "garbage collection") is handled automatically by the R runtime environment.  "R holds objects it is using in virtual memory". Virtual memory is a combination of physical main memory and secondary storage, the mixture determined by the server OS configuration.  You can use command line to change the amount of virtual memory allocated to processes. (Consult with your favorite Linux expert on this.)

Third, your speed of your program may or may not be limited by memory speed. You may be compute-bound.  Do some testing to find out what is constraining performance.

Fourth, you should first ask yourself whether your program implements an efficient algorithm.  Even if you aren't programming with loops and branches, the functions you call may use them, and maybe not efficiently for your application.  For example, there are dramatic performance gains to be had by switching to data.table from data.frame.

Fifth, once you have chosen an efficient algorithm, you can put effort into parallel execution.  The simplest way to do this is by using functions that automatically vectorize operations.  A bit more complicated is to recode your program use the packages doParallel and foreach.  With doParallel, you can specify the number of CPU cores to use, which on a server may range from 32 to 64 or more.  Finally, if your server has a Graphics Processing Unit (GPU), it's possible for some algorithms to reprogram using the GPU commands and get massive parallelism.  This option takes the most effort and has the most constraints.


Fuzzy and FuzzyWuzzy: what are the differences in text comparison?

In the source code you can find a simple explanation of what does partial ratio in fuzzywuzzy does:

Return the ratio of the most similar substring
as a number between 0 and 100

In this code snippet you can find the differences
from fuzzywuzzy import fuzz

fuzz.ratio("this is a test", "this is a test!")
Out:    97

fuzz.partial_ratio("this is a test", "this is a test!")
Out:    100

[1] https://github.com/seatgeek/fuzzywuzzy
[2] https://github.com/seatgeek/fuzzywuzzy/blob/master/fuzzywuzzy/fuzz.py


Extracting data and saving data from hdf5

You can always try using the h5py module in Python.

import h5py

# Load existing file
filename = "myfile.hdf5"
dataset = h5py.File(filename, "r")

# Show keys
[print(item) for item in dataset.items()]

# Assuming that you want to select the first 10 rows of key "A"
data = dataset["A"][:10]
dataset.close()

# Create the new dataset / file
dataset2 = h5py.File("myfile2.hdf5", "w")
dataset2.create_dataset("A", data)
dataset2.close()


Why does putting a 1/2 in front of the squared error make the math easier?

A major reason for using MSE is to optimize the parameters of a regression model. From calculus, you know how to find the minimum of a function by taking the derivative. That puts a "2" out in front, which is irritating to keep writing, so it is reasonable to put a "1/2" at the beginning so the derivative doesn't need a constant out front.

We get away with it because the minimum of f(x) and f(x)/2 is achieved at the same value(s) of x.


Latent variable graph in Variational Autoencoder

As Nikos said in a comment, this is a graph showing the different classes (digits) in the space of the latent variable.
First a remark: from the point of view of visualization design I think the authors made a mistake by using a gradient of colours for the classes, it should have been a completely different colour for every digit: here the digits are a categorical variable, there is no notion of order between them. This is misleading because the reader of the graph would naturally expect some logic in the gradient, but there is none.
The space itself is not interpretable, in the sense that the values on the X and Y don't have a meaning (for a human). However what this graph can tell you is how close or how distinguishable are two classes: ideally the different classes are fairly distinct from each other, which means that the model is sucessful at separating them.
Unfortunately due to the mistake I mentioned above it's quite difficult to see things clearly in this graph, but if I see the colours correctly there are a few things observable:

the 1 is the big purple thing on the top right, it's mostly distinct from the rest.
The 0 is the smaller dark purple group on the top left, also quite distinct.

The other digits are much more mixed with each other, it's hard to see anything:

Apparently the 9 (yellow) spreads a bit everywhere in the centre, and there are many points for 7 (I think?) in the middle: this means that potentially the model could sometimes confuse 7 and 9 (it makes sense: the handwritten 7 and 9 can have the same shape).
The 6 seems to be the rather thin greenish thing which is closest to the 0, and I can imagine indeed that these two have a similar shape.

My eyes are not that good so maybe I made some mistakes, but I hope you see the idea.
Basically the more the different classes/colours are well separated on the graph, the better the model, so you should be able to see some difference between some very good hyper-parameters and some very bad ones. But it would be pointless to rely on the graph for that: a good old evaluation measure (or confusion matrix) is much more accurate.


How to create more dimensions for data mining?

In general, this is called feature engineering. Expansion of dataset attributes via attribute transformations is called basis function expansion. These are often nonlinear functions of the original attributes. Popular nonlinear functions are polynomials.


How can I make a prediction in a regression model if a category has not been observed already?

Technically, you can't. That is one of the limitations of regression models; they are really only effective for values/ranges that they have seen before. Your use of categorical values makes it even more complex. But even with continuous variables, it is not recommended to use regression models for these "unknown" values.


ground truth fit is worse than cross validated fit on noisy data?

The main problem in your test is that your X has the same scale as your noise level (0-1), as a result, adding a noise changes your data distribution significantly.
This is your data distribution before and after adding noise.

It is like the noise is 50-200% more than your initial data. That's why you get a better result with CV than the "ground truth weights". Because the new fitted model by CV is tuned for an X with a new distribution.
However, if you increase your X values scale, for example 100*np.random.rand(ntrials,nneurons), the your data have same distribution before and after adding noise.

Now if you run your code, you will get what you expected.


Pandas how to fill missing values in one column if the values in another column are equal


Can I use Tensorboard to manage-fjobs and visualize learning on non-Tensorflow algoithrms? (e.g. Scikit?)

For logging values with TensorBoard (without running TensorFlow), you can use the tensorboard logger library.


What could be a dataset in which the presence of an outlier dramatically affects the performance of Ordinary Least Squares (OLS) regression?

If you are looking for a real-world data set here is one on Harvard's Dataverse that examines state social politics research for outliers for the same purpose you are looking for. If you are looking for one for more illustrative purposes one data set worth knowing is Anscombe's quartet for demonstrating how misleading some descriptive statistics can be. For your own investigations, many data sets with and without outliers can be found on Google's beta dataset search and are worth exploring if you are curious!


what should be the order of class names in sklearn tree export function (Beginner question on python sklearn)

The names should be given in ascending numerical order.

Saw this in the code's docs:

class_names : list of strings, bool or None, optional (default=None)
    Names of each of the target classes in ascending numerical order.
    Only relevant for classification and not supported for multi-output.
    If ``True``, shows a symbolic representation of the class name.


Is there a rule of thumb when designing neural network in deep reinforcement learning?

The process of reinforcement learning already implies that you have a base model to work from, that's what you're reinforcing. So, presumably, that underlying model is already good, otherwise you wouldn't be using it, right?

The whole point of reinforcement learning is to introduce your (functioning) model to new information and/or changing conditions. Reinforcement learning isn't going to take a bad model and turn it into a good one. If your base model is not performing to your satisfaction, then you should go back to a model design/selection phase; reinforcement learning is not going to help you in that scenario.


What are the best tools for manually annotating textual topic segmentation?

Since I couldn't find a suitable tool, so I made a tool for textual topic segmentation.

To anyone interested in this question, I forked YEDDA's entity recognition project and changed it, accordingly. You can check it here in GitHub: https://github.com/ThiagoSousa/YEEDASeg. My project is called YEDDASeg.

Edit: People mentioned to use BRAT, but it is good to select a word and tag with with a label. My case is a bit more complicated, because it can be many paragraphs. In YEDDASeg, you basic just click on the text and press a key to label a topic in the text. It is simpler for this purpose.


Yes, there are decision tree algorithms using this criterion, e.g. see C4.5 algorithm, and it is also used in random forest classifiers. See, for example, the random forest classifier scikit learn documentation:

  criterion: string, optional (default=”gini”)
  The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain. Note: this parameter is tree-specific.

Please note that usually it is referred to as cross-entropy or information gain rather than log loss.

For some background information you can give this and the links provided in the answers a read (although they may refer rather to decision trees):
When should I use Gini Impurity as opposed to Information Gain?


I think that you can use a topic model such as Latent Dirichlet Allocation (LDA). For example, in this paper https://pdfs.semanticscholar.org/9e6f/33bdd04df0536f6ad6783d33cccfbc54b1b1.pdf it is used for music and images. I suggest you to take a look at it :) . In general, in topic modeling you end up with a list of topics, where each topic contains a set of associated keywords. In clustering, depending on the algorithm, you might have hierarchy of dependencies. You can also use algorithms which assign each sample to only one class. In addition to this, when doing clustering, you usually have a distance metric which you have to pre-define (e.g. Euclidean distance). The topic models, especially LDA are based on the assumption that your data represents a distribution of topics with their corresponding distribution of keywords (one keyword can be contained in many topics). In other words, you already assume how the texts/documents have been generated.


How to interpret the mean for output clusters for expected-maximization?

It seems like you are trying to create a mixture of 2 univariate distributions, but you happen to get a bivariate distribution (which is why you have a 2x2 array).

This is because you reshaped the X_train array. You need to change your penultimate line to:

gmm.fit(X_train)


Tensorflow take ages for tf.cond and eval() - python code (sorry but i asked on Stackoverflow but none answer me)

The reason why tf.tensordot(), tf.math.argmax() doesn't change running regardless of the input size is because they don't make the operations, but rather declare it.
It is similar to passing a partial function using functools.partial, the function is not really executed, it just determines it's inputs (or part of it).
from functools import partial
def tensordot(a,b):
    return partial(lambda a,b: a*b,a,b)

While running eval will run the previously mounted model
dot = tensordot(a,b) #dot here is a function
dot() # <- done inside eval

Actually, on tensorflow these "partials" are objects and when you eval one of then, the previously declared objects that it depends of will be evaluated


Inverse_Transform and multistep data - how to scale and when

On the original code:
sc = MinMaxScaler()
test_set_tomorrow = dataset_train.iloc[-time_step:,:].values

Hm, you missed the shuffling, but it's OK -- maybe your data was shuffled earlier.

Original code:
scaled_test_set = sc.fit_transform(test_set_tomorrow)

The model should be trained on the train set not the test set - otherwise, data leakage occurs. It should be either:
sc.fit(dataset_train.iloc[:time_step])
scaled_test_set = sc.transform(test_set_tomorrow)

or
scaled_train_set = sc.fit_transform(train_set_tomorrow)

Original code:
scaled_test_set = test_set_tomorrow

You just erased all the pre-processing. If you check the data, scaled_test_set is exactly dataset_train.iloc[-time_step:, :]

Original code:
X_test = []
for i in range (0,len(test_set_tomorrow)):
  X_test.append(scaled_test_set[i,:])
X_test = np.array(X_test)
X_test = np.reshape(X_test,(1,time_step,-1))

can be done quicker. Note that the MinMaxScaler returns the numpy array already.
X_test = np.expand_dims(scaled_test_set, axis=0)

Finally, you apply the inverse transform to the labels while training on the features. For example, consider the model where given height, we predict weight. You create a map that normalizes inches (or centimeters). In case of centimeters, say the smallest height is 150, but it is not true that the smallest weight is 150 kgs or pounds. Different scalers should be used.


Different results obtained for OneVsOneClassifier (or OneVsRestClassifier) when using ordinary KFold and StratifiedKFold cross validation

TL;DR

Use StratifiedKFold and the best performing model (either OneVsOneClassifier or OneVsRestClassifier).

Splitting your training and test set.

Below is are the two approaches you could use for splitting your train and test set. The KFold and the StratifiedKFold:

The main difference between the two is that StratifiedKFold will ensure that the class distributions are always the same in train and test. If you would use simple KFold, it might be possible that a class will appear in the test set, but never in the training set. If that happens, all the samples from that class will get wrongly predicted. So, StratifiedKFold at least gives your model a chance to see one sample from each class.

Regardless of the model you pick, use StratifiedKFold to evaluate your model.

OneVsOne or OneVsRest

It is difficult to say which model would perform best because it will depend on your data's distribution. So, I would say for you to evaluate both and pick the best performing one.

What is interesting, is that you get to choose what best performing means. It could be
the mean precision/recall/F1 score in a cross-validation setting. But it could also be the one that uses less memory, or the fastest. In general it is a mix of all these things with tradeoffs between them. In general, the OneVsOneClassifier performs better in the F1 score, but it is slower than the OneVsRestClassifier, but please not the words "in general". Perform your own tests and see which one works for you.

I usually prefer the OneVsOneClassifier, because it tries to model every class against each class. This way, classes are self contained and finding boundaries around that space is more convenient. When you have a OneVsRestClassifier, you need to model the "rest" class, which can be really complex since it is made up of various classes, which can become problematic.


Storing and loading bottleneck features for transfer learning on large data sets (Keras)

You can first write the bottleneck features into a tfrecords file, and then load them as a dataset for the training phase.
In the tensorflow documentation you can find complete examples of how to do both.


What exactly is .csv in machine learning?

A csv file is a way to store an array (similar to an excel table).
With a code like this one, you can save a pandas dataframe to a csv file:

 import pandas as pd
 df = pd.DataFrame({'c1':[1, 2, 3, 4], 'c2':['a', 'b', 'c', 'd']})
 df.to_csv('df.csv')

which will look like that

,c1,c2
0,1,a
1,2,b
2,3,c
3,4,d


Problems with decision tree labeling of nodes

Decision Tree does assign the label based on majority given the attribute test condition and its value.

Regarding the class label assignment-

In case DT has a longer  depth, there might not be enough instance left for a certain branch/test condition/node . then this might not be the reliable estimation of the class label statistically. This is also called Data fragmentation problem.

so a DT with 50 nodes, at dept 10, for day = Humid there is only 1 instance left which is -ve. So Its assigned as -ve but there is not enough data ideally to support this.

One way to solve this is to dis-allow to grow the tree beyond a certain threshold in terms of number of node i.e. stopping condition.

Which also brings us to Over-fitting, Regarding Over-fitting- There is this classic Error vs number of nodes graph on train and test to show how over-fitting happens in DT.

As you can see in below graph, tree with more number of nodes has lower training error but while its being tested error is higher. The gap between test and training error is telling us that the tree is over-fitting/has captured the noise when tree size is growing.

Now Random Forest is a Assembly/forest of multiple Decision Trees. While classifying the example we take majority voting out of Trees.


Estimating Titan X graphics card impact on performance

There are a lot of parameters which matter when using GPU's for machine learning, some of them are:

CUDA core count
Memory bandwidth (GB/s)
Memory per core (MB)
Raw Speed (MHz)
Total Memory available (GB)
Performance on 16-bit, 32-bit floating ops/sec

Tim Dettmers has an excellent (frequently updated) blog where he's compared different cards, near the end he's also given a simple speed-up comparison. Using that as a guide, it can be estimated that Titan X pascal would be upto 5 times faster than a GTX 970 for Deep Learning.


An average of averages problem, or two valid results?

In you first approach you take into account the number of logins for each user, which you do not do in the second approach. Take the following example with two users with a different amount of logins:
$
u_1 = {7, 5, 10, 10, 8} \\
u_2 = {1}
$
Using the first approach the average would be 6.8 days whereas the second approach would give an average of 4.5 days. The second approach does not take into account that we have more observations for the first user (with a higher number of days between logins) than for the first user. Which approach is the best would depend on what question you are trying to answer, but I think generally the first approach would be more accurate in getting the average time between logins.


How to use df.groupby() to select and sum specific columns w/o pandas trimming total number of columns

I think the answer depends on what you want to do with column 6. Keep in mind that the values for column6 may be different for each groupby on columns 3,4 and 5, so you will need to decide which value to display. Typically, when using a groupby, you need to include all columns that you want to be included in the result, in either the groupby part or the statistics part of the query.
If you don't want to group by that column, you can just display the min or mode value. In general, if you want to calculate statistics on some columns and keep multiple non-grouped columns in your output, you can use the agg function within the groupyby function.
Example with most common value for column6 displayed:
df.groupby('Column1').agg({'Column3': ['sum'], 'Column4': ['sum'], 'Column5': ['sum'], 'Column6': pd.Series.mode})

Full example with code:

If there is a tie for most common, with one Mary and one Jane both being Female Engineers, this will generate an error as mode doesn't reduce to a single answer:

You will need to use another aggregate in that case, such as min, which will choose Jane as an alphanumeric min:

If you don't like the look of the multi-index, you can remove it using as_index=False and replacing the column names with a list(map(...join...))

or remove it using to_flat_index() which gives a slightly different format for the columns:


Clustering a variable based on another variable or set of variables

First of all, clustering is used only for numeric values, especially for continuous values. What you are trying to do here is to cluster a categorical variable, that too an ID column. I'm unsure about the goal but this is not a good technique to cluster values.

That being said, I'm not sure how many unique entries you have in the ID column. You have to convert it into categorical numbers before clustering. Then take only the ID column and the CRPO_VDNN column and use KNN to cluster. You can set the number of clusters in the KNN too.


Remove input data to test for leakage

This is very generalized question, so without knowing the types and provenance of the input data, this can be hard to answer.

But, in general, to check for leakage, you can use the model on some subsets of the input variables while removing other input variables. If you get data from multiple sources, then try removing all input variables from a single source, then re-run your models. You may be able to identify the source of the data leakage. Alternately, if computational power allows, you can brute force it by running the model with each of the 150 input variables removed, or all sets of two variables, etc.

Use customer-centered time data

Regarding model meta-data, again I would investigate data provenance. Are you predicting churn using the complete patterns of customers who stopped using the service? What I mean to say is, instead of looking backwards from a fixed real-time period, like today, to all customers who did or did not stop using the service, try looking from a fixed customer-time. Use only data from the first year that each customer used the service, and attempt to predict whether each customer will remain with the service for another year.

The warning signs of a customer dropping the service may be obvious in the months leading up to that customer dropping the service, but by then, the predictive power of your model may be too late to stop that customer from leaving. Instead, index the time component of each customer's history to zero when the first start using the service, and run your model on this data.


What's the difference of Symbolic translation approach and statistical translation?

By symbolic translation, I assume that you meant rule-based translation. As a simple example, consider translating the English sentence "I eat apples" to Spanish.

Rule-based translation works by first tagging the words with their part of speech ("I" as noun/subject, "eat" as verb, "apples" as noun/object). Then, create an English parse tree and transform that into a Spanish parse tree (this requires knowledge of how English phrases map to Spanish phrases). Lastly, translate each English word into its corresponding Spanish word, and use the Spanish parse tree to determine the structure of the sentence.

The simplest statistical machine translation system are n-gram models based on n-gram frequencies. For an explanation of the mathematics behind statistical translation, see this paper: http://www.mitpressjournals.org/doi/pdf/10.1162/coli.2006.32.4.527.


Model with 2 datasets: combine time series data and statistics

There are multiple approaches. Two of them are:

Convert time series data to structured data
Convert your statistic data to time series data

This article is a good introduction.
For your second question about giving more weight to some features, please read this post.
Hope it helps.


Is there a python code to perform the following task in python

If dataset means pandas.DataFrame then you can use

dataset['rate'] < 6

to get dataset with values True/False which can sum because True is converted to 1 and False is converted to 0

count = sum(dataset['rate']<6)  # 17

If you get number of all rows in dataset len(dataset) then you can calculate percentage value

percent = count/len(dataset)   # 0.8 (80%)

Working example

import pandas as pd

dataset = pd.DataFrame({
    'rate': [9, 6, 2, 2, 8, 5, 4, 4, 5, 4, 4, 2, 4, 5, 3, 5, 7, 3, 5, 4, 1]
})

count = sum(dataset['rate']<6)
percent = count/len(dataset)

print(count, percent, '{:.2%}'.format(percent))

Result

17 0.8095238095238095 80.95%


Even I'm also not sure about Weka, but so far I read, it provides functionality for data mining, data pre-processing, model training.

Generally what I do,

In case of pre-processing: model generated after pre-processing of training data(it includes SMOTE data), I save this model. Saved model is used to pre-process test data.

Let me know in case, you are expecting something different in answer.


What should I care about while stacking as an ensemble method?

To directly answer your question about stacking: you should care about minimizing 1) bias, and 2) variance. This is obvious, but in practice this often comes down to simply having models which are "diverse". (I apologize that link is behind a paywall, but there are a few others like it and you may well find it other ways)

You don't want ensembles of like-minded models - they will make the same mistakes and reinforce each other.

In the case of stacking, what is happening? You are letting the outputs of the probabilistic classifiers on the actual feature input become the new features. A diverse set of classifiers which can in any way give signals about edge cases is desirable. If classifier 1 is terrible at classes A, B, and C but fantastic at class D, or a certain edge case, it is still a good contribution to the ensemble.

This is why neural nets are so good at what they do in image recognition - deep nets are in fact recursive logistic regression stacking ensembles! Nowadays people don't always use the sigmoid activation and there are many layer architectures, but it's the same general idea.

What I would recommend is trying to maximize the diversity of your ensemble by using some of the similarity metrics on the classifiers' prediction output vectors (ie, Diettrich's Kappa statistic) in training. Here is another good reference.

Hope that helps.


Is it a good idea to train with a feature which value will be fixed in future predictions?

Given that in your training data this feature has different values and some predictive power, I think not keeping this feature would be a mistake (without looking into overfitting due to having too many features). You cannot just discard the feature from your training set if it does influence the target because then these would be from a different population than your predictions and it will be able to learn from the other features.

Extreme example where x_2 will always be 5 in the future:

x_1  x_2  y
2    8    6
3    7    5
2.5  5    1.5
3    5    0.5

Just removing x_2 loses a lot of information and would create a significant bias towards higher targets.


For the number to be same when seen upside down:

It can only contain either 0,1,8,6,9.

For first and last digit :
For it to be a 6 digit number first digit and the last digit cannot be 0.
If first digit is 6 then last should be 9 and vice-versa.
If first digit is other than 6 or 9 (ie. 1,8) then they both should have same digit.

For second and fifth digit / For third and fourth digit :
If first digit is 6 then last should be 9 and vice-versa.
If first digit is other than 6 or 9 (ie. 1,8) then they both should have same digit.

Total Number of 6 digit numbers = 9 * 10 * 10 * 10 * 10 * 10  ([9] as first digit cannot be 0)
Numerator = 4 * 5 * 5 ([4] as first and last digit cant be 0)
So,
Probability should be = Numerator / Total number of 6  digit numbers


Don't know how to preprocess my dataset for image classification

Find a working Keras setup for a pretrained model with custom data generator here.

Note that validation data should not be augmented!

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=20,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=20,
        class_mode='binary')

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=lr_first), # 2e-5
              metrics=['acc'])


I need to find Simplest exercises available for learning neural networks

I'm learning myself too and I found kaggle.com and excellent resource for learning and practice, they have lots of data sets available for you to try your own models.

There is also a learn section with lots of topics for beginners

Hope that helps.


The source, around line 274 is where the default scoring for cross_validation_score gets set, if you pass in None for the scorer argument.  For classifiers, the usual default score is accuracy.  For regression, it's rmse, IIRC.  So, since you're applying a decision tree classifier, cross_val_score splits the data into 5 equalish sized pieces, trains on each combination of 4 and gives back the accuracy of the estimator on the 5th.  The mean and std of these accuracies presumably tells one something about the performance of the family of decision tree classifiers on your dataset, but I would take it with a grain of salt.


Class Imbalence Problem even after Balancing Data

The fact tha balancing the dataset Will prevent the overfitting and thus, good results in the test set, Works under the assumption that the model does not underfit.

What it may be happening is that the model is too bad that it can't perform the classification task in either the un-balanced and the balanced dataset.

It's difficult to analyse the problema without more information, you may have facing convergence issues (too large learning rate, vanishing gradients), you may also be computing the accuracy in a bad way...


How does Google categorize results from its image search?

I am not working at Google, but I think it is some sort of recommendation system based on the words which millions of users searched before. So those people who search for "animals" often search for "wild animals" for example. Like in many online stores they recommend you to buy something in addition to the product you are looking for based on the previous purchases of other users.

There are many approaches how to build such recommendation system using machine learning, no one knows for sure what google uses.


The error is caused by passing a numpy array into a function that expects an integer value. read_csv() will read a file, and create a numpy array from the data inside. You can slice off the column of the numpy array that you want to use, convert it to a list and then pass this one by one into classes[]
# this should work, but change the value of 12 to the column you want to convert to a class label
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

x_new = p.read_csv('train.csv', sep=',').values # x_new: <class 'numpy.ndarray'>
y_predict = train_test_split(x_new) # y_predict: <class 'list'>

z = np.asarray(y_predict)[0].T  # transpose your y_predict dataset
for i in range(0,len(z)):
    print(z[12][i])
    print(classes[z[12][i]])

Issues with the datatypes in your original code:
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

x_new = p.read_csv('train.csv', sep=',').values
print('x_new:',type(x_new))
y_predict = train_test_split(x_new)
print('y_predict:',type(y_predict))
print('y_predict[0]',type(y_predict[0]),y_predict[0])
print(classes[y_predict[0]])
print(classes[y_predict[1]])

x_new: <class 'numpy.ndarray'>
y_predict: <class 'list'>
y_predict[0] <class 'numpy.ndarray'>

[[2078 90.59 'y' ... 0 0 0]
[7384 106.47 'x' ... 0 0 0]
[2860 115.75 'ak' ... 0 0 0]
...
[7897 87.76 'f' ... 0 0 0]
[5029 87.16 'f' ... 0 0 0]
[2414 99.16 'aj' ... 0 0 0]]

    TypeError                                 Traceback (most recent call last)
    <ipython-input-57-ea39b293dd61> in <module>
         10 print('y_predict:',type(y_predict))
         11 print('y_predict[0]',type(y_predict[0]),y_predict[0])
    ---> 12 print(classes[y_predict[0]])
         13 print(classes[y_predict[1]])

    TypeError: unhashable type: 'numpy.ndarray'

To give you an idea of what is expected, here is an example that works:
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

y_predict = [1,1,0,0]
print(type(y_predict))
print(type(y_predict[0]))
print(classes[1])
print(classes[y_predict[0]])

<class 'list'>
<class 'int'>
Thunder
Thunder


Multiple filtering pandas columns based on values in another column

Bit late but my preferred solution to this is

# verbetim from @tuomastik

import pandas as pd

df1 = pd.DataFrame({"Random numbers 1": pd.np.random.randn(6),
                "Campaign": ["A"] * 5 + ["B"],
                "Merchant": [1, 1, 1, 2, 3, 1]})

df2 = pd.DataFrame({"Random numbers 2": pd.np.random.randn(6),
                "Campaign": ["A"] * 2 + ["B"] * 2 + ["C"] * 2,
                "Merchant": [1, 2, 1, 2, 1, 2]})

# modification

def pair_columns(df, col1, col2):
   return df[col1] + df[col2]

def paired_mask(df1, df2, col1, col2):
   return pair_columns(df1, col1, col2).isin(pair_columns(df2, col1, col2))

identical = df1.loc[paired_mask(df1, df2, "Campaign", "Merchant")]


How to add a calculated column in a Pandas dataframe?


Metrics values are equal while training and testing a model

This may happen by different reasons:

You're model learned to fast (less than one epoch). In this case, u need to increase dataset (by adding some data or augmentations)
You're loss function doesn't work correctly return zero gradients every call
You 'train' model not in train mode
You data load incorrect. Test what is input for u model


np.c_ converts data type to object. Can I prevent that?

I'd use DataFrame.join() in this case:

data = data.join(pd.SparseDataFrame(Titles_ohe, index=data.index, columns=Titles))


Using layers will allow the dependency to be more reusable and potentially easier to maintain and deploy.

The short version is:

create a requirements.txt from pip freeze or similar that looks like this:

pandas==0.23.4
pytz==2018.7

create get_layer_packages.sh bash script to be run by docker which looks like this:

#!/bin/bash

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

Run the stuff from above like this in the terminal:

chmod +x get_layer_packages.sh
./get_layer_packages.sh
zip -r my-Python36-Pandas23.zip .

I'm not so experienced with python and spent a decent chunk of time messing around with zipping pandas and virtual envs up, and have never really used docker for anything IRL, but this process is actually far more accessible (and better documented) than the venv > zip > upload process I was using before.


When should I use normal Q learning over a DQN?

I haven't worked with reinforcement learning but in general one should judge based on the data (or in this case states). The shift of algorithms from normal ones to deep ones is made when we require more power. So it boils down to your need!
If you can get results with simple algorithms always choose that but if simple algorithms don't cut it you will have to go for the deeper ones.
Not going on the true computational meaning but generally deeper models are more complex and one should always avoid complex models where ever possible.
You also mention "slower". That has a lot of perspective into it so you shouldn't make a final judgement on the models in that aspect!


How do I iterate over my images in dataset?

Your images_dir actually seems to be the path to a single image... but nevertheless,

I would simply create a single number array with shape: (num_images, height, width, channels) by doing the following:

import os
import numpy as np

# Root directory holding all images (I recommend removing the space in "Atom projects")
images_dir = "/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/"

# Number of images you want to load
N= 125

# Get all paths and take the first N
n_image_paths = sorted([f.path for f in os.scandir(data_root)])[:N]

# Load the images using on of the variants of loading images
images = np.array([Image.open(f) for f in n_image_paths])

# Be careful with variants: the order of channels is different for different methods!
# images = np.array([plt.imread(f) for f in n_image_paths])    # Matplotlib
# images = np.array([cv2.imread(f) for f in n_image_paths])    # OpenCV

Now images can be passed directly to your model:

autoencoder.fit(x=images,
                epochs=50, ...)


What is difference between transductive and inductive in GNN?

In inductive learning, during training you are unaware of the nodes used for testing. For the specific inductive dataset here (PPI), the test graphs are disjoint and entirely unseen by the GNN during training.


How to develop Top-N recommendation for evaluating my system

Discounted cumulative gain (DCG) is one of the most common metrics for ranking quality. The gain is accumulated from the top of the result list to the bottom, with the gain of each subsequent result logarithmic discounted.

The DCG accumulated at a particular rank position n is defined as

Since search result lists vary in length, normalized DCG is used. It is normalized by the ideal ranking, sorting all relevant documents in the corpus by their relative relevance and producing the maximum possible DCG:


What is the objective that is optimized with Random Search?

According to the documentation, the function RandomizedSearchCV accepts a scoring string that can take any value from this table and you can even implement your own custom scorer depending on what your goal is.

The default parameter is None in which case it uses the models score function that is defined to:

  Return the mean accuracy on the given test data and labels.


About the time differences in the Bellman equation

During the derivation of the Bellman Equations, when the expected cumulative rewards are calculated on an infinite horizon, meaning $t \to \infty$, shouldn't they be equal for different time steps; like $V^{\pi}(s_i=s) = V^{\pi}(s_{i+\Delta i}=s), \forall s$?

Yes, provided the state $s$ has the Markov property, which means it contains all relevant data about immediate state progression and reward, then this is true. I would read your equations as over-specified in fact, and often see the equations written differently:

$$v_{\pi}(s) = \mathbb{E}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s, A_t \sim \pi]$$

although there are several roughly equivalent ways to describe value functions in RL (for instance, and trivially, you could write $\mathbb{E}_{\pi}$ for the expectation which already implies the condition $A_t \sim \pi$). The one in this answer makes it clear that the value of the state $s$ is independent of the time step on which it is measured.

It is worth noting that although the expectation is always the same and independent of $t$, that actual returns seen can be different, and if you know from some other context that you are seeing the first, or last visit to a state - or from the time step whether this is more or less likely given $\pi$ - then that may change the expectation when loops are possible.


Siamese Network - Sigmoid function to compute similarity score

Your observation is correct @Stefan J, but did you observe that there is a Dense layer too? The absolute L1 distances are multiplied with the weights of the Dense layer. These outputs are then fed to the sigmoid function. If we are looking at a Keras implementation from One Shot Learning with Siamese Networks using Keras,

you see the last Dense layer.
Also, from the research paper,

The L1 distances are multiplied with $\alpha$ which is a training parameter. This sum is then passed on to the sigmoid $\sigma$ function. We may interpret these values of $\alpha$ as the weights of the last Dense layer. These weights get smaller after training.

Another obvious reason of a sigmoid function is to get similarity scores in ( 0, 1 ). The binary cross-entropy loss function is used with it.


Good accuracy on train dataset with cross validation, but low accuracy on test dataset

I think you are using commented lines in the loop to calculate the cross validation accuracy.

# y_predict = clf.predict(X_train)

It should be

 y_predict = clf.predict(X_test)

And then you should check accuracy score with 'y_test' not 'y_train'.

You are predicting on the data KNN has learnt on. And since KNN algorithms kind of learns the data. You are getting good accuracy.


CNNs: understanding feature visualization Channel Objectives (SOLVED)

As you noticed, the activations on mixed4a for a normal sized input is [14,14,512]. Intuitively, these dimensions correspond to [x_position, y_position, channel].

When we talk about "channel" and "neuron" objectives for visualization of channel 11, we mean:

Neuron objective: maximize activations[7, 7, 11]

maximize channel 11 in one position (generally middle)

Channel objective: maximize tf.reduce_sum(activations[:, :, 11])

maximize channel 11 everywhere

This diagram from feature visualization also goes over the difference:


Closed form solution of linear regression via least squares using matrix derivatives

Matrix derivatives work a bit different than regular ones. The scalar parallel of $\nabla_{\theta} \theta^{T} X^{T}X\theta$ you make should be more like $\theta x^2 \theta$ which in turn is just $x^2 \theta^2$. (Note that I changed $\alpha$ for $x$ to avoid confusion with $X$.) You do not need to modify $\theta^{T} X^{T}X\theta$ so that the two $\theta$'s are beside each other, this actually is what you want.

Just like you usually have $\frac{d}{d\theta} (x^2 \theta^2) = 2 x^2 \theta$, then in matrix notation the rule is
$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T}X \theta.$$

Hence, I feel he's not really "skipping steps", but applying a different step than the one you expected.


Follow Up Question - Why use fit when already have fit_transform?

It probably is fairly rare to need to use fit and not instead fit_transform for a sklearn transformer.  It nevertheless makes sense to keep the method separate: fitting a transformer is learning relevant information about the data, while transforming produces an altered dataset.  Fitting still makes sense for sklearn predictors, and only some of those (particularly clusterers and outlier detectors) provide a combined fit_predict.
I can think of at least one instance where a transformer gets fitted but does not (immediately) transform data, but it is internal.  In KBinsDiscretizer, if encode='onehot', then an internal instance of OneHotEncoder is created, and at fit time for the discretizer, the encoder is fitted (to dummy data) just to prepare it to transform future data.  Transforming the data given to KBinsDiscretizer.fit would be wasteful at this point.
Finally, one comment on your post:

we have fit_transform which is much faster than using fit and transform separately

In most (but not all) cases, fit_transform is literally the same as fit(X, y).transform(X), so this should not be faster.


Image Classification using Single Class Dataset using Transfer Learning

You can use Tensorflow Object Detection API.

In the API, you dont have to have 2 or more labels. It can be used with only one label. By transforming images to tfrecords, you can train and predict other images. The prediction will you give you bounding boxes and probabilities that how much an object identified as a vehicle.

The challenge in here is labeling object locations on the images. LabelImg is a great tool for that.

Hope it helps!


I tried training with a 1080 and a 2080 ti and I found that I didn't get any speed up from multi-gpu training because the 1080 acted as a bottleneck. So I while I think this would work fine, you would be better served to run a different model on each GPU rather than trying to train across GPUs.


Proper evaluation method for recommendation system with implicit feedback?

First off, in terms of how to make a train/validation/test split, I would only include users with 2 or more interactions, and for each user, remove the latest interaction. Use this latest interaction as a target and recommend, for each user, their top streamers. You then compare your recommendations with the target, and compute a performance metric. Depending on your performance metric of choice (next paragraph), the number of recommendations you make might differ. This gives you a training/validation split. If you want more training data, and you believe a user's interactions are not sequential in nature (eg. liking StreamerA before StreamerB displays a different preference than vice-versa), you can permute each user into more users by leaving out a different target each time. So for each streamer u, who has n_u interactions, leave out one interaction (target), and keep the rest, and do this n_u times.
Secondly, the appropriate performance metric depends on your use case. If you are going to recommend, say k, streamers for each user, then the performance metric of choice should consider only these k. In other words, for any model performance, rank k+10 is no better than rank k+20. Moreover, maybe it is only of importance that the target is part of the top-k recommendations, then some XYZ@K-metric is most appropriate. Personally, I would probably go with top-k accuracy and average this across all users. This metric then specifies, for all my users, how often, on average, is the target in their top k recommendations.


I think the following issue is related to your question:
https://stats.stackexchange.com/questions/12187/are-decision-trees-almost-always-binary-trees

Let's say you have for each node, 3 children with three different conditions:

n1 -> cond a -> n2
   -> cond b -> n3
   -> cond c -> n4

For binary trees, however, each node holds only one condition and it is evaluated to either True or False, thus meaning you go to the right or the left child. For making the previous simple tree binary, you just need to chain the conditions of a node in the binary tree. For example:

n1 -> cond a -> n2
   -> not cond a -> _n2 -> cond b -> n3
                        -> not cond b -> _n3 -> cond c -> n4


One of the easiest and quick way would be to plot a 'boxplot' where the "dots" above the upper whisker(T shaoed line) of the boxplot denotes an outlier(refer to the attached image). I hope this was what you are searching for. Do let me know if otherwise, may be I can help.


The nth centroid is chosen from a distribution proportional to $D(x)^2$, but pay careful attention to how $D(x)$ is defined.  From the paper (top of page 3):

  In particular, let $D(x)$ denote the shortest distance from a data point to the closest center we have already chosen.

Notice that $D(x)$ is the distance from $x$ to the nearest centroid.  Compare that to the description of how you are choosing the points:

  I repeat this and calculate distances from the red point to all remaining points and take the furthest away, but this puts me back near the green point, but I was trying to get to the center cluster.

Instead of computing the distance between all remaining points and the red point, you should instead compute the distance between each point and its nearest centroid.

For the points in the cluster around (1, 1) you'll compute the distance from the red point, and all the distances will be relatively small.  Similarly, for the points around (-1, -1), you'll compute the distance from the green point, and all these distances will be pretty small.

However, for points around (0, 0), some of them will be closer to the red centroid and some will be closer to the green centroid, but they are not very close to either one.  This means that $D(x)$ will be large for points in the (0, 0) cluster, and you are likely to choose one of these points as the next centroid.

Does that make sense?


Gradient Checking LSTM - how to get change in Cost across timesteps?

Interesting question.

  Like standard backpropagation, [backpropagation through time] consists of a repeated application of the chain rule. The subtlety is that, for recurrent networks, the loss function depends on the activation of the hidden layer not only through its influence on the output layer, but also through its influence on the hidden layer at the next timestep.

It looks like both approaches would have similar results but at a different granularity (adding noise at different levels) this is because the backprop is not really disturbed by this addition as it is still chained from last time step to the first. Hence it boils down to why you actually want to add noise in the first place, as mentioned in this paper , sections IIIC and IV.


Excel file merge with different headers but same data

I suggest SSIS (SQL Server Integration Services) .
It is designed for handling collecting data  from different sources and export  to database.
You can design a simple data flow and run it from SQL serevr on schedule (daily in your case)


Computing Jaccard Similarity between two documents

You forgot a few 2-shingles (bigrams but without duplicates) in the second set but  you got the idea right:

$S_1$ = { "the quick", "quick brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog" }

$S_2$ = { "jeff typed", "typed the", "the quick", "quick brown", "brown dog", "dog jumps", "jumps over", "over the", "the lazy", "lazy fox", "fox by", "by mistake" }

Remark: For this particular example, in each of these two sets every sequence of 2 words appears only once, so there's no need to remove duplicates to obtain the set. In the general case this might be necessary (see the Wikipedia example).

To calculate Jaccard similarity we need to count:

The intersection $S_1 \cap S_2$, i.e. the 2-shingles in common:  | { "the quick", "quick brown", "jumps over", "over the", "the lazy" } | = 5
The union $S_1 \cup S_2$, i.e. all the distinct 2-shingles: | { "the quick", "quick brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog", "jeff typed", "typed the", "brown dog", "dog jumps", "lazy fox", "fox by", "by mistake" } = 15

Jaccard similarity:

$$\frac{S_1 \cap S_2}{S_1 \cup S_2} = \frac{5}{15} = 0.33$$


If you do not specify them, as it is clear in the signature of the functions you are referring to, the function will use the default value for them. For instance, you can see that the default value for stride is (2, 2) which means if you don't define the value of stride the method will use the mentioned value as the default value for the stride. Consequently, if you don't specify it, it does not mean there isn't such thing. In programming this approach helps programmers not to define many different overloads of a typical function.

As the response for the second question, again, if you don't specify the initial weights, TensorFlow itself will use Glorot method for initialization. So, definitely the filters will have values called weights in order to operations like convolution, actually cross correlation, be applicable.

As a recommendation, I highly suggest you taking a look at here and here.


Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

You are absolutely correct that this is an problematic approach. Your testing set should only be used at the last possible stage before deploying a model.
By using your testing set to make modeling decisions you will introduce bias which will favor the observations found in your test set and may not generalize. In an ideal world, your test set would represent the real distribution perfectly, however in practice this is never the case. Thus the suggested approach would result in a model that matches the test set's distribution more closely, but does not generalize to the real distribution.
The correct approach is to separate your data into a

training set,
validation set and,
testing set.

You then set the testing set aside until you have chosen a final model. With the validation set you are very much permitted to do what your professor suggested. You can fit multiple models and then pick the best one, or use all the models together in some bagging structure. Once you are satisfied you can then test against your testing set to see if the selected model generalized well.


What is a good way to transform Cyclic Ordinal attributes?

The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sink with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24).

You need both variables or the proper movement through time is lost.  This is due to the fact that the derivative of either sin or cos changes in time where as the (x,y) position varies smoothly as it travels around the unit circle.

Finally, consider whether it is worthwhile to add a third feature to trace linear time, which can be constructed my hours (or minutes or seconds) from the start of the first record or a Unix time stamp or something similar.  These three features then provide proxies for both the cyclic and linear progression of time e.g. you can pull out cyclic phenomenon like sleep cycles in people's movement and also linear growth like population vs. time.

Hope this helps!

Adding some relevant example code that I generated for another answer:

Example of if being accomplished:

# Enable inline plotting
%matplotlib inline

#Import everything I need...

import numpy as np
import matplotlib as mp

import matplotlib.pyplot as plt
import pandas as pd

# Grab some random times from here: https://www.random.org/clock-times/
# put them into a csv.
from pandas import DataFrame, read_csv
df = read_csv('/Users/angus/Machine_Learning/ipython_notebooks/times.csv',delimiter=':')
df['hourfloat']=df.hour+df.minute/60.0
df['x']=np.sin(2.*np.pi*df.hourfloat/24.)
df['y']=np.cos(2.*np.pi*df.hourfloat/24.)

df

def kmeansshow(k,X):

    from sklearn import cluster
    from matplotlib import pyplot
    import numpy as np

    kmeans = cluster.KMeans(n_clusters=k)
    kmeans.fit(X)

    labels = kmeans.labels_
    centroids = kmeans.cluster_centers_
    #print centroids

    for i in range(k):
        # select only data observations with cluster label == i
        ds = X[np.where(labels==i)]
        # plot the data observations
        pyplot.plot(ds[:,0],ds[:,1],'o')
        # plot the centroids
        lines = pyplot.plot(centroids[i,0],centroids[i,1],'kx')
        # make the centroid x's bigger
        pyplot.setp(lines,ms=15.0)
        pyplot.setp(lines,mew=2.0)
    pyplot.show()
    return centroids

Now lets try it out:

kmeansshow(6,df[['x', 'y']].values)

You can just barely see that there are some after midnight times included with the before midnight green cluster.  Now lets reduce the number of clusters and show that before and after midnight can be connected in a single cluster in more detail:

kmeansshow(3,df[['x', 'y']].values)

See how the blue cluster contains times that are from before and after midnight that are clustered together in the same cluster...

QED!


Implementation of the paper 'Perceptual Generative Adversarial Nets for small object detection'

The generator function is just a CNN that maps an image to a feature map here (well, ROI pooled anyway).
There is nothing specific to it being a GAN; what makes something a GAN is its use as a density estimator and how it is trained (i.e., adversarially), not its architecture. So don't worry about the lack of familiarity with GANs.

In any case, the complex part here, in my opinion, is the presence of the object detection framework parts. To better understand it, though you probably have done so already, I suggest reading the "RCNN trilogy papers": RCNN, Fast RCNN, and Faster RCNN, or at least the middle one, which is relevant here. See also this post and this one about ROI pooling, which takes the dis-similarly sized ROIs and maps each of them to a small fixed-size feature map.

The rest of the architecture are standard convolutional layers mostly, with which you are presumably familiar. It might be useful to review ResNet though, since ResBlocks are used here.
The core ("backbone") of the pipeline relies on the VGG network; its use is explained in more detail in references 24 and 41 of the paper.

Anyway, here's what's happening. Given an image $I$ with proposals $P$ (where $n=|P|$), we first run conv1 on $I$, giving us the low-level feature map $L$. Two things then happen:

On the "top path", $L$ is run through convi for i=2:5, resulting in a new feature map $L_T$, which is then ROI pooled via $P$. This gives us a stack of proposals $S_1,\ldots,S_{n}$, each of which is a featurized image.
On the "bottom path", $L$ is run through two small conv layers and then ROI pooled as well, giving a similar proposal stack $f_1,\ldots,f_{n}$. These are then processed by $B$ standard ResBlocks, to give us $n$ residuals $r_1,\ldots,r_n$.

Finally, we get the ultimate output of the generator, which is a stack of super-resolved proposals given by $$ Y = (y_1,\ldots,y_n) = ( S_1 \oplus r_1,\ldots, S_n \oplus r_n ) $$
where $\oplus$ here just means element-wise sum (i.e., S_k + r_k in most array-based languages).

Basically, the problem is that the ROI pooled features will "look nice" for big objects but terrible for small ones, due to the fewer number of pixels in the latter. The fix for this here is to do "standard" object detection in the top path, but learn a "residue" $r_j$ in the bottom path for each ROI, so that when this residue is added to a small proposal $S_j$, it is given the details normally only present in large proposals. In other words, we are basically doing super-resolution on the small ROIs to help the detector on small objects.

As the paper authors write in the caption:

  The generator is a deep residual network which takes
  the features with fine-grained details from lower-level layer as input and passes them to 3 × 3 convolutional filters followed by 1 × 1
  convolutional filters to increase the feature dimension to be aligned with that of “Conv5”. Then B residual blocks each of which consists
  of convolutional layers followed by batch normalization and ReLU activation are employed to learn the residual representation, which is
  used to enhance the pooled features from “Conv5” for small objects to super-resolved representation through element-wise sum operation.

As well as in the main paper:

  As shown in Figure 3, the generator takes the feature
  from the bottom convolutional layer as the input that preserves many low-level details and is informative for feature
  super-resolution. The resulting feature is first passed into
  the 3 × 3 convolution filters followed by the 1 × 1 convolution filters to increase the feature dimension to be the
  same as that of “Conv5”. Then, B residual blocks with the
  identical layout consisting of two 3×3 convolutional filters
  followed by batch-normalization layer and ReLU activation
  layer are introduced to learn the residual representation between the large and the small objects, as a generative model.
  The learned residual representation is then used to enhance
  the feature pooled from “Conv5” for the small object proposal through RoI pooling [11] by element-wise sum operation, producing super-resolved representation.


Does YOLO give preference to color over shape or vice-versa while detecting an object?

YOLO has 3 input channels (RGB typically) and is a CNN, a neural net that uses stacked (trainable) convolution filters. Single convolution filters act as feature detectors and can be (trained to be) sensitive for straight edges from dark to light or a gradual passage from green to red. By stacking those you can build filters for more complicated features (such as faces).
TL;DR The network can learn to differentiate green and red circles if you want to.


Derivative of a custom loss function with the logistic function

Each parameter will use the partial derivative of Loss w.r.t the parameter i.e.
$\mu = \mu - {\partial L}/{\partial \mu}$
$p = p - {\partial L}/{\partial p}$
$o = o - {\partial L}/{\partial o}$
$u = u - {\partial L}/{\partial u}$
$v = v - {\partial L}/{\partial v}$
If the direct derivative is not possible, then may need the chain-rule.


How to calculate temporary/periodic similarities of an increasing series in real time?

well, This is an interesting question! Let's formulate it again to be precise in answer:

  I have two real-time streams of values. There are temporary similarities (transient similarities) between two. How to capture it?

As you are talking about similarity, I did not limit the scope to correlation. One may define another measure of similarity. I would say you should define your similarity function first (you may choose correlation. I do as well for my example) and a period of inspection (it's simply a window of length $m$ ending at current timestamp).

Using these two and starting from $m+1$th sample, you can calculate your similarity measure (here correlation) as a time-series. This time-series starts after $m$th sample of input series so keep it in mind to keep similarity measure and original timestamps synchronized. The interval between two consecutive points in output (similarity series) is defined based on the resolution you need. after $m+1$th sample you can shift your window on step at time and calculate the similarity accordingly. Then the intervals in output will be as same as input.

Of course it keeps you more away from real-time as the frequency and number of calculations is maximum. You can define a margine for yourself (let's say $d$ sample) and calculate similarity every $d$ time. Then you lose some info (discarding the samples in between) but you get closer to real-time (you will have $d*n$ seconds to perform computation).

Please let me know if it need clarification. Welcome to the community BTW :)


Classification of Colors using Machine Learning from an Array of Pixels

This is a classical problem for a convolutional neural network. Typically, these work with series of layers using 2d-convolution to find features (e.g., "curved" or "yellow"), then pooling (to borrow from adjacent convolutions), then some combination.

I think the main question is how you are planning to get a good, labeled, dataset (that is, one where a banana is already labeled as "banana", for training your NN).


How to train the model with for loop instead of the built-in epochs

You can do it creating a custom training function. I have created a whole set of TensorFlow 2 tutorials about it. It's simpler than it looks like.
This is the code of some generic training function:
import tensorflow as tf

# This loss and optimizer are just examples, use the one you need
loss = tf.keras.losses.MeanSquaredError()
optimizer = tf.keras.optimizers.Adam(learning_rate)

@tf.function
def train_on_batch(X, Y):
    with tf.GradientTape() as tape:
        current_loss = loss(model(X), Y)
    gradients = tape.gradient(current_loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return current_loss

tf.GradientTape() and tape.gradient() record the gradient calculated on the loss function. Then, optimizer.apply_gradients() produces the actual weight update (the training).
Please keep attention at the @tf.function decorator. This is a feature available in TensorFlow 2.x called Autograph. It's a super powerful trick that transforms the whole training function into a TensorFlow op, making it an order of magnitude faster. To make it work, make sure it receives numpy arrays, and that it contains exclusively TensorFlow operations.
Custom training function are extremely useful when you need to implement some non-standard training. For example, I recently trained a GAN for the imputation of missing data, and using custom training made my code easier to implement and control in all its details.
Good luck!


Is "adding the predictions to the real data for new training and prediction" a good idea for LSTM?

Are you saying you would like to predict 10 days ahead in this instance?

If this is the case, the LSTM model is able to do this and iterating in the way you are suggesting is unnecessary and could give you unreliable results.

For instance, consider a dataset whereby we are attempting to predict one-step ahead:

# Training and Test data partition
train_size = int(len(dataset) * 0.8)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t and Y=t+1
previous = 1
X_train, Y_train = create_dataset(train, previous)
X_test, Y_test = create_dataset(test, previous)

# reshape input to be [samples, time steps, features]
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

# Generate LSTM network
model = Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, Y_train, epochs=100, batch_size=1, verbose=2)

# Generate predictions
trainpred = model.predict(X_train)
testpred = model.predict(X_test)

# Convert predictions back to normal values
trainpred = scaler.inverse_transform(trainpred)
Y_train = scaler.inverse_transform([Y_train])
testpred = scaler.inverse_transform(testpred)
Y_test = scaler.inverse_transform([Y_test])

# calculate RMSE
trainScore = math.sqrt(mean_squared_error(Y_train[0], trainpred[:,0]))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(Y_test[0], testpred[:,0]))
print('Test Score: %.2f RMSE' % (testScore))

In this batch of code, you can see that we have set the previous parameter equal to 1, meaning that the time step being considered by the model is t-1.

In this particular instance, here are the training and test predictions compared to the actual series:

Now, the same model is run, but this time the previous parameter is set to 10. In other words, the previous 10 days are being considered as one time step, and the model is forecasting for time t+10 in this instance. Here is another sample prediction comparing the test set with the actual. A fuller example of this is provided here:

In this regard, my advice would be to define the time series you wish to forecast and then work off that basis. Using iterations only complicates the situation, and could even cause issues with prediction.


RNN package and problems with "Predictr"

The manual gives needed dimensions (see help on trainr function):

Dimensions of input X need to be as:

dim 1: samples
dim 2: time
dim 3: variables

and dimensions of output Y should be:

dim 1: samples (must be equal to dim 1 of X)
dim 2: time (must be equal to dim 2 of X)
dim 3: variables (could be 1 or more, if a matrix, will be coerce to array)


Recurrent neural network producing same predictions

Try increasing your batch size. If your batch size is small, the gradients are a rough approximation of the true gradients.


It is not perfect but you can try to re-create synthetic data based on the mean and sd. In R, you can use the rnorm function to create a normal distribution from mean and sd. The following is one of the ways to do it. Hope it helps! P.S. I just choose n = 1000 to illustrate how it can be done; you can try using different numbers.

a <- c(20,30,10,15,15,20,30,10)
b <- c(45,55,10,50,40,40,45,40)
c <- c(25,10,10,30,15,20,20,25)
d <- c(70,60,80,75,80,60,65,70)

n <- 1000

e <- c(rnorm(n, mean(a), sd(a)) +
   rnorm(n, mean(b), sd(b)) +
   rnorm(n, mean(c), sd(c)) +
   rnorm(n, mean(d), sd(d)))

mean(e)
sd(e)

Try it online


Dimensionality reduction and prediction when all columns have approximately same variance

Concerning the dimensionality reduction
I encourage you to check non-linear dimensionality reduction if PCA does not give satisfying results. It can happen that some low dimensionality manifold hides behind a much higher number of features. The excellent sklearn's guide explains manifold learning in detail.
Concerning the algorithm
Almost any algorithm can manage 15 features with similar variance. It is always the case when you standardize your data before feeding the algorithm with it.
You might want to train any model on a sample of the data and check the results. It can highly reduce the computation time without hurting too much the performance. (there is a saturation of performance with respect to data for a given complexity (features + model's parameters)).
Feature Engineering importance
When performance seems to ceil when changing models, you should consider improving your feature engineering. (products of features, polynomial features, ...)


Resampling pandas Dataframe keeping other columns

To add missing indices, use:

full_idx = pd.date_range(start='<start_date>', end='<end_date>', freq='H')

df.reindex(full_idx)

That will put NaN's in SITE_NB and VALUE columns.

If each dataframe has only one value of SITE_NB, you can use:

df['SITE_NB'].fillna(df['SITE_NB'].unique()[0], inplace=True)

which replaces all NaN's with the first non-null values in the column.

If you actually have one dataframe with multiple SITE_NB values, could you please show what that looks like? Are the time indices overlapping in some cases or not?

Also, this answer might work for you: https://stackoverflow.com/questions/32275540/pandas-reindex-dates-in-groupby


Low accuracy in multi-class classification despite all data being generated from rules

I think neural network will be computationally intensive and would require you to have a good GPU along with good amount of training data.
You can try running a clustering algorithm like k-means or logistic regression using warm_start


Feature Importance based on a Logistic Regression Model

No, you do not need to re-scale the coefficients. To the contrary - if they are scaled, you can use them as a way to compare feature importance.

Let's assume that our logistic regression model has coefficients {$ a_i$}, relating to the different (scaled) variables {$x_i$}.
A change of $\Delta x_i $ in the variable $ x_i $  will result in an increase (or decrease, if $a_i$ is negative) of $ a_i \Delta x_i $ in $ log({\hat p_i \over {1-\hat p_i}}) $, i.e. the logit function of $ \hat p_i $, where $ \hat p_i $ is the predicted probability that the i-th example is in the positive class.

So, if the variables are scaled, you can say that if $ a_i$ is larger, then $x_i$ is more important in the model.


What is your keras version? Two things you could try are:

Update your keras to latest version
If the above does not work, open the nasnet.py file in keras, and change this part of the code:

elif default_size == 331:  # large version
    if include_top:
        weight_path = NASNET_LARGE_WEIGHT_PATH
        model_name = 'nasnet_large.h5'
    else:
        weight_path = NASNET_LARGE_WEIGHT_PATH_NO_TOP
        model_name = 'nasnet_large_no_top.h5'

    weights_file = get_file(model_name, weight_path,
                            cache_subdir='models')
    model.load_weights(weights_file)

to this:

elif default_size == 331:  # large version
    if include_top:
        weight_path = NASNET_LARGE_WEIGHT_PATH
        model_name = 'nasnet_large.h5'
    else:
        weight_path = NASNET_LARGE_WEIGHT_PATH_NO_TOP
        model_name = 'nasnet_large_no_top.h5'

    weights_file = get_file(model_name, weight_path,
                            cache_subdir='models')
    model.load_weights(weights_file, by_name = True)


Several options:

You can use the defintion of the derivative to have an approximation....

def f(x):
    return x[0]**2 + 3*x[1]**3

def der(f, x, der_index=[]):
    # der_index: variable w.r.t. get gradient

    epsilon = 2.34E-10
    grads = []

   for idx in der_index:
       x_ = x.copy()
       x_[idx]+=epsilon

       grads.append((f(x_) - f(x))/epsilon)

   return grads

print(der(f, np.array([1.,1.]), der_index=[0, 1]))

If you can solve it analytically, it is better you write the derivative function by yourself
You can also use symbolic programming, like in Matlab, with the library sympy https://towardsdatascience.com/taking-derivatives-in-python-d6229ba72c64
Another way to do it is going for the "differentiable programming" paradigm or "software 2.0"


Choosing a distance metric and measuring similarity

It appears to me that what you're looking for in your use-case is not clustering - it's a distance metric.
When you get a new data point, you want to find the 3-5 most similar data points; there's no need for clustering for it. Calculate the distance from the new data point to each of the 'old' data points, and select the top 3-5.
Now, which distance metric to pick? There are options. If you're using SKLearn, I'd look over this page for example of distance(/similarity) metrics.
If your features are continuous, you can normalize them and use cosine similarity; Start with this, and see if it fits.


Why the performance of VGG-16 is better than Inception V3?

What are the cons?
It has so many weight parameters, the models are very heavy, 550 MB + of weight size.
Which also means long inference time
Why not just make the model deeper?
More heavier model
More training time
Vanishing gradient problem

With a given receptive field(the effective area size of input image on which output depends), multiple stacked smaller size kernel is better than the one with a larger size kernel because multiple non-linear layers increases the depth of the network which enables it to learn more complex features, and that too at a lower cost. 3X3 kernels help in retaining finer level properties of the image.


What is the best way to visualize 10 Fold Cross Validation Scores?

Do you only have one single model? If you were only mixing the data up (and not trying different parameter values) then any plot might not be very really useful. If performance is on the y-axis, what would be on the x-axis? The fold number itself doesn't really give any insight regarding the results (assuming random batch selection).

Are there some statistics you could compute for each of the 10 folds? If there is any metric you can compute for the x-axis, then using a simple line chart would help see how performance varies with that metric. For example, if you have images, you could compute the average pixel value, or the number of true positives/negatives in that batch (if that idea applies to your problem?). An example from Scikit Learn is this, where the gamma parameter of an SVM is altered on the x-axis:

Additionally, if you have several models (e.g. CNNs with varying numbers of layers / depth), you could use a nice box-whisker plot. Seaborn offers some nice variations. Here is one example that would show four different models:

You could make a box plot for a single model too, which wouldn't be so appealing, but might look like this:

Again, I'm not really sure what you would be putting on the axes.


We have recently completed one such exercise using Markov chains. The data that we used for this project was as follows:
For a selected set of users, for a 6 month time period their complete campaign history. Format as follows:
Userid, campaign-id, campaign-date, response tag, response value

The response tag is a binary tag indicating responded or no.

Our markov chain approach did not really care about the time of campaign, it only used the order and getting null state vs success.

What approach are you using in your model? I'd be happy to compare notes. While I'd not be able to share the data, we could run your code on our data.

We had real campaign history data for a client, hence didn't have to simulate it.


Do we need to do multiple times deep learning and average ROC?

No, you don't need to do that. Do probability calibration (temperature scaling) on your trained network. Here a link to do that.


What the article mentions is random tree embeddings.
This is a kind of unsupervised feature extraction method based on random trees.
In python you can find the Scikit-learn implementation.
From the documentation:

Transform your features into a higher dimensional, sparse space. Then train a linear model on these features.

First fit an ensemble of trees (totally random trees, a random forest, or gradient boosted trees) on the training set. Then each leaf of each tree in the ensemble is assigned a fixed arbitrary feature index in a new feature space.

These leaf indices are then encoded in a one-hot fashion.
Each sample goes through the decisions of each tree of the ensemble and ends up in one leaf per tree. The sample is encoded by setting feature values for these leaves to 1 and the other feature values to 0.
The resulting transformer has then learned a supervised, sparse, high-dimensional categorical embedding of the data.

You may also find these references very helpful:

https://gdmarmerola.github.io/forest-embeddings/

https://blog.davidvassallo.me/2019/08/06/3-uses-for-random-decision-trees-forests-you-maybe-didnt-know-about/


Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

Similarity is computed in a vector space model with a primary assumption that the pair of objects for which you want to compute similarity must be represented in the same vector space using some pre-processing operation.

Second dataset is a JSON. You'll need to extract text from it. One way to do it: you can use title and tags strings for each video, concatenate it to a string to find string representation for each video. Use any text vectorization method post that.

So get the video metadata into a text and then you can compute similarity.


If I understand correctly you want to cluster urls by using the keywords extracted as features. As these features are really sparse, you can try to use dimensionality reductions methods to help you.

One way is to treat each URL keywords as a document. Then you can use document embeddings algorithms such as LDA or doc2vec that learn denser representations of your documents.

If you want to group keywords, you can try to use word embeddings methods that learn representations of words. Using this you can then measure the similarity between words and groups of words.
An example is the well-known word2vec. Recent methods like FastText can be alternative that take the morphology into account.


How to keep only significant weights in an ANN

I'll address your last question first:

  is it tested in theory that some weights contribute more than the others?

When learning about NNs I thought it would be better to have a really large number of nodes in the first hidden layer, and then reduce down over subsequent hidden layers.  It is conceivable that many of the weights in this configuration are approaching zero, but this misses two main ideas: the values sent into the activation function are additive, so even small values can still make a difference if the input is large, whereas, 0 times anything is 0; the nodes are not representative or assigned a specific responsibility, so I assume, unless you start with exactly the same initial weights, a node may learn different features every time you train the model.

Subsequently, I have come to the conclusion that, if many of the weights are approaching zero, I probably have too many nodes in my hidden layer.  This does not necessarily take the activation function into account, but it is how I approach model architecture.

Having covered that, you could just convert the matrix to a sparse matrix, sort the sparse matrix based on values and delete everything below your cutoff. This would accomplish what you asked I believe.

However, I would ask why you want to save just three of the weights? It is inconceivable to me how reducing the size of the weight matrices is of any benefit.  If you can get the same performance under this concept, I think you should address your model architecture.  I am not aware of any specific research that would address whether this is an acceptable practice, but it just seems like a waste of effort to train a model and then throw out a majority of the model.  Simplify the model.

HTH


What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I think the two aspects you mention are two faces of the same medal: if your weights are too high or low, the activation of a layer ends up being too high or low. If you use $tanh(z)$ or $sigmoid(z)$ as activation functions, you'll end up with values that will have a derivative almost to zero, and the gradient will vanish as it is back-propagated into the network.
By using $relu(z)$, instead, you use a linear function for $z>0$ and therefore the derivative is constant as the activation grows. This helps the gradient which does not approach zero while it is back-propagated.

This works in every case: both if you use weights that are too high or too small.
Tanh and sigmoid are similar in the sense that they are basically linear near $z=0$, but almost flat when $|z|>>0$, ReLU solves the problem in the latter case, because it becomes linear for any $z>0$.

On the other hand, another strategy is to prevent $z$ from being too large or too small, and therefore you attempt to keep it near the origin ($z=0$). In this way, both tanh and sigmoid are linear, as ReLU would have been for any $z>0$.

The expected result is the same: the derivative of the function is almost linear, and therefore the gradients can traverse the network without vanishing or exploding.

Of course, there are other factors here that must be considered (e.g. ReLU clamps to 0 every negative value, which has other sort of effects), but beside these, in both cases you get a similar outcome (non-vanishing gradients) with different approaches (by tweaking the activation function or by tweaking the weights).


Add new attribute to the input of nerual network

When adding an attribute, increase the number of neurons in the input layer, then train the network again.

Having 100% accuracy may indicate that the model suffers over-fitting. To solve the problem:

1- make sure that the model doesn't see the test set at training time

2- If you have sufficient number of records, I would recommend going for K-folds validations rather than leave-one out cross-validation (for example 70% training 30% testing)

3- if the number of records is insufficient, (such as in your case: 38 records at best won't allow you go deep), either

try to get more more data
or simplify your model (maybe just 1-2 layers with 1-3 neurons!)
or try machine learning methods other than neural networks. plz see (https://www.cs.sfu.ca/~jpei/publications/Sequence%20Classification.pdf).


Basically, it does exactly what you specify. The used numpy function appends values in each dimension. The amount of "pads" on each axis is specified by ((0,0),(2,2),(2,2),(0,0)), given the dimension of your dataset, which is:

10000 (samples) x 28 (image dimension 1) x 28 (image dim. 2) x 1 (grayscale value!)

Now let's see what your specification means in that regard:

(0,0)Pad 0(as in the amount) values before and after each row
(2,2)Pad 2 before and 2 after each value of dim. 1 of your image data: 28 values -> 32
(2,2)Pad 2 before and 2 after each value of dim. 2 of your image data: 28 values -> 32
(0,0)Pad, again, nothing in the grayscale value dimension

That means you will end up with a 32x32 image in the respective dimension. Now, the only thing that's left is: Which values do we pad? The answer is quite simple, you do not specify any constant_values, meaning it will pad with the default constant_values (which is specified on the above linked page). Namely this value is 0.

To sum it up, simply imagine you have a 32x32 image, your 28x28 is in the middle, and on the outside you have a 2-value-thick border of 0's.


ML on dataset with heavy dependency on prefix

Consider LSTMs. They're a type of recurrent neural network that are generally very good at sequence prediction tasks and have the ability to learn very long range dependencies like you describe.

Andrej Karpathy gives a tremendously great intro to RNN/LSTM on his blog.

Although going as far as a neural network opens up a whole new range of problems, such as long training time and finding the right architecture and gradient descent parameters. If your problem is very dependent on only a handful of terms then you may be able to get adequate results with a tabular multinomial regression approach and some smart encoding of your sequence. Something like:

y prefix1 prefix2 prefix3 has1 has2 has3  ...
A       1       2       0    1    1    0  ...
B       1       3      -5    1    0    1  ...


2 solutions:

You aggregate each sequence of numbers into a single number, which use as a feature. There exist plenty of aggregation functions, such as some derived from descriptive statistics root-mean-square, kurtosis, skewness, max, min, duration, standard deviation, crest factor, mean, or more specific aggregation such as fourier transforms or wavelet transforms.
You use some model that accepts sequences as input. Sequences may be of variable length. Example of such model: recurrent neural networks, Dynamic Bayesian networks.


( All emails are or in french or in english )

Pre-processing methods :

Merge " Summary " and " Content " which are the head of the email and the content
Remove all emails adress
Remove all " From : Someone To : someone ... subject : something "
Remove all images included in the email
Order the email according to their class
Replace all french accent by no accent like é -> e ; ê -> e ...
Put text to lower
remove punctuation
remove numbers
strip white space
remove some first name from a list and some specific words
remove stop words in french and english
stem document in french and english
Remove term

--> Then Document term matrix, with TF-IDF

The two classes are from a technical support, two categories "complex" and "easy", "complex" are topics around finance ( in theory ) , "simple" problem with the software ( in theory ) but in practice they have a lot words in common.
And the other classes I don't take them into account I just focus on the two for now

Actually the kind of classification algorithm I used is not so relevant because I tried with 5 algo and none gave good results

Example ( confusion matrix )
Decision tree:

  Decision tree:

pred:

335 | 10
59  | 12

SVM:

331  | 1
83   | 1

Knn (n=10):

330  | 2

83   | 1

Naive Bayes:

1   |  83

12   |  320


You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings.

Almost all of them are unsupervised approaches that require no labels to detect the anomalies.

They also automatically handle some of the issues you mentioned, like seasonality.


My understanding is that mini-batches are not really for speeding up the calculations... but to actually allow large datasets to be calculated.

If you have 1,000,000 examples, it would be tricky for a computer to compute forward and backward passes, but passing batches of 5,000 elements would be more feasible.

For your case, I recommend you two things

Try different batch sizes.
Make sure you shuffle your batches!!! that will certainly help you a bit.