You're probably aiming too high: a research topic doesn't have to lead to a major breakthrough, and very often it's impossible to know what it leads to before doing it. A Master thesis is not very long so you need to find a research topic which is feasible within the time frame. A common mistake by Master students is to spend too much time on implementation, then botching the analysis and/or writing a poor quality dissertation because there's not enough time left.

Discuss with your advisor, they have the experience and they know what is a good topic.
Read a few random papers in your domain. Why? Because if you read only the top quality papers (yes, of course these are important to read), you can get the impression that you need to achieve the same level of quality. But that's not the case: these top quality papers are 1 in a million and are often done by people who have years of experience behind them. A "regular" research contribution is often just a small improvement or a decent analysis on a specific problem.
Typical process: just pick a general topic/problem you're interested in, then study the state of the art thoroughly (write your state of the art section at this stage), reproduce a few experiments and analyze the results: what can be observed? what are the limitations? what would be a reasonable idea to overcome one of these limitations? Then you implement and evaluate this idea.

Keep in mind that in research what matters is not the complexity of the topic or even the performance, it's the rationale and the method (and of course originality of the work, but even that doesn't matter that much at Master level).


DistinctCount for relatedtable with additional filter

I would wrap your expression in a CALCULATE function, so something like:

activedays = CALCULATE (
    COUNTAX ( RELATEDTABLE ( ActivesInactiveData ) , ActivesInactiveData[Surface Date] )
    , ISBLANK ( ActivesInactiveData[Topic] ) = FALSE )

I assumed [Topic] is text, so NULL will converted by DAX to BLANK().


Coefficient of determination is close to 1 but the value of RMSE is large. What does it mean?

Compared to $R^2$ $RMSE$ is dependent on the variance of values. Even if $R^2$ is close to 1 but the standard deviation is high the value of $RMSE$ will also be high.
$$RMSE=\sigma_{y_{true}}\sqrt{1-R^2}$$
where $\sigma$ is a standard deviation

As can be seen from the formula $RMSE$ and $R^2$ have a strict mathematical relationship that shows that $RMSE$ changes according to standard deviation ($\sigma$) and the higher the standard deviation of the true values the higher the value of $RMSE$.

$RMSE$ shows the deviation of the error in units of the values, while $R^2$ shows the share of variance that is explained by the model. You can explain 99% of the variance ($R^2$) but your numbers are in millions of dollars and your error still varies in thousands of dollars ($RMSE$) Whether thousands of dollars is ok for your error or not depends on the case. If you predict profit plus or minus 1000\$ is ok. If you predict revenue then plus or minus 1000\$ could be a difference between a profitable and a non-profitable company.

Proof of the formula:

According to sklearn the formula for $R^2$ is

$$R^2 = 1 - \frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}$$

RMSE is
\begin{align}
RMSE &= {\sqrt{\frac1n\sum_i^n(y_{true} - y_{pred})^2}}
\\RMSE^2 &=\frac1n\sum_i^n(y_{true} - y_{pred})^2
\\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\frac1n\sum_i^n(y_{true} - y_{pred})^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=1-\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=R^2\\1-\frac{RMSE^2}{\sigma_{y_{true}}^2} &=R^2
\\RMSE&=\sigma_{y_{true}}\sqrt{1-R^2}
\end{align}

where $\sigma$ is a standard deviation and $\overline{y}$ is an average


How to train the same RNN over multiple series?

just re-use

  model.fit()

on the fresh datasets using the already trained model, simple as that :) !

(given that you do it in Keras)


quest for online tutorials on CNN backprop

Here is one good material: https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199

This one also might help: https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/convolution_layer.html (it also has other useful materials).


How to read html tables under multiple headers and combine them in a single pandas dataframe?

With a bit of parsing via BeautifulSoup, we can get a pandas.Dataframe using pandas.read_html() like:

Code:

def get_tables(source):
    elems = iter(BeautifulSoup(source, 'lxml').find_all(['table', 'h1']))
    df = pd.DataFrame(
        pd.read_html(str(next(elems)), header=0)[0].iloc[0].rename(h1.text)
        for h1 in elems)
    df.index.names = ['Caption']
    return df

Test Code:

import pandas as pd
from bs4 import BeautifulSoup

with open('test.html', 'r') as f:
    print(get_tables(f))

Results:

         Col1  Col2  Col3
Caption
Group 1  ValA  ValB  ValC
Group 2  ValP  ValQ  ValR


What is the correct way to compute Mean F1 score?

This paper Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement by Forman and Scholz discuss the different methods for computing the average F-score in cross validation. The paper shows that under very high class imbalance some of the computation methods (average of individual folds F-scores or F-score based on average of individual folds precision and recall) can lead to biased results. The paper recommends computing the F-score by adding the TP, FP, FN from each fold, computing the precision and recall and finally the F-score.


Keras: Misunderstanding the Conv2D's param "filters"?

filters are the numbers of kernels or feature detectors that we choose for the convolutional layer to learn.
in the end, the number of feature maps that we get equals to this number of filters that we have declared here.
Now, what are these feature detectors? these are matrixes usually smaller (much smaller) than the input images. It's occasionally $3by3$, $5 by 5$ or like the one in Alexnet $7by7$.
In the convolution layer, a convolution operation is done using each of these filters, each time we take a filter and our input image, then we put the filter on the left top corner of the input image and multiply each matching cells, then we would sum up the results and put that in the very first cell of our output (feature map) matrix.
Then we move this filter on the image and do the same,... till we fill the whole feature map (stride defines how many cells we move in the image). Once we have our feature map number one that is a result of applying filter number one on the input image, we take another filter and do the same and so on and so forth.
Obviously, the outputted feature maps are smaller than the input, the amount by with the input size is reduced depends on the stride that we choose, moreover, we have kinda aggregated features in our feature map (also called convolution/convolved map)

Why is it possible to indicate the number of features maps (results of the scalar product between the kernel and the input), since we specify the kernel's size and the input?
Indeed, if we know the kernel's size (width and height) and the input's size (width and height), thus we know the number of features maps.

The thing is if we just know the input size and the size of the kernel, we don't know the stride, then setting a stride of 2 or 1 or anything else would give us different results. If you assume your stride to be a specific number, then you would know how many filters should be used.


Classifying objects in video without machine learning

You could try SIFT (Scale-invariant feature transform) to recognize the labels on the bottles. Althoug you need some preprocessing, just for comparing the labels on the bottle and to distinguish the different brands it should work.
Also it could get kind of complex (if you show the bottles from behind and the brandlogo is not visible). So you maybe have to extract features from many perspectives of the bottle.

Here is one example: https://towardsdatascience.com/bibirra-beer-label-recognition-8546c233d6f4


How to speedup K-Means used in 'for loop'

Using the multiprocessing library I was able to parallelize the loop and speedup execution.


How to approach All vs All classification problem

I think you can carry out a usual multiclass classification instead of manually carrying out the one-VS-all strategy (i.e. in the for loop), provided that you can generate the multilabels target(i.e. all the possible combinations of diseases you said). So, I would do something like:

generate the correct labels --> if you have n possible deseases combinations, you should have in an end n target labels (so using the apply-lambda expression generates n binary columns for you, which you can also convert to a single numeric column with n possible integer values).
Below you can find a way to get your final column of target values (an integer value for each possible combination); using the wines dataset as an example where I also consider all possible combinations (here a flavour type is similar to a possible disease in your case). I guess you have a dataframe like this:

so you can create a dict mapping the possible combinations to an integer target value (for the multiclass classification):
target_labels_dict = {tuple([1, 0, 0]): 0, tuple([0, 1, 0]): 1, tuple([0, 0, 1]): 2, tuple([1, 1, 0]): 3, tuple([1, 0, 1]): 4, tuple([0, 1, 1]): 5, tuple([1, 1, 1]): 6}

and you have this:

where for instance the tuple (1, 1, 0) means 'acid' and 'sweet'...
and finally, create your target column:
target_flavours_array = np.zeros(len(y_train)).astype('int')

for ind in y_train_df.index:
   combination_values = tuple(y_train_df.iloc[ind])
   target_flavours_array[ind] = target_labels_dict[combination_values]

getting something like:

use the softmax (or its analogous sofprob function check it out here) with the xgboost instance --> below a quick example of this (training on the original 3 classes dataset for the shake of the example):

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
import numpy as np
import xgboost as xgb

dataset = load_wine()
X_train, X_test, y_train, y_test = train_test_split(dataset.data,
dataset.target, test_size=0.1, random_state=42)

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {
    'max_depth': 4,  # maximum depth of each tree
    'eta': 0.2,  # learning rate
    'objective': 'multi:softprob',  # for multiclass problem, providing probabilities
    'num_class': len(np.unique(dataset.target))} # number of classes

iters_number = 30
xgb_model = xgb.train(params, dtrain, iters_number)
xgb_model.predict(dtest)

and, the result is the prediction probability for each possible class (diseases-combination in your use case):

here you can select the highest probability value for each row.


Why the my Tensorflow code just use one GPU when I assign more than one

If a TensorFlow operation has both CPU and GPU implementations, TensorFlow will automatically place the operation to run on a GPU device first. If you have more than one GPU, the GPU with the lowest ID will be selected by default.
TensorFlow 2.0 now has the tf.distribute module to distribute training across multiple GPUs, multiple machines or TPUs. It builds on the concept of "distribution strategies". You can use tf.distribute.MirroredStrategy() as a scope, like
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    #DO WHATEVER YOU WANT TO DO HERE

Please check this Distributed training with TensorFlow guide for implementation details and other strategies.


What is LSTM, BiLSTM and when to use them?

RNN architectures like LSTM and BiLSTM are used in occasions where the learning problem is sequential, e.g. you have a video and you want to know what is that all about or you want an agent to read a line of document for you which is an image of text and is not in text format. I highly encourage you take a look at here.

LSTMs and their bidirectional variants are popular because they have tried to learn how and when to forget and when not to using gates in their architecture. In previous RNN architectures, vanishing gradients was a big problem and caused those nets not to learn so much.

Using Bidirectional LSTMs, you feed the learning algorithm with the original data once from beginning to the end and once from end to beginning. There are debates here but it usually learns faster than one-directional approach although it depends on the task.

Yes, you can use them in unsupervised learning too depending on your task. take a look at here and here.


Understanding the generality of the NER problem

The problem described is not a more general version of Named Entity Recognition, it is a different problem called parsing. Parsing consists in extracting the syntactic structure of a text, usually in the goal to better capture its semantics. There are various approaches:

Shallow parsing only identifies the constituents of the sentences (based on your example this could be sufficient in your case)
Statistical parsing and in particular Dependency parsing represent the full structure of the sentence, including the links between its constituents.

There are various libraries and datasets for parsing: one of the most famous is probably the Stanford parser, but there are many others often included in NLP toolkits such as OpenNLP. The Universal Dependencies project is a vast multilingual collection of annotated text which can be used to train parsers.
Semantic Role Labeling (SRL) is a closely related task which consists in identifying the semantic relations between a predicate (verb) and its related constituents (e.g. subject, object).


training gradient boosting algorithm in python testing in Golang

There's actually a few libraries that handle the inference part well.  https://github.com/dmitryikh/leaves is probably the most common one and seems to fit your need.


Date transformation for KNN to get distance

What do you consider a meaningful distance? One possibility would be days since some reference date.

If seasonality also matters, you could additionally (or also) have calendar date or some sine or cosine functions with period 1 year/n for integer n - the idea being to get a periodic spline. The same kind of idea (or flagging weekday vs. weekend etc.) could apply to day of the week, if that makes sense for your problem.


Advanced Activation Layers in Deep Neural Networks

ReLU
Simply rectifies the input, meaning positive inputs are retained but negatives give an output of zero. (Hahnloser et al. 2010)
$$
f(x) = max(0,x)
$$
Pros:

Eliminates the vanishing/exploding gradient problem. (true for all following as well)
Sparse activation. (true for all following as well)
Noise-robust deactivation state (i.e. does not attempt to encode the degree of absence).

Cons:

Dying ReLU problem (many neurons end up in a state where they are inactive for most or all inputs).
Not differentiable. (true for all following as well)
No negative values means mean unit activation is often far from zero. This slows down learning.

Leaky ReLUs
Adds a small coefficient ($<1$) for negative values. (Maas, Hannun, & Ng 2013)
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
0.1 x & \text{otherwise}
\end{cases}
$$
Pros:

Alleviates dying ReLU problem. (true for all following)
Negative activations push mean unit activation closer to zero and thus speeds up learning. (true for all following)

Cons:

Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).

PReLUs
Just like Leaky ReLUs but with a learnable coefficient. (Note that in the below equation a different $a$ can be learned for different channels.) (He et al. 2015)
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
a x & \text{otherwise}
\end{cases}
$$
Pros:

Improved performance (lower error rate on benchmark tasks) compared to Leaky ReLUs.

Cons:

Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).

ELUs
$$
f(x) = \begin{cases}
x & \text{if } x \geq 0 \\
\alpha(exp(x)-1) & \text{otherwise}
\end{cases}
$$
Replaces the small linear gradient of Leaky ReLUs and PReLUs with a vanishing gradient. (Clevert, Unterthiner, Hochreiter 2016)
Pros:

Improved performance (lower error and faster learning) compared to ReLUs.
Deactivation state is noise-robust.


Although there is no data provided I'll try to share my ideas.
Stacking
We stack models by making several predictions with hold-out data sets and then collecting these predictions to form a new data set, where you can fit a new model on it
Often times the stacked model (also called 2nd-level model) will outperform each of the individual models due its smoothing nature and ability to highlight each base model where it performs best and discredit each base model where it performs poorly
Model diversity
But one of the most important thong in stacking is model diversity, how different a model is to each other. You should consider what is the new information each model brings into the table?
It will find why one of your model is good, and when another one is bad or fairly weak.
You don't need to worry too much to make all the models really strong.
Therefore, what you really need to focus is what information does this model brings, even though it is generally weak? Such models bring in new information that the meta model could leverage.
Normally, you introduce diversity from two forms:

By choosing a different algorithm or by . Which makes sense, certain
algorithms capitalize on different relationships within the data.
For example, a linear model will focus on a linear relationship, a
non-linear model can capture better a non-linear relationships. So
predictions may come a bit different

Running the same model, but you try to run it on different
transformation of input data, either less features or completely
different transformation. For example, in one data set you may treat
categorical features as one-hot encoding. In another, you may just
use label encoding, and the result will probably produce a model
that is very different.

So you need to test different models. As @Aditya said - the improvement isn't guaranteed either ways until and unless you have strong baselines


How to equalize the pairwise affinity perplexities when implementing t-SNE?

It simply means that you should set the bandwidths through binary search. The way it works is that you start with a preset target perplexity (Mark's link suggests values from 5 to 50 as reasonable values), and bounds for the bandwidth. If the target perplexity is inside the interval defined by the boundary perplexities, you iteratively halve the search space until you converge to the target:

$$2^{H(p; \sigma_L)} < PP_\mathrm{target} < 2^{H(p; \sigma_U)}$$

If the target was not in the initial interval, you expand the interval and try again.


Here is a simple way of fitting a linear model to each salesperson by week, month and quarter. You'll need to extract the week, month and quarter from the date using the lubridate package.

library(lubridate)
library(tidyverse)

#create fake data
n_var <- 50
id <- sample(x=seq(from=1,to=4),size=n_var,replace=TRUE)
dte <- sample(seq(as.Date('2018/01/01'), as.Date('2019/01/01'), by="day"), size=n_var)
sales <- runif(min=0,max=25,n=n_var)

sales_data <- data.frame(id,dte,sales) %>%
  mutate(
    weeknumber=week(dte),
    month=month(dte),
    qtr=quarter(dte)

  )

#fit an OLS model
ols_model <- sales_data %>%
  group_by(id) %>%
  summarise(
    week_model=list(lm(sales~weeknumber)),
    month_model=list(lm(sales~month)),
    qtr_model=list(lm(sales~qtr))
    )

#extract coefficients
ols_model$week_model[[1]]$coefficients
ols_model$month_model[[1]]$coefficients
ols_model$qtr_model[[1]]$coefficients

ols_model$week_model[[2]]$coefficients
ols_model$month_model[[2]]$coefficients
ols_model$qtr_model[[3]]$coefficients

#simple version on one id
df <- sales_data %>%
  filter(id==1)

mod <- lm(sales~qtr,data=df)
mod$coefficients


Overlaying a line graph and an area graph: adding recession bars to a time series

Using the data from FRED for both labour data and recession data and slight adjustments to your code I think I get the result you want:
import pandas as pd
import matplotlib.pyplot as plt

# read in data
rec_data = pd.read_csv("USREC.csv")
ls_data = pd.read_csv("LABSHPUSA156NRUG.csv")

# make sure date columns are actual dates
rec_data["DATE"] = pd.to_datetime(rec_data["DATE"])
ls_data["DATE"] = pd.to_datetime(ls_data["DATE"])

# create plot
fig, ax = plt.subplots()
ls_data.plot.line(ax=ax, figsize=(8, 5), x='DATE', color="blue")
rec_data.plot.area(ax=ax, figsize=(8, 5), x='DATE', alpha=0.5, color="gray")
plt.xlim("1950-01-01", "2020-01-01")
plt.ylim(0.58, 0.65)


In absence of class labels, your best choices are either an unsupervised machine learning approach (e.g., k-means clustering), or leverage descriptive statistics to see whether you can find any pattern in the data have. For example, looking at the distribution of your features may reveal outliers that could be associated with your outcome of interest.


All it does is make a difference in the step size of the learning step. Scaling the outputs down makes the gradients smaller, so the gradient descent updates are smaller. We want that so we do not jump over a good solution with bigger steps.

Let's say the activation function is $a =\sigma(x)$ where $x$ is the input to the activation function and $a$ is the output.
The range of true outputs $y$ is on the order of 10 times of the range of $\sigma$. We can either scale $\sigma$ up, or scale $y$ down.
Scaling activation function
Lets scale the output by 10: $y' = 10a$.
The loss function $L$ compares predictions vs. truth: $L(y', y)$. We need the gradient w.r.t $x$ for backpropagation. That is, to find how the loss changes with respect to input so we can change the input:
$$
\frac{\partial L(y', y)}{\partial x} =
\frac{\partial L(y', y)}{\partial y'} \times \frac{\partial y'}{\partial a} \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(y', y)}{\partial y'} \times 10 \times \frac{\partial a}{\partial x} \\ =
\frac{\partial L(10a, y)}{10\cdot \partial a} \times 10 \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(10a, y)}{\partial a} \times \frac{\partial a}{\partial x} \\=
\frac{\partial L(10 \cdot (a, 0.1y))}{\partial a} \times \frac{\partial a}{\partial x}
$$
Scaling output
On the other hand, lets scale down to data, so $y_s = 0.1y$. This means we do not need to scale $a$. The loss function gradient now is:
$$
\frac{\partial L(a, y_s)}{\partial x} = \frac{\partial L(a, 0.1y)}{\partial a} \times \frac{\partial a}{\partial x}
$$

Now for both cases, note the final forms of the gradient. The only difference is the arguments of $L$. For the case where the output was scaled, the arguments of the loss function are 10 times smaller. Which means that the gradient will be smaller. Which means that the step update made to $x$ will be smaller. We usually want small updates so we can converge to an optimal solution.
But also note, we can make the step size smaller anyways by reducing the learning rate too.
So scaling the output down instead of the activation up is a nice rule of thumb to get better convergence. It is not a rule.


Reason for capping Learning Rate (alpha) up to 1 for Gradient Descent

Setting a hard cap on the learning rate, for example at alpha = 1, is certainly not mandatory. It is also not necessarily advisable to set such a cap, as the merits of using different values for the learning rate are highly dependent on the exact function upon which you are performing gradient descent, what you hope to achieve in doing so, and what measures you will use to measure the relative success of one value choice over another.

I think the information you provided demonstrates this concept well. For example, if all you care about is moving towards some local minimum of your cost function, ultimately finding parameters for your model that achieve a cost less than say .01, and all else being equal accomplishing these tasks in the least number of iterations possible, we can see that among the values you tried alpha = 1.2 is indeed the best value (among the runs you showed us, it reached the cost of .01 in the least number of iterations). However, many people care about other properties of their gradient descent algorithms. For example, one may prefer a learning rate which is more likely to arrive at whichever (if any) local minima is nearest to the initialized parameters; lower learning rates seem better suited for this goal, since a high learning rate has a higher potential of 'overshooting' one minimum and landing in the basin of another. Or one may prefer a learning rate which produces a very smooth looking cost over time graph; lower learning rates seem better suited for this goal too (for an anecdotal example, your alpha = .03 learning curve looks smoothest).

There are many resources and methods available for choosing "ideal" learning rates and schedules out there, and I think it is worthwhile to read up on them to get a flavor for what people typically do. Most suggestions are heuristic, and not guaranteed to be meaningful in any particular example. Setting a cap of alpha = 1 is one such heuristic, and is probably suggested because it has been useful for many people with a lot of experience. Since many people have devoted significant time to studying this question, I don't think it is necessarily a bad idea to postpone thinking too hard on the topic when one first uses gradient descent, and instead just use the defaults in things such as scikit-learn's implementations, or take suggestions such as never setting alpha larger than 1. Personally, though, I share your desire to not blindly use defaults when I have the time to think on alternatives, and think it would be informative (if potentially not useful) to spend time investigating exactly how learning rate choices affect the goals you have in your gradient descent implementation.


What does the output of model.predict function from Keras mean?

The output of a neural network will never, by default, be binary - i.e. zeros or ones. The network works with continuous values (not discrete) in order to optimise the loss more freely in the framework of gradient descent.

Have a look here at a similar question that also shows some code.

Without any kind of tweaking and scaling, the output of your network is likely to fall somewhere in the range of your input, in terms of its nominal value. In your case, that seems to be roughly between 0 and 2.

You could now write a function that turns your values above into 0 or 1, based on some threshold. For example, scale the values to be in the range [0, 1], then if the value is below 0.5, return 0, if above 0.5, return 1.


It's is a good practice to use batches to train neural networks. As Yann LeCun said:

  Training with large minibatches is bad for your health.  More
  importantly, it's bad for your test error.  Friends dont let friends
  use minibatches larger than 32.

Although, it will not help you deal with large images. This is what convolutions are for.

If you are using Keras, then the batch implementation is entirely made for you. If you use Tensorflow, then you can use the tf.data api or do your own implementation. The code you provided seem to work for your particular training size, but you may want to adapt it for all kinds of training size (namely using num_iterations = training_size // batch_size) and making sure that the case where not all examples where shown to the network, the last remaining examples are included as well before ending the epoch...

I have had good results with the Adam Optimizer, although you might want to tweak the hyperparameters to get better results.


how to make new class from the test data

I think you need to read about Online learning, it refers to learning when new data is being constantly added. In these cases you need an algorithm that can update itself as new data arrives (i.e. it doesn't need to recalculate itself from scratch). In other words, incrementally.

There are incremental versions for support vector machines (SVMs) and for neural networks. Also, bayesian networks can be made to work incrementally.


Time series data: How I measure influence of new product sales on existing product sales (statistically)?

You could build an ARIMAX model. This would permit to include autoregressive (AR) terms as well as well as the sales in product 5 as an Exogenous Input (X). This would give you a potential model where the sales for a product $i$ at time $t$ is given by $s^i_t$ then,

$s^1_t=\alpha_1 s^1_{t-1} + \alpha_2 s^1_{t-2} + \ldots + \beta_0 s^5_t + \beta_1 s^5_{t-1} + \ldots $

Note that you may need to make the series stationary first, but see more on that below. You could estimate this model with the seasonal R package that relies on the X-13ARIMA-SEATS software developed by the US Census Bureau.

I would recommend to ensure that your time series are all stationary, see for example this post before you use X13. I would also run cointegration tests. See for more explanation this excellent post.

Since you only have data on week 5 I would start modeling in week 5 but you could include autoregressive (AR) terms related to the sales of product 1 prior to week 5.


The solution to the above problem was to use XGBRegressor instead of XGBClassifier. Just swapping it in seems to have worked.


I too would fall back on parsing either the HTML or the entities using regular expressions. My experience is though that this always gets unelegant quickly.

Do you have a somewhat clear idea of the relevant sources? If the
better part of the relevant data comes from a limited number of
pages, you could maintain a list of sources with matching wrappers.
Then within those relevant documents, I would search for the least
complex most valuable features to extract.

Example

For instance, if you'd be interested in the quarterlies of Alphabet, I would scrape this link. You're smart, you can figure out the next one.

A quick glance learns me that the first hit on revenue(s)  $ returns me the revenue for the quarter.

So something like this:

(?:revenue[s]?)(?:\s[\w]+\s)(\$[\d]+\.?\d\s[\w]+)

Testing that one the reports on the site seems to work on q1, q2 and q3 while q4 yields the annual revenue. Easy enough to fix.

My experience is that thee patterns hold for a while, and then change. No big deal, just add a couple of tests! Fi: Is the result not empty and is it in a believable range?


XGboost classification with very small data set

You can use grid search or xgb.cv for find the best iteration. Run xgb.cv for example 500 trees and add early stopping criteria. Then you can use the best iteration for xgb.train. Have a look at these links.

Hypertuning XGBoost parameters

https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees


A neural network is in principle a good choice when you have A LOT of similar data and classification tasks. Predicting the next character (or word... which is just multiple characters) is such a szenario. I don't think it really matters which kind of language you have, as long as you have enough training data of the same kind.

See The Unreasonable Effectiveness of Recurrent Neural Networks for a nice article where a recurrent neural network (RNN) was used as a character predictor to write complete texts. They also have code on github.com/karpathy/char-rnn ready to train / go. You can feed it with a start string and ask for the next characters / words.


How to visualize categorical data with numeric columns

Clearly there's no way to have the names of the drugs.
Assuming the relation between the two columns is important, a scatter plot with units prescribed as X and number of patients as Y might work. You could even add the name of the drug for a few isolated points. Transparency/opacity can be used to show the dense areas.
In case the relation between the columns is not important, you could just plot the two distributions (histograms) with different colours on the same graph.


Implementing the SVHN CNN architecture in Srivastava et al. 2014 Dropout paper

This means that your input shape was too small by the time it reached the last Conv2D layer.

By removing the last Conv2D and MaxPool2D layer, this is how the model looks like:

You can see that the last output before the Flatten layer has height and width of 4, which is smaller than your kernel_size of 5 in the last Conv2D layer.

Heres what you can do:

Increase the input image height and width OR
Set padding='same' in the Conv2D layers


Why are convolutions still used in some Transformer networks for speech enhancement?

We find some justifications in the Conformer paper:

Convolutions are better than Transformers at detecting fine-grained patterns:

While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are
used as the de-facto computational block in vision.

Together, they Transformers and convolutions work better than separately:

Recent works have shown that combining convolution and self-attention improves over using them individually [14]. Together, they are able to learn both position-wise local features, and use content-based global interactions.


Keep in mind that the DQN based method have replay memory that is sampled and used to optimize the model at every time step while A3C is only optimized when the episode finishes. So for 4000 timesteps, and 1000 timestep per episode, A3C optimized 4 times while DQN could have optimized 3000+ times depending on the implementation. This may explain why given the same number of timesteps, DQN is learning faster.


Can GPS coordinates (latitude and longitude) be used as features in a linear model?

You cannot use them directly, as it is unlikely there is a true linear relationship unless you're looking to predict "how far east or north" someone is. As mentioned in the comments, you need to convert them into zones. If you wanted to keep it really simple, you could use a kNN clustering algorithm with a low number of potential clusters and then assign each instance a new feature with the cluster ID, and then one-hot encode that.

You may also want to read about how people interpolate coordinates to predict values across a whole map. The first example is with temperature stations, but you can also imagine it being "hot zones" for crime.

(DOCS)


How should I collect data for an "identifier" rather than classifier?

It depends on the use you are giving. The $not.lion$ class should contain sufficient data of different things, but also should contain tigers, cats, african landscapes without lions, etc.

Your idea of 50-50 is good.

Add this: The weights $w$ could help you define where your model should focus: Lesser weights when there is something easily distinguishable from a lion like a computer, but larger weights when there is an african landscape.


Logistic Regression : Solving the cross-entropy cost function analytically

If you use just one neuron (linear->sigmoid), you can find the minimum with too much small error or maybe reach to the minimum using approaches like gradient descent or other optimization algorithms. The reason for finding the minimum is that cross entropy is a non-convex shape, but if you use sigmoid function as the activation function of logistic regression, the cross entropy cost function becomes convex and it is easy to find the only global minimum.


How to clone Python working environment on another machine?

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.

As for the question itself - you can export your Anaconda environment using:

conda env export > environment.yml

And recreate it using:

conda env create -f environment.yml

Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.

To create a virtual environment in Anaconda you can use:

conda create -n yourenvname python=x.x anaconda

which you activate using:

source activate yourenvname


The defaultdict in this case is meant to return the next increasing integer each time a new element is encountered in P.flat().  But to also return the same integer each time that same element is found again. You should have encountered the error:

  TypeError: first argument must be callable or None

defaultdict expects a callable. It will call the passed in function each time there is a lookup miss, and set the dict to that value. To fix that error you need to remove the () from the __next__() to leave:

mis = defaultdict(range(1, P.max() + 1).__iter__().__next__)

However this is overly complicated for what it is doing, and can be simplified a bit by recognizing that there is no reason to have the upper limit in the range, it is just an increasing integer which can be done with itertools.count() liike:

import itertools as it
mis = defaultdict(it.count(1).__next__)


How do I know the best pruning criteria for decision trees?

Experimentally: using cross-validation on a subset of your training data, compute the performance of every option that you want to consider. Then select the best option and train the final model using this option.

// different settings for hyper-parameters,
// for instance different pruning criteria:
hpSet = { hp1, hp2, ...}

trainSet, testSet = split(data)

for each hp in hpSet:
    // run cross-validation over 'train' using hyper-parameter 'hp'
    // and store resulting performance
    perf[hp] = runCV(k, trainSet, hp)

bestHP = pick maximum hp in 'perf'
model = train(trainSet, bestHP)
perf = test(model, testSet)


extract document topic vectors from lda model

I've done this before in Gensim, hopefully it will help:

train_vecs = []
for i in range(len(your_training_examples)):
    top_topics = lda_train.get_document_topics(train_corpus[i], minimum_probability=0.0)
    topic_vec = [top_topics[i][1] for i in range(20)]
    train_vecs.append(topic_vec)

The above would give the top 20 topics for every document. 'train_corpus' is the result of doing something like this in Gensim once you have a bigram object from the 'Phrases' Gensim model class:

train_corpus = [id2word.doc2bow(text) for text in bigram]


How to best read large dataset from disk

A common way is to create a class inheriting tf.keras.utils.Sequence. This class implements a function __getitem__ which is called when you use model.fit() method. In this method, you can simply load one batch at a time, so no need to load the whole dataset. See the documentation. You can also directly use the .npz files when calling __getitem__.
Shuffle data

Do I save my data as is, or do I need to save shuffled batches?

You would implement the method on_epoch_end() that does the shuffling of a list of indices. Then when you load a .npz file, use something like data[indices[i]] which load the index i in the shuffled list of indices.
on_epoch_end is called at the end of each epoch by .fit() method, and you can also use it in __init__ of your Sequence class to initialize the shuffle.
From Keras to Pytorch
Pytorch has similar module called torch.utils.data.Dataset. The conversion is straightforward. See this tutorial.


Depthwise separable convolution followed by 1x1 convolution is an alternative to the convenient 2d convolution. They were introduced in https://arxiv.org/abs/1704.04861 to aiming on reducing the computational effort. The combination of depthwise and pointwise convolutions often learn pretty much the same representation as the normal conv.
Depthwise separable conv is an extreme case of grouped conv, where num_groups=num_input_channels. Setting number of groups to 2 means slicing the input into two inputs across the channel dim and learning two separate filters for each sliced input. Here is nice tutorial on that. The output is combined by stacking each group's output. Often this is followed by 1x1 conv to mix the data flow between the groups.


CoCalc provides Jupyter notebooks with realtime collaboration, unlike Colab, Kaggle, etc.  You just make a project drag and drop ipynb and data files, add collaborators, and everybody can edit everything simultaneously.  You can also share content publicly at the share server.   I think CoCalc is currently the overall most mature of the realtime Jupyter collaboration platforms (and it is the only open source one 4), but Deepnote is another option that is more focused on data science (but is closed source).


Using predicted probabilities and bayesian inference to update beliefs

Anything that can be done in the classical, frequentist statistical approach, has a Bayesian counterpart.

Beliefs can be described, under certain assumptions, as probability distributions. You can take a given PDF as your prior belief, combining it with the Likelihood function of your data through Bayes theorem, and obtain a posterior distribution describing your updated belief in light of the empirical evidence provided.

Unfortunately I have no information of your problem at hand, on what data you are working with, and their nature.

If you are interested in Bayesian statistical models I suggest you to take a look at an R package called JAGS, that you can use to implement pretty much any Bayesian model, and with ready-to-go MCMC algorithms.

I wish I could be more specific, please provide more details about your problem and your data.


Solutions for Continuous Online Cluster Identification?

Stability-Plasticity Dilemma, Learning Rates, and Forgetting Algorithms:

First, let me say that this is a really great question and is the type of thought provoking stuff that really improves one's understanding of ML algorithms.

Does this "problem" have a name that it can be referred to?

This is generally referred to as "stability".  What's funny is that stability is actually a useful concept in regular clustering i.e. not online. The "stability" of the algorithm is often chosen as a selection criterion for whether the right number of clusters have been selected.  More specifically, the online clustering stability issue that you have described is referred to as the stability-plasticity dilemma.

Are there "standard" solutions to this and ...

First, the big picture answer is that many online clustering algorithm are surprisingly stable when they have been well trained with a large cohort of initial data.  However, its still a problem if you want to really nail down the cluster identities of points while allowing the algorithm to react to new data.  The trickiness of you point is briefly addressed in Introduction to Machine Learning By Ethem Alpaydin.  On page 319 he derives the online k-means algorithm through the application of stochastic gradient descent, but mentions that the stability-plasticity dilemma arises when choosing a value for the learning rate.  A small learning rate results in stability, but the system looses adaptability where as a larger learning rate gains adaptability, but looses cluster stability.

I believe the best path forward is to choose an implementation of online clustering which allows you to control the stochastic gradient descent algorithm and then choose the learning rate so that you maximize stability and adaptability as best as you can using a sound cross-validation procedure.

Another method that I've seen employed is some sort of forgetting algorithm e.g. forgetting older points as the data stream matures.  This allows for a fairly stable system on fast time scales and allows for evolution on slower time scales.  Adaptive Resonance Theory was created to try to solve the stability-plasticity dilemma.  You might find this article interesting.

I'm not well-versed enough in R to suggest an algorithm, but I suggest you look for a mini-batch k-means algorithm that allows you to control the learning rate in its stochastic gradient descent algorithm.

I hope this helps!


Yes, there is a reason. It has to do with how you initialize your weights.

There are 16 local minimums that have the highest probability of converging between 0.5 - 1.

Here is a paper that analyses the xor problem.


You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $\text{log}(0.5) < 0$ and $\text{log}(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

Probability: $P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$ (using softmax as mentioned in Figure 10.3 caption, and assuming $\mathbf{o}=(o_1,..,o_K)$ is the output of layer before softmax),
Unnormalized probability: $\tilde{P}(i) = e^{o_i}$, which can be larger than 1,
Log of unnormalized probability: $\text{log}\tilde{P}(i) = o_i$, which can be positive or negative.


How to know the model has started overfitting?

Let's say we want to predict if a student will land a job interview based on her resume.

Now, assume we train a model from a dataset of 10,000 resumes and their outcomes.

Next, we try the model out on the original dataset, and it predicts outcomes with 99% accuracy… wow!

But now comes the bad news.

When we run the model on a new (“unseen”) dataset of resumes, we only get 50% accuracy… uh-oh!

Our model doesn’t generalize well from our training data to unseen data.

This is known as overfitting, and it’s a common problem in machine learning and data science.

Overfitting V/s Underfitting

We can understand overfitting better by looking at the opposite problem, underfitting.

Underfitting occurs when a model is too simple – informed by too few features or regularized too much – which makes it inflexible in learning from the dataset.

Simple learners tend to have less variance in their predictions but more bias towards wrong outcomes (see: The Bias-Variance Tradeoff).

On the other hand, complex learners tend to have more variance in their predictions.

Both bias and variance are forms of prediction error in machine learning.

Typically, we can reduce error from bias but might increase error from variance as a result, or vice versa.

This trade-off between too simple (high bias) vs. too complex (high variance) is a key concept in statistics and machine learning, and one that affects all supervised learning algorithms.


The text is obfuscated for you but to the computer, this is as incomprehensible as English for instance. When classifying text, you always first vectorize your input, meaning that your text becomes a vector. This vector typically represents the frequency of the tokens in your vocabulary in that particular string (bag-of-words representation).

What you can do is represent each document/string as a vector representing the frequencies of characters, character n-grams, or even words. One you have your input, you can deal with it as any classification task.

Naive Bayes has shown to be a good lower bound for these types of tasks.


You can do that by

>>> import pandas as pd
>>> m = pd.DataFrame({'gender': [1, 2, 2, 1, 1, 2, 1], 'rating': [3, 4, 2, 1, 3, 1, 5]})
>>> m.groupby(['rating','gender']).size().to_frame('count').reset_index()
   rating  gender  count
0       1       1      1
1       1       2      1
2       2       2      1
3       3       1      2
4       4       2      1
5       5       1      1

Hope this is what you want to pull.

Edit:
As mentioned, I did not account for zero values.

You need to do one more step additional to get what you want. Finding the combinations that are missing and then joining it. The one liner solution would be

>>> from itertools import product
>>> m.groupby(['rating', 'gender']).size().to_frame('count').reset_index().merge(
        pd.DataFrame(list(set([i for i in product(*[m.gender, m.rating])])), columns=['gender', 'rating']),
        on=['rating', 'gender'],
        how='right').fillna(value=0)
   rating  gender  count
0       1       1    1.0
1       1       2    1.0
2       2       2    1.0
3       3       1    2.0
4       4       2    1.0
5       5       1    1.0
6       2       1    0.0
7       4       1    0.0
8       3       2    0.0
9       5       2    0.0

Explanation

Get the original grouped counts as mentioned before the first edit but this time you need to join with the combinations that are missing to get the zero counts. Use itertools.product to get all combinations of gender and rating and right join it with original grouped frame on rating and gender to get merged DataFrame which has numpy.na values if no count is present and then use fillna method to fill it with zero. The only loophole is if a rating say 4 is not there in the original data, there is no combination that follows it.

Hope this helps.


MAP@k is normally used in recommendation systems, but also in other kinds of systems. Quoting from here:

If you have an algorithm that is returning a ranked ordering of items, each item is either hit or miss (like relevant vs. irrelevant search results) and items further down in the list are less likely to be used (like search results at the bottom of the page), then maybe MAP is the metric for you!

Some application examples are these Kaggle competitions:

Expedia Hotel Recommendations
Santander Product Recommendation
National Data Science Challenge 2019

And these are some resources with more information about it:

Mean Average Precision (MAP) For Recommender Systems

How mean Average Precision at k (mAP@k) can be more useful than other evaluation metrics


Text PreProcessing not improving accuracy

You can find that information on the class's documentation, here's the link.

Quick summary:

lowercasing: True by default.
lemmatization: I think the closest parameter is preprocessor; it's
set to None by default but you can specify as the value a callable
object of your own and then the vectorizer will apply it on every
word to do the lemmatization, e.g.NLTK's WordNet lemmatizer.
tokenizer: None by default (because the class expects its input to
be already tokenized words, so no need for this unless that's not
so).
stopwords: None by default (but you can specify a list of strings as
the argument and then the class will filter out any words in that
list).

I agree that it is suspicious that you're getting the same Accuracy, generally it is expected to go up when using TFIDF. Make sure you're

1) sending the right input to the TfIdfVectorizer class (it takes a list: list: str, dataset: sentence: each word),

2) using its output (perhaps some variable in your code is still pointing to the non-TFIDF representation instead of the TFIDF one?)

I think it would not be theoretically impossible for SGDClassifier to approximate TFIDF behavior over a sufficiently large number of training iterations with the right learning rate (if all the stopwords' weights are effectively minimized while optimizing loss, TFIDF might have little effect at that point). It would be good to know what parameters you're using and/or to see the code to help you pinpoint the issue.


I would suggest using a Pandas DataFrame (you didn't mention it in your question).

Then it would be as simple as the following

Create the dataframe from you list x, calling the single column x:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame(x, columns=["x"])   # x is defined in your question

Add a new column (I call it action), which holds your result. As it will be either -1 or +1, I fill it all with +1 to begin with, then only change the values to -1 where your criteria is met:

In [3]: df["action"] = 1

In [4]: df.head()
Out[4]:
         x  action
0  2352.60     1
1  2410.26     1
2  2443.31     1
3  2525.78     1
4  2506.58     1

Use the diff() method to find the difference between each row of x. If the difference is less that zero, we can make action equal to -1.

In [5]: df.action[df.x.diff() < 0] = -1

In [6]: df
Out[6]:
          x  action
0   2352.60       1
1   2410.26       1
2   2443.31       1
3   2525.78       1
4   2506.58      -1
5   2530.69       1
6   2530.49      -1

If you want to shift the difference up one row, because of your rule:

  Action [ i ] = (-1) if x[ i+1 ] < x[ i ]

Then you simply include that in the assignment part, filling the action column from the start index to the penultimate element:

In [7]: df.action[:-1] = df.action[1:]

In [8]: df.head()
Out[8]:
          x  action
0   2352.60     1.0
1   2410.26     1.0
2   2443.31     1.0
3   2525.78    -1.0
4   2506.58     1.0
5   2530.69    -1.0

Note that the last row will contain a value that could be wrong (you cannot know, without knowing the next value of x).

After edit from OP

If you want to do it in normal lists, you can do this:

In [9]: action = []

In [10]: for i, val in enumerate(x):
     ...:    if i == (len(x) - 1): break
     ...:    if x[i+1] < val:
     ...:        action.append(-1)
     ...:    else:
     ...:        action.append(1)

In [11]: action[:10]         # Look at the first ten elements
Out[11]: [1, 1, 1, -1, 1, -1, 1, 1, -1, -1]

enumerate iterates through the iterable you give it (here, x) and also returns the index of the current element.  We need the check that we do not exceed the length of x, because we check i + 1 each time, so could try to check a value one element too far in the list x.


Is there an analog to SQL's STRING_AGG (or FOR XML PATH) function in Python?

Here a versatile solution. As you can see, you can modify the aggregation function in order to format the data as you want.

#the original DataFrame
df=pd.DataFrame({'ID_NO': [20, 20, 30, 30, 30], 'RUN_NO': [1,2,1,2,3], 'START': ['F2','F3','F9','F11','F14'],
                'STOP': ['F3','F2','F11','F14','F6',]})

#convert 'RUN_NO' to string. This will make the aggregation formula easier to read
df['RUN_NO']=df['RUN_NO'].apply(str)

#The aggregation function
def agg_f(x):
     return pd.Series(dict(Sequence = "%s" %  ' '.join('(' +x['RUN_NO']+ ') ' + x['START'] +' to ' + x['STOP'])))

df_agg=df.groupby('ID_NO').apply(agg_f)

The output will be:


Best machine learning algorithm for understanding specific conditional structures

To me, this describes conditional probabilities. And the first thing that springs to my mind when it comes to learning conditional probabilities is a Bayesian network (BN), which is an example of a probabilistic graphical model. In fact, BNs can be fitted using Maximum Likelihood Estimation (MLE), which corresponds to counting occurrences (similar to "Naive Bayes", if you have used that). An excellent book on this is Barber.

Another model that corresponds to your probabilistic problem, and which is more neural network-like, is a Restricted Boltzmann Machine (RBM):

(source)

Once again, this involves learning conditional probabilities, like $p(v_0 | h_0, h_1, h2), p(v_1 | h_0, h_1, h2), p(v_2 | h_0, h_1, h2), \ldots$. This is a nice introduction to them. Unlike neural networks, which are trained using backpropagation and are not probabilistic, RBMs are indeed probabilistic and are trained using Contrastive Divergence.


Feature selection/reduction technique for combination of features in image processing

I'd recommend assessing the performance of the model "as-is" and then decide for feature selection or not.

If you have good performance for both train / test sets, then there's no need to perform feature selection.

If you end up overfitting your data (low train error, high test error) you may want to reduce the number of features using feature selection / regularization (the cost function is penalized when the model uses a large number of parameters)

If you have low train performance, this means that you have high bias (the model does not learn properly from those features). This usually happens if the model is not complex enough to extract structure from the data. You may want to add features in this case, rather than selecting them out.

Hope this helps.


As you probably know by now, cross-validation is a method of:

partition dataset into 'train set' and 'test set'
fit model to train set, get the prediction error on test
repeat (1-2) in a bunch of different, 'pre-defined' ways, storing the prediction error each time.
average all the prediction errors to understand how your model will behave in the wild!

All the 'magic' of different cross-validation (CV) types happens at (3), basically. Say your dataset has $n$ observations.

leave $p$ out CV separates your dataset into a train set of size $n-p$ and a test of $p$, gets the prediction error, and repeats this process for every single possible $p$-sized subset of the dataset.
leave $1$ out CV is a special case of leave $p$ out where $p=1$
$k$-fold cross CV partitions your dataset into $k$ equal sized partitions, trains on the first $k-1$ partitions and tests on the last partition, and gets the prediction error. The result at the end is ten $k$ prediction errors to average over. (Note that $k=n$ is the same as leave $1$ out CV.)

As for disadvantages and advantages, it's basically just a trade-off between exhaustiveness and speed. And I guess it depends on your dataset. The upper limit of exhaustiveness is basically doing leave $p$ out CV for all $p=\{1,2,...,n\}$. Probably depends too on the linearity of your dataset. In my experience, I haven't had trouble using $k$-fold cross CV.

Reference


How to overcome the class imbalance issue in a text classification problem?

Option 1: play with the thresholds for classifying something – perhaps you could set a threshold for classifying something as a minority class should even if it doesn't have the highest probability/score, or a ratio of probabilities/thresholds.

Option 2: upsample your data (take resamples with replacement of the minority classes until they have as many observations as the others).

Option 3: use some cost-sensitive algorithm (e.g. some forms of classification trees) - probably won't translate into very accurate results.


I don't know any Georgian stemmer or lemmatizer. I think, however, that you have another option: to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of Georgian to devise some heuristic rules to identify the stem among them.
This kind of approach consists of a model trained to identify morphemes without any labels (i.e. unsupervisedly). The most relevant Python package for this is Morfessor. You can find its theoretical foundations in these publications: Unsupervised discovery of morphemes; Semi-supervised learning of concatenative morphology.
Also, there is a Python package called Polyglot that offers pre-trained Morfessor models, including one for Georgian. Therefore, my recommendation is for you to use Polyglot's Georgian model to segment words into morphemes and then write some rules by hand to pick the stem among them.
You should be able to evaluate the feasibility of this idea by adapting this example from Polyglot's documentation from English to Georgian (by changing the language code en and the list of words):
from polyglot.text import Text, Word

words = ["preprocessing", "processor", "invaluable", "thankful", "crossed"]
for w in words:
  w = Word(w, language="en")
  print("{:<20}{}".format(w, w.morphemes))


Are there trained neural networks, that can distinguish a book's author point from what he stands against?

Absolutely. You've described topic modeling and sentiment analysis. You might want to train separate models for each part of the problem and stack them, but this is a totally solvable problem.


How to speed up passing of images to a GPU

Try to use batch version for training. You have to implement a function to manage your data which exist in the memory. For each batch, you can load the data into your memory as much as possible, high RAM size can help. Then you can pass a batch to your network, the size of graphical memory matters here, using the current data which is in memory now. By doing that you reduce the number of disk calls. When it happens to load data from disk, you fill the memory as much as possible, the disk speed matters here; consequently, high speed SSDs are valuable for facilitating the reading operation.


What is the best way to find minima in Logistic regression?

Being a second-order method, Newton–Raphson algorithm is more efficient than the gradient descent if the inverse of the Hessian matrix of the cost function is known. However, inverting the Hessian, which is an $\mathcal O(n^3)$ operation, rapidly becomes impractical as the dimensionality of the problem grows. This issue is addressed in quasi-Newton methods, such as BFGS, but they don't handle mini-batch updates well and hence require the full dataset to be loaded in memory; see also this answer for a discussion. Later in his course Andrew Ng will discuss neural networks. They can easily contain millions of free parameters and be trained on huge datasets, and because of this variants of the gradient descent are usually used with them.

In short, the Newton–Raphson method can be faster for relatively small problems, but the gradient descent scales better with complexity of problem.


Is this a data issue, or a model issue? A Keras binary classification model

Possible problems:

You have classes 0 and 0.1. You want to make them 0 and 1 since the formula for the loss function
$$-{(y\log(p) + (1 - y)\log(1 - p))}$$
expects $y$ to be 0 or 1.
shuffle = False why would you want to turn shuffling off?
The classes are unbalanced. You have ~100 examples of class 1  and ~20000 total examples. So if you just predict class of everything to be 0 you will be right (20000-100)/20000 = 99.5% times. You can try these tricks and see if they help.


Difference between Jordan, Elman and normal RNN

The architectures in your second and third pictures are identical because the link $I$ from the hidden units to the context units in the Elman's network is not trainable. It is always equal to 1.
In his paper [1], Elman first provides the Jordan's architecture, then introduces his own network but does not discuss why his one is better.
Anyway, this is what I think about why it may be preferred.
The neurons in the hidden layer learn features or internal representations of the inputs. Backward connections allow hidden units to learn features which encode sequential properties of the inputs.
In a deep network, the neurons in the layer that is further from the input learn more abstract features than the neurons in the layer closer to the input. Feeding the output from a layer back to the same layer allows learning sequential properties at the same level of abstraction. Feeding the output from a layer to the input of a different layer mixes the levels of abstractions.
In the Jordan's network, the output of the whole network directly influences the hidden layer's input, so the learned representation is split between the hidden layer and the output layer. Elman's network allows more separation of features by levels of abstraction.
Elman spends most of his paper trying to understand the features his neurons have learned even for those tasks where his network does not perform well. I am not completely sure but I guess that backward connections between different layers would make such analysis more difficult.

Elman, J.L. (1990), Finding Structure in Time. Cognitive Science, 14: 179-211. doi:10.1207/s15516709cog1402_1


Feature selection might be consider a stage to avoid.
You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are not optimal since the problem is NP-Complete.
Using it doesn't sound like an offer that you cannot refuse.

So, what are the benefits of using it?

Many features and low samples/features ratio will introduce noise into your dataset. In such a case your classification algorithm are likely to overfit, and give you a false feeling of good performance.
Reducing the number of features will reduce the running time in the later stages. That in turn will enable you using algorithms of higher complexity, search for more hyper parameters or do more evaluations.
A smaller set of feature is more comprehendible to humans. That will enable you to focus on the main sources of predictability and do more exact feature engineering. If you will have to explain your model to a client, you are better presenting a model with 5 features than a model with 200 features.

Now for your specific case:
I recommend that you'll begin in computing the correlations among the features and the concept. Computing correlations among all features is also informative.
Note that there are many types of useful correlations (e.g., Pearson, Mutual information) and many attributes that might effect them (e.g., sparseness, concept imbalance). Examining them instead of blindly go with a feature selection algorithm might save you plenty of time in the future.

I don't think that you will have a lot of running time problems with your dataset. However, your samples/features ratio isn't too high so you might benefit from feature selection.

Choose a classifier of low complexity(e.g., linear regression, a small decision tree) and use it as a benchmark. Try it on the full data set and on some dataset with a subset of the features. Such a benchmark will guid you in the use of feature selection. You will need such guidance since there are many options (e.g., the number of features to select, the feature selection algorithm) an since the goal is usually the predication and not the feature selection so feedback is at least one step away.


MAE estimation for k-fold cross-validation

It’s an issue of units.

Compare what you’re doing in RMSE and in MAE.

RMSE is a way of getting MSE back to the original units, like how we take the square root of variance to get standard deviation. This makes more physical sense. Sure, we can make sense of square meters, but what about square dollars?

When you do MAE, you don’t have that squaring action to give you squared units. Consequently, while MSE in in square units, MAE is in the original units.

You could take the square root of MAE, but then you’d wind up with measurements with units of $\sqrt{\$}$ or the square root of whatever units you’re using. The result is that your measure of dispersion is not in the original units, which a probably what you want.

I do not see any use for taking the square root of MAE. If you do, please do share. That would be very interesting.


Extract relevant features from time series data

I think the first question you should ask before you start going off on a deep learning model is, can you tell when the failure is going to occur just by looking at a plot of your data? If you can't, then no model will help you deduce when a failure will occur.

You shouldn't overlook some basic models also such as exponential or poisson distribution models that should model your current problem well.

Lastly, since it's a time series for rare events, some ways to gain more insight might include time between failures, identifying thresholds of the data value triggering a failure, splicing all the data into intervals and the number of failures, etc.. You should also check if the failures follow a pattern.


Your idea isn't wrong, however in k-NN there always might be a case where you have the same number of votes for 2 or more classes (e.g. you have $k=6$ and you have 3 samples of one class vs 3 of another). With your solution you are just overcoming one very small case of ties and the $k$ that you choose might not be the optimal $k$ for classification, which is your primary objective (i.e. you might hurt your classification performance with a sub-optimal $k$ so that you don't get one type of ties).
Because there are also other times of ties, you always need to have a tiebreaker condition. I have seen a lot of strategies for this, like reducing $k$ by one until you break the tie, selecting the class with the minimum distance, selecting the class with the fewest samples overall, etc.
Since you already have such a condition, you could also handle your situation, without selecting a sub-optimal $k$.


Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?

In "Transformer model", during the training, all the predictions are computed in one step. That's one of its strength points.

It is during inference that the target sentence is generated sequentially invoking a beam search (usually with a small beam width).

During training (emphasizing on training) just use target word indexes as inputs to the decoder, you just need to trim the index of  token from end of target sequence.


Hot Encoding

For each user, create a vector with the length of your movie catalog and add 1 to the movies the user watched and 0 to the ones the user did not watch.

This is a very naive approach but depending on your task might be enough.

Do keep in mind that your data is suddenly sparse, so choose algorithms that can deal with such data. The good thing is that any kernel above linear is often not necessary for sparse data.

Taking Duration into account

You could also do a Hot Encoding where instead of adding a 1 to the movies watched, you add the duration that the person watched.

However, this can start to become biased for movies that are longer than others. So, you can try two normalization techniques:

Movie length normalization. Take the time the person watched the movie and divide be the actual length of the movie. (e.g. the user watched 75 mins, but the movie was 90 mins, so you get 75/90 = 0.83)
TF-IDF usually used in NLP, but you can apply it here. This method also takes into account the frequency in which a movie is watched by all users. Balancing the weight between movies that are important (i.e. viewed by many users) and less important (i.e. moveis viewed by very few or just one user).

Other

Depending on your use case, you can start to be creative and group movies:

Group movies by genre (e.g. 10 genres) and count the time watched in each genre.
Check the percentage of the movie watched and group them into watched only the beginning, watched more than half and watched the full movie and count each category for each user.


AUC ROC in keras is different when using tensorflow or scikit functions.

No, you shouldn't have the same numbers. All depends on the additional parameters:

tf.metrics.auc(
    labels,
    predictions,
    weights=None,
    num_thresholds=200,
    metrics_collections=None,
    updates_collections=None,
    curve='ROC',
    name=None,
    summation_method='trapezoidal'
)

This means that this curve will have 200 points, so very smooth.

sklearn version doesn't have this kind of parameters:

roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)

The number of outputs depends on the curve and the number of points if I remember properly.


Relating ROC curves with class statistics

I don't want to offend you in any way but the problem you mentioned above is not really a problem but an opinion that you presented on ROC vs F1 score, and I guess you want other people's opinion on the same.
I would highly suggest you to ask to the point questions next time since if the questions get downvoted you can get banned (I am not downvoting so don't worry), just trying to be informative.
Now from what I was able to grasp from your post, I think that you are having difficulty in choosing the right metric for evaluating the three neural net models you have shown results for.
AUC vs F1 :
In general, the ROC is for many different levels of thresholds and thus it has many F score values. F1 score is applicable for any particular point on the ROC curve.
You may think of it as a measure of precision and recall at a particular threshold value whereas AUC is the area under the ROC curve. For F score to be high, both precision and recall should be high.
Consequently, when you have a data imbalance between positive and negative samples, you should always use F1-score because ROC averages over all possible thresholds.
Conclusion :
So in your case particularly since I can clearly see through precision and recall values that your classes are highly imbalanced, use F1 score to choose the model instead of AUC-ROC score.


Creative Analysis - What are the possible approaches?

If you are looking to find effect of each feature on CTR then a correlation matrix can help you with it and you can do statistical test to prove it.
But if you want to find hidden causes then it a classical problem of prediction vs causation.
Suppose we find that giving discount help increase our campaign you may conclude that giving discount helps but it may actually happen that sales are usually higher that time of year and so giving more discounts necessarily doesn't help. Causal analysis can help us find hidden causes which will be hard to find.
For doing Causal Analysis you can refer Dowhy package.


source and target functions in igraph package in R

Given

library(igraph)
set.seed(1)
g <- sample_pa(5)

you got

ends(g, E(g))
#      [,1] [,2]
# [1,]    2    1
# [2,]    3    1
# [3,]    4    1
# [4,]    5    2

and

head_of(g, E(g))
# + 4/5 vertices:
# [1] 2 3 4 5

tail_of(g, E(g))
# + 4/5 vertices:
# [1] 1 1 1 2

source() is an important base function.


Noramlization Time Series to Predict Stocks exact Price

You should always fit the scaler with the train data. This is one of the typical validation problems, it is hard to replicate train test conditions.

Have you considered doing rolling/rollout window split?

This way the range of values will be closer between train and test.

THe problem of this is that you loose  some of the training information. On the other hand information about 40 years ago of stock has not much to do when forecastin the present


How does recommendation by matrix factorization deal with new movies / users for which there are ratings?

tl;dr:  It can't!

Why not?

This is one of the main problems of collaborative-filtering recommender systems (i.e. ones that rely solely on the user-item interaction matrix to generate recommendations).

This is referred to as the cold start problem:

  As collaborative filtering methods recommend items based on users' past preferences, new users will need to rate sufficient number of items to enable the system to capture their preferences accurately and thus provides reliable recommendations.
  source

The same is true for new items.

How can we combat this?

The easiest way to combat this issue is to initially identify similar users through demographics and recommending popular items to the user group that best represents him.

Another very popular approach is making a hybrid recommender system, i.e. employs both collaborative-filtering (CF), content-based (CB) and/or even dummy (D) techniques. For instance it could initially recommend the most popular items (D) for new users, then start recommending similar items to the ones he initially liked and finally when the user has rated enough items start looking for users with similar tastes.

In practice most real-world recommender systems are hybrid.

  Netflix is a good example of the use of hybrid recommender systems. The website makes recommendations by comparing the watching and searching habits of similar users (i.e., collaborative filtering) as well as by offering movies that share characteristics with films that a user has rated highly (content-based filtering).
  source


Library/package/tool for geographical data visualizations

Assuming you want your visualizations to be available on the internet, there are many options for this:

Google Maps (and Google Fusion Tables)
Mapbox (and the related LeafletJS)
CartoDB
D3

It really comes down to exactly what you want to do and how much control you want to have over the map.  Additionally you should consider what sort of interactivity you need.

Google Maps is probably the most user friendly for very basic map functions but is somewhat limited in what you can do stylistically.

Mapbox and CartoDB are both user friendly and offer good options for styling and displaying different varieties of data.  However, they also both tend to require subscription fees for anything other than small maps. Also, the last time I checked, CartoDB explicitly handles animation and time-series data where Mapbox does not.

D3 will probably give you the most control over display, animation, and interactivity but also has a long learning curve. Even if you don't want the map to be available on the internet, this is still a very good tool for making interactive visualizations that run in the browser.

If you don't care as much about the visualizations being online, you can get a lot of work done in open GIS software like QGIS or GrassGIS, though I don't know if user interactivity is really an option there.

As I said though, it really comes down to the specifics of exactly what you're trying to do and how comfortable you are with various aspects of mapping and coding.


Pandas dataframe resample aggregation by mills too slow


Is it possible to connect three neural networks in Matlab?

If you want to combine the results from three different Neural Networks to "boost" the performance :) , you might want to look at the different Ensemble Learning Methods as I mentioned earlier.

Which method you should use, depends on how you share or divide the training data between the three NNs. For example if the NNs are trained on same data but have different parameters, you can look at simple voting ( if you are doing a classification task) or averaging ( if you are using them for regression).

The more advanced methods like AdaBoost divide the training data between the classifiers. You can read about it in Boosting Neural Networks


Machine learning algorithm which gives multiple outputs from single input

Keras functional API's are a way your can solve you problem. Using keras functional API, we can build models that resembles more like graphs such as this:

In order to build a model like this, you can use keras as follows:

from keras.models import Model
from keras import layers
from keras import Input

input_layer = Input(shape=(100,), dtype='float32', name="Input")
split_layer = layers.Dense(32, activation='relu', name='split_layer')(input_layer)
first_layer = layers.Dense(32, activation='relu', name='first_layer')(split_layer)
second_layer = layers.Dense(32, activation='relu', name='second_layer')(split_layer)
model = Model(input_layer,[first_layer, second_layer])
model.summary()

In order to compile this model, we can define different loss functions for different layers

model.comile(optimizer=optimizer,
            loss={'first_layer':'mse', 'second_layer':'binary_crossentropy'},
            metrics=['accuracy'])

Once you are done with building the network, you could simply fit you data as follows:

model.fit(X,
        {'first_layer': first_layer_Y,
         'second_layer': second_layer_targets},
          epochs=10
        )


Predict correct answer among ten answers for a given question

It's not DL but I suggest you start with the following approach: for every question $Q$ and its set of candidate answers $(A_1,..,A_{10})$, represent each pair $(Q,A_i)$ as an instance with its label 0 or 1. You could start with a few simple features such as:

number of words in common
similarity score, e.g. cosine TFIDF
... other indicators of how well question $Q$ and answer $A_i$ match

Train a regression model on this (e.g. decision tree, SVM,...). When the model is applied to a new question+answers, it returns a score (mostly between 0 and 1) for each of the 10 answers; finally select the answer which has the highest score.

You can certainly improve on this idea, e.g. with sentence embeddings.

Note: a simple baseline system based on the same idea would be to select the answer which has the maximum number of words in common with the question.


Weighted linear regression with a DNN (in Keras)

This is a suggestion. I don't know if this will work. Maybe you could adapt a feature selection problem to a data point selection problem. For a standard linear regression, The LASSO technique can be used to do both the regression and the feature selection. The idea is to penalize the use of too much features by adding a term to the loss.

Where beta are the coefficients to be learned of your linear regression.

With only one feature x_i and with your quality index q_i (they should sum to 1) your loss can be written as :

I see 2 ways of doing this :

Minimize the loss above. beta_1 and beta_0 are the only coefficients that can be learned. The process will not eliminate the points with low quality but they will contribute just a little to the loss.
Consider the q_i as learnable parameters. You don't use the quality index that you already have. And then you can add a penalization term on the q_i. Those who end up to 0 will not contribute to the loss. The loss will look like this :

I think there are many details to solve. Like adding a constraint to the q_i like 0 < q_i < 1. Maybe this can not be solved easily. I suggest you to use Pytorch. This will be way much easier to implement.


Can I train two stacked models end-to-end on different resolutions?

I would suggest you to split your problem into two:

How to train?
How to make inference when resources are limited?

Very common pattern is to train model on large data (by renting AWS server, for instance) and then use its predictions to train much smaller network.
Here is fundamental paper on the subject from Hinton et al.
This way you can stick to more conventional object detection method without stacking involved and then deploy 'squeezed' network to your device.
But back to your question. If you are rigid in your approach for various reasons and can't do like I described above, then lots of end-to-end approaches are available in literature, take a look at this to get started.
And even code.
I will just cite the paper:

A unified deep neural network, denoted the multi-scale CNN (MS-CNN),
is proposed for fast multi-scale object detection. The MS- CNN
consists of a proposal sub-network and a detection sub-network.
In the proposal sub-network, detection is performed at multiple output
layers, so that receptive fields match objects of different scales.
These complementary scale-specific detectors are combined to produce a
strong multi-scale object detector. The unified network is learned
end-to-end, by optimizing a multi-task loss.
Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs.
State-of-the-art object
detection performance, at up to 15 fps, is reported on datasets, such
as KITTI and Caltech, containing a substantial number of small
objects.

But nevertheless feel free to experiment with network compression (take a look at this). It typically has high degree of redundacy, and by being creative in exploring this redundacy it's possible to have many gains


Classifications as long-term memory and short-term memory in LSTM

I recommend you to read http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Basically, the long-term memory and short-term memory are reflected in the structure of the LSTM Network. Please take a look at the diagram from the website I've mentioned.

The long-term memory is constructed in the "bus" on the top, where the previous value can be passed, depends on the $\sigma$ value. The $\sigma$ is between $[0,1]$ and $0$ means- forget the last value, $1$ means- get all (exact) the value from the last state. Sigmas for each cell are established during the training phase.


Difference between Non linear regression vs Polynomial regression

the difference is probably easily seen with an example. Linear regression assumes a form $$f(x, \beta) = \beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n$$ with some covariates $x_i$ and some parameters to estimate $\beta_i$. An example of non-linear regression would be something like $$f(x,\beta) = \frac{\beta_1 x_1 + \beta_2 x_2}{\beta_3 x_3 + \cdots + \beta_n x_n}.$$ Essentially you are assuming your model to be of a nonlinear form. Polynomial regression on the other hand is a fixed type of regression where the model follows a fixed form $$f(x, \beta) = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_n x^n$$ which is a nonlinear function, however it is still linear in the parameters $\beta$ you are trying to estimate.

That is to say,

Polynomial regression is non-linear in the way that $x$ is not linearly correlated with $f(x, \beta)$; the equation itself is still linear.

In the other hand, non-linear regression is both non-linear in equation and $x$ not linearly correlated with $f(x, \beta)$.


Random Forest Target/prediction maximum minimum

Random Forest Regressor doesn't have such an option, usually, if you have enough data and a correct training process, predictions should be in an acceptable range. Of course, they might go above and beyond your range, but the error might be still acceptable.

If you are looking for a model that is able to utilize prediction boundaries, I suggest you look at Facebook Prophet as it may be useful for your problem at hand.
Python Prophet Quick Start

For range-predictions, you can specify "Forecasting Growth":

  When forecasting growth, there is usually some maximum achievable point: total market size, total population size, etc. This is called the carrying capacity, and the forecast should saturate at this point.

Here is a link for this part
Forecasting Growth and Saturating Minimum

And another note - Prophet proves to be quite useful in predicting any seasonal related time series, so you can achieve even better performance predicting sales compared to Random Forest.


Strings/ features in Turicreate decision tree

I'm not familiar with the package, but the documentation suggests that strings as input will be handled internally by dummy-encoding (with a reference level).  The linked section is for linear regression, but is linked to from the Advanced Features section of the decision tree page, so I assume that applies.


Comparing Efficiency of Machine Learning Algorithms

One deep learning method can apply to multiple problems (for example computer vision, natural language processing etc), the performance of the method may be different depends on the problem that the method handles, but there is "popular dataset" that usually use by deep learning researchers to benchmark their method. Let say you have deep learning method, if you want to know how efficient your method for handwritten classification you can benchmark your method by classifying mnist dataset and compute its accuracy. If you want to compare the performance of your method with another method you can search on the internet another method that uses the same dataset for benchmark, here is an example


I'm trying to do a time series model without a datetime field in python. Is this possible?

You can use timedelta function from datetime to achieve this, starting from a known date.
Example code can be something like,
from datetime import datetime, timedelta

start = datetime.strptime('2021-01-01', '%Y-%m-%d')

all_dates = [start + timedelta(x) for x in range(10)]

print(all_dates)

Replace the range(10) above with the sequence of the actual days in your dataset.


Yep, this is the correct interpretation. The kernels make a difficult classification problem into a much simpler one by making the data linearly separable by transforming it into a higher dimension. I think this image does a good job of illustrating that.


difference between scaling/normalizing data at a specific step

Case I -
With a single scaling step, you might leak the test info into the train. In this approach you have a common min/max otherwise it would have been two pairs
See, the plot for one of the Features of Iris dataset
Also, we don't scale the target. But I see this in your code.

Case II -
 This is fine but you should also consider the online cases, where you will not have a test set to scale the new test data.
Case III -
 This is a better and an agnostic approach.
Your code implementation incorrect as suggested in the comment


Curse of dimensionality: Data set with 1 million rows, max number of features can I have approximately?

The curse of dimensionality means that your intuition fails at a certain number of features. See Average Distance of Random Points in a Unit Hypercube for some examples. It does not matter how many points you have.

For example, at around 100 dimensions every two points randomly sampled points are really close.


Getting GitHub repository information by different criteria

My limited understanding, based on brief browsing GitHub API documentation, is that currently there is NO single API request that supports all your listed criteria at once. However, I think that you could use the following sequence in order to achieve the goal from your example (at least, I would use this approach):

1) Request information on all public repositories (API returns summary representations only): https://developer.github.com/v3/repos/#list-all-public-repositories;

2) Loop through the list of all public repositories retrieved in step 1, requesting individual resources, and save it as new (detailed) list (this returns detailed representations, in other words, all attributes): https://developer.github.com/v3/repos/#get;

3) Loop through the detailed list of all repositories, filtering corresponding fields by your criteria. For your example request, you'd be interested in the following attributes of the parent object: stargazers_count, forks_count. In order to filter the repositories by number of committers, you could use a separate API: https://developer.github.com/v3/repos/#list-contributors.

Updates or comments from people more familiar with GitHub API are welcome!


I'm getting same MAE value in decision tree model for maxdepth value from 1 to 10

In this line:

for(i in 1:10){mae <- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}

You are setting maxdepth to 3 for every i. You have to change it for:

for(i in 1:10){mae <- get_mae(maxdepth = i, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}


In an SVM, does a more negative/positive decision score mean that it is further from the seperating hyperplane?

Larger scores do mean farther from the separating hyperplane.

The SVM model itself isn't modeling probability, so you can't quite say that farther points are considered more probably in one class or the other by the model. There are ways to estimate a probability (see the scikit implementation) for example and yes it will generally be true if so that farther points are more certain, according to that estimate.


I think in a lot of cases, when people are using pre-trained models, they force their words to lower and singular case and then test to see if the lowercase and singular (not plural) version were trained. Sometimes, you have to check yourself like A.M. Bittlingmayer said.

To provide some insight into why sometimes people remove stopwords versus not --- I think it has to do with whether they want to value infrequent words more or less. Because you don't want to value too highly very frequent words, the algorithm 'punishes' extremely frequent words. So, one could argue that because this already happens, you don't need to remove stopwords since they will be 'punished' anyways. So, not everyone removes stopwords.


How to create custom Activation functions in Keras / TensorFlow?

First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function:

from keras import backend as K

def swish(x, beta=1.0):
    return x * K.sigmoid(beta * x)

This allows you to add the activation function to your model like this:

model.add(Conv2D(64, (3, 3)))
model.add(Activation(swish))

If you want to use a string as an alias for your custom function you will have to register the custom object with Keras. It can be done like this:

from keras.utils.generic_utils import get_custom_objects

get_custom_objects().update({'swish': Activation(swish)})

This allows you to add the activation directly to layer by name:

model.add(Conv2D(64, (3, 3), activation='swish'))

For more advanced activation functions, with trainable parameters and such, it is best to implement them as a Keras Layer. Here the swish function is used in a layer, allowing beta to be learned while training:

from keras.layers import Layer

class Swish(Layer):

    def __init__(self, beta=1.0, trainable=False, **kwargs):
        super(Swish, self).__init__(**kwargs)
        self.supports_masking = True
        self.beta = beta
        self.trainable = trainable

    def build(self, input_shape):
        self.beta_factor = K.variable(self.beta,
                                      dtype=K.floatx(),
                                      name='beta_factor')
        if self.trainable:
            self._trainable_weights.append(self.beta_factor)

        super(Swish, self).build(input_shape)

    def call(self, inputs, mask=None):
        return swish(inputs, self.beta_factor)

    def get_config(self):
        config = {'beta': self.get_weights()[0] if self.trainable else self.beta,
                  'trainable': self.trainable}
        base_config = super(Swish, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

    def compute_output_shape(self, input_shape):
        return input_shape

You then would add the activation function the same as any other layer:

model.add(Conv2D(64, (3, 3)))
model.add(Swish(beta=1.0, trainable=True))


How to replace NaN values for image data?

Given you have images stretched out as columns in a table with ~48,500 rows, I am assuming you have the raw images that are 220x220 in dimension.

You can use a function available via OpenCV called inpaint, which will restore missing pixel values (for example black pixels of degraded photos).

Here is an image example. Top-left shows the image with missing values (in black). Top-right shows just the missing values (the mask). Bottom-left and bottom-right are the final output, comparing two different algorithms for filling the images.

I would suggest trying both methods on your images to see what looks best.

Have a look at the Documentation for more details on the algorithms themselves. Here is the documentation of the actual function.

As for code, it will look something like this:

import opencv as cv    # you will need to install OpenCV

dst = cv.inpaint(img, mask, 3, cv.INPAINT_TELEA)

the first argument is your image with missing values
the second is the mask, with locations of where missing pixels are, i.e. which pixels should be filled/interpolated.
third is the radius around missing pixels to fill
fourth is the flag for the algorithm to use (see link above for two alternatives)

For each image, you can generate the mask with something like this:

mask = image[image == np.nan]


If f(x) is a deep network, you could use the differential network implementation of this repository. This differential network adds a few computations to the feed-forward graph to compute the analytical Jacobian w.r.t. to the network input. In contrast to the autograd implementation, the computational overhead is minimal and the Jacobian is computed using forward differentiation. Furthermore, the examples of the Repository show that you want to use softplus activation to get smoother Jacobians.


Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark.

Then, you're using the features available in spark to optimize it.

Reference: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html


It could be your model or your data. You need to perform the experiment of changing your model (while holding data constant) to isolate the causal reason.

Yes - models can get worse with more data. One of the primary reasons is models often have a limited capacity to learn. Simple models can only successfully model simple relationships. Again - if you fit a more complex model, you'll start to understand the mechanism that could be causing this observation.


Is there a paper accomplishing finding physical law from observation without premade perception, using machine learning?

You can check this paper

Discovering physical concepts
with neural networks - https://arxiv.org/pdf/1807.10300.pdf

I quote one of the first sentences

  Here, we present a neural network architecture that can be used to discover physical concepts from experimental data without being provided with additional prior knowledge


This will give you all the  values under <tr>:

bs=BeautifulSoup(data, "lxml")
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    cols=[x.text.strip() for x in cols]
    print cols