How to handle the CEO expectations from a company that's new to data science?

I have not worked as a data scientist.  But managing expectations is something we've all done. Considering you're referring to the company CEO, tact is especially required.  There are three practices that have helped me a lot over the years:

Never say "it can't be done" right away.  Many, many, many times I have said that (as a programmer), only to think of a solution later on.  When I'm skeptical that something can't be done, I've learned to say, "Can I think about it and get back to you?"  That approach has kept egg off my face many-a-time.  It also gives me time to do some research.
Even when something can't be done, focus on what you can provide. If the requested solution is unrealistic for some reason, try to think of what is realistic and offer that.
When discussing what can and cannot be realistically done, clients often don't care about the technical details (even though that's what excites you, and what you want to talk about).  Be ready to go into the technical details if asked, but start off with a very high level explanation.  Rather than focus on the negative details of why something cannot be done, describe the positive details of what it would require to accomplish the task. That shows you have thought it through, and that you're trying to be team player rather than a road block.


Please suggest some good ways to test my RNN

When I built my first vanilla RNN from scratch, I was very curious about it's correctness. So, I trained it on a portion of a sine curve and tried to generate the part of the curve that comes next. And consequently, I plotted mine and the actual to visualise.

(P.S.: here is my code!)


We love the normal form

In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view!

Which Transformation?

The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions.

What makes physical (biological, economic, whatever) sense, for example in terms of limiting behaviour as values get very small or very large?
This question often leads to the use of logarithms.

Can we keep dimensions and units simple and convenient? If possible, we prefer measurement scales that are easy to think about.

The cube root of a volume and the square root of an area both have the dimensions of length, so far from complicating matters, such transformations may simplify them. Reciprocals usually have simple units, as mentioned earlier. Often, however, somewhat complicated units are a sacrifice that has to be made.

When to Use What?

The most useful transformations in introductory data analysis are the reciprocal, logarithm, cube root, square root, and square. In what follows, even when it is not emphasised, it is supposed that transformations are used only over ranges on which they yield (finite) real numbers as results.

Reciprocal: The reciprocal, x to 1/x, with its sibling the negative reciprocal, x to -1/x, is a very strong transformation with a drastic effect on distribution shape. It can not be applied to zero values.  Although it can be applied to negative values, it is not useful unless all values are positive. The reciprocal of a ratio may often be interpreted as easily as the ratio itself:    Example:

population density (people per unit area) becomes area per person
persons per doctor becomes doctors per person
rates of erosion become time to erode a unit depth

(In practice, we might want to multiply or divide the results of taking the reciprocal by some constant, such as 1000 or 10000, to get numbers that are easy to manage, but that itself has no effect on skewness or linearity.)

The reciprocal reverses order among values of the same sign: largest becomes smallest, etc. The negative reciprocal preserves order among values of the same sign.

Logarithm: The logarithm, x log10 x, or x log e x or ln x, or x log 2 x, is a strong    transformation with a major effect on distribution shape. It is    commonly used for reducing right skewness and is   often appropriate    for measured variables. It can not be applied to zero or negative    values. One unit on a logarithmic scale means a multiplication by the    base of logarithms being used. Exponential growth or decline.

$y = a exp (bx)$

is made linear by
- $ln y = ln a + bx$
       so that the response variable y should be logged. (Here exp() means    raising to the power e, approximately 2.71828, that is the base of    natural logarithms).
       An aside on this exponential growth or decline equation:
       $x = 0$, and $y = a exp(0) = a$ so that a is the amount or count when    x = 0. If a and b > 0, then y grows at a faster and faster rate (e.g.    compound interest or unchecked population growth), whereas if a > 0    and b < 0, y declines at a slower and slower rate (e.g. radioactive    decay).

Power functions :
$y = ax^b$ are made linear by $log y = log a + b log x$
so that both variables y and x should be logged. An aside on such power
functions: put $x = 0$, and for $b > 0$,
$y = ax^b = 0$    so the power function for positive b goes through the origin, which    often makes physical or biological or
economic sense. Think:    does    zero for x imply zero for y? This
kind of power function is a    shape    that fits many data sets
rather well.

Consider ratios y = p / q where p and q are both positive in       practice.

Examples are:

Males / Females
Dependants / Workers
Downstream length / Downvalley length

Then y is somewhere between 0 and infinity, or in the last case, between 1 and infinity. If p = q, then y = 1. Such definitions often lead to skewed data, because there is a clear lower limit and no clear upper limit. The logarithm, however, namely
log y = log p / q = log p - log q,
is somewhere between -infinity and infinity and p = q means that log y = 0. Hence the logarithm of such a ratio is likely to be more symmetrically
distributed.

Cube root: The cube root, x 1/3. This is a fairly strong transformation with a substantial effect on distribution shape: it is weaker than the logarithm. It is also used for reducing right skewness, and has the    advantage that it can be applied to zero and negative values.  Note that the cube root of a volume has the units of a length. It is commonly applied to rainfall data.

Applicability to negative values requires a special note. Consider
(2)(2)(2) = 8 and (-2)(-2)(-2) = -8. These examples show that the
cube root of a negative number has negative sign and the same
absolute value as the cube root of the equivalent positive number. A
similar property is possessed by any other root whose power is the
reciprocal of an odd positive integer (powers 1/3, 1/5, 1/7, etc.)
This property is a little delicate. For example, change the power just a
smidgen from 1/3, and we can no longer define the result as a product of
precisely three terms. However, the property is there to be exploited if
useful.

Square root:The square root, x to $x^(1/2)$ = sqrt(x), is a transformation with a moderate effect on distribution shape: it is
weaker than the logarithm
   and the cube root. It is also used for reducing right skewness, and also
   has the advantage that it can be applied to zero values. Note that the
   square root of an area has the units of a length. It is commonly applied
   to counted data, especially if the values are mostly rather small.

Square: The square, x to $x^2$, has a moderate effect on distribution shape and it could be used to reduce left skewness. In
practice, the main reason for   using it is to fit a response by a
quadratic function $y = a + b x + cx^2$. Quadratics have a turning
point, either a maximum or a minimum, although the turning point in a
function fitted to data might be far beyond the limits of the
observations. The distance of a body from an origin is a quadratic if
that body is moving under constant acceleration, which gives a very
clear physical justification for using a quadratic. Otherwise
quadratics are typically used solely because they can mimic a
relationship within the data region. Outside that region they may
behave very poorly, because they take on arbitrarily large values for
extreme values of x, and unless the intercept a is constrained to be
0, they may behave unrealistically close to the origin.

Squaring usually makes sense only if the variable concerned is zero
or positive, given that $(-x)^2$ and $x^2$ are identical.


What are the differences between logistic and linear regression?

As you have mentioned, the output of linear regression is a real value while logistic regression's represents classes(classification). Their main difference is this.

The loss function of linear regression is convex which means you always can find the optimal point using customary optimisations while if you use that for logistic regression, you may get stuck in a non-global minimum which is not optimal. Consequently, people take logarithm of that loss and call it cross entropy. For simple logistic regression tasks, it is convex.

Another difference that has to be cared about is the non-linearity which is usually applied for different tasks. For logistic regression, it is customary that people use non-linearites like Tanh or Sigmoid after the linear part, inner product of weights and inputs, for specifing the similarity to class 1 or 0 for typical binary classifications. For linear regression, people usually use Linear activation function. There is a point here. The idea of using the linear activation is not due to the need for employing a linear function. It is used because its output is not limited. Consequently, you can use other functions that are one-to-one and are not limited. Consider $y = x^3$ as an example.


How to evaluate performance of a new feature in a model?

Informally democracy.

So how many classifiers did it improve, only 1 than dont add it.

Formally there are a couple of statistical tests.

Cochran's Q test

Is a generalisation of the McNemars test for comparing Machine Learning models.

or read this formal paper where they discuss it.


Selecting most important features for multilinear regression

Be careful choosing features based on correlation! Yes it is true that features that are correlated with the response variable may be good predictors, however if the features are correlated with each other then you are introducing multicollinearity into your model, which is bad.
If you want to avoid this you should choose features which are correlated with your response variable but not with each other. For example...
y = a + b + c
Assume:
y is correlated with all three (a, b, c)
a is correlated with b
c is not correlated with a and b
You should only use one of either a or b and c to predict y.
With regard to choosing which model is best, you should use a combination of metrics. As you increase the number of features your R^2 will tend to increase regardless of model performance. AIC and BIC can be used only to compare similar models and the "best" model according to AIC/BIC will be the one with the lowest score relative to the other models.
In my opinion RMSE is the best indicator of a good model, but you should also evaluate R^2, AIC, and BIC.
If you are unsure of which features to use I suggest you try stepwise regression to evaluate many models quickly and (maybe) find the best one.


Database options for JSON storage, queried with Apache Drill

Would be beneficial to know the structure of the data and what you are wanting to do with the data. Do you really need a database at all? Drill will read from a filesystem/S3/etc?

But to answer your question with the information you provided.

The pro of MongoDB over PostgreSQL would be in scaling and read performance (after a certain scale). If this is mainly a read only, or analytic database (primarily reads) then MongoDB would probably work.

PostgreSQL has the advantage of ACID compliance and more than likely better performance if you don't go above the ~10's of millions of records (especially with JSONB, which supports indexes on the datatype).

Without knowing much about the data, you could simply have two columns within your postgresql JSON table (id, JSON_field) and then use the functions within PostgreSQL to query your JSON field. It is a little more involved in querying JSON data within PostgreSQL but after you get used to it, it is pretty simple.


Why class weight is outperforming oversampling?

You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods.

Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for unbalanced datasets) by purposely biasing the model to favor more accurate predictions of the higher weighted class (the minority class).

Oversampling and undersampling methods essentially give more weight to particular classes as well (duplicating observations duplicates the penalty for those particular observations, giving them more influence in the model fit), but due to data splitting that typically takes place in training this will yield slightly different results as well.

SMOTE creates new observations of the minority class by randomly sampling from a set of "similar" minority class observations. Synthesized observations are computed based off adding a random percentage of the difference between two randomly chosen "similar" observations, for each coordinate (i.e. column). Similar observations are defined typically using the k most closest neighbours to a particular observation of the minority class. This means that, depending on the value of k chosen, as well as numerous other factors such as how similar your observations are in general, the distance measure, etc. SMOTE may or may not be useful for your particular problem.

Methods that deal with class imbalance do not all work the same and is a large area of study. It appears you have noticed that the class weight method is more effective, which is why any sort of method chosen in addressing any potential class imbalance problem needs to be wrapped within a model validation scheme to see if the particular method (or really, using any method at all) yields any benefit.


One method you can try first is cosine similarity. It works by counting the number of occurrences of each word in the vocabulary for each individual document. Next, you put these counts into vectors and then you take the cosine of the angle between them.

If you have more than one text on a topic, you can combine them into a single text for the purpose of finding the cosine similarity between various topics.

If you have many texts that are examples of each topic, then you can create a supervised machine learning model. However, since your purpose is interpretation instead of prediction, I would recommend a technique like cosine similarity before getting insight from interpreting a predictive model.


Help needed in interpreting the loss, val_loss vs epoch plots for an autoencoder training?

You tagged your question with vae, so I will assume that this is a variational autoencoder. In that case, your loss has 2 components: reconstruction error and KL divergence of the bottleneck distribution, probably balanced with a weight $\beta$. You should plot both components separately in order to understand how the training dynamics evolve. The reconstruction loss is what will tell you if there's overfitting. The KL divergence will tell you if there is posterior collapse.


Retraining EfficientNet on only 2 classes out of 4

The problem you describe is called transfer learning, which uses uses a pretrained model (such as EfficientNet) that is trained on an existing dataset and applies it to a different dataset. Since this method uses a pretrained model, the new dataset on which the transfer learning needs to happen doesn't need the same number of training examples compared to when the model would be trained from scratch.

In your specific case, you could look into using an existing EfficientNet model that is pretrained of a different dataset (such as ImageNet) and add one or two layers to make the model specific for your dataset (i.e. number of output neurons equal to the number of classes you are trying to classify). You can then train the pretrained model on your specific dataset, however, make sure to start by fixing the parameters of the pretrained model to make sure the model keeps the knowledge it has learnt.


At the end each distribution can be described by a function with parameters. Can be a Gaussian, polynomial etc. In principle you can choose functions that only have one free parameter and fit this one. Depending on your data you might be able to guess a function class that seems to fit the underlying distributions. You can then use the fit parameters as input for your data set. Example: Fit a Gaussian and use mean, normalization and sigma = 3 parameters for your model.


Visualize interval type data graphically and regression

You can visualize intervals using histograms. For regression, try out OLS. Or you can reframe it as a classification task.


calculation of average ROC in IMageNet paper?

I would guess it is using the one vs all approach. You measure the auc of the ROC on each class adding the probabilities of all other classes to create the other class. Then you average among classes the obtained areas under the curve.


I will assume that by "random" you mean that the numbers don't follow any particular mathematical function. If they were really random then there wouldn't be any pattern to discover so there would be no point trying to predict anything.

From your description I understand the following:

The value of the digit doesn't have any numerical property. In particular the natural order doesn't play any role. This suggests that the digits can be considered as categorical variables.
The data is sequential (the order in the sequence of digits matters) but there is no notion of time involved.
Apparently the two devices produce two independent sequences. You might want to check but if this is the case this calls for two distinct models, one for each device (otherwise you should use a single joint model).

Based on these observations I would use a simple sequential model such as Hidden Markov Model or Conditional Random Fields.


There is another reason: words which don't appear in the dictionary. Of course a dictionary approach will correctly stem all the forms which are known in the dictionary, and depending on the language this may indeed lead to better accuracy. However the dictionary approach cannot do anything about unknown words, whereas a generic stemmer can try to apply its generic rules. This can be particularly important with texts which are either very domain-specific (e.g. medicine), which often contain technical words which are not in a general dictionary, or recent user-generated texts such as social media posts where people may use neologisms or words borrowed and sometimes transformed from another language.


multivariate clustering, dimensionality reduction and data scalling for regression

Great question, I will try to answer the aspects related to dimensionality reduction mentioned above.
$Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would still ask you to do a correlation analysis of the variables.

Pca removes these correlated features and gives you a set of features which amount to explain most of your data. One advantage of dimensionality reduction is in regression. Correlated features often are a cause of multicollinearity. Doing a dimensionality reduction helps us get rid of this problem. Also once we have a reduced set of features we can apply the cluster analysis. The reason is K-means calculates the l2- distance between data points. in very high dimensions the concept of euclidean distance becomes less useful because of the curse of dimensionality. (probably the reason of the problems with your elbow curve.)

Hence bringing down the number of features using pca and clustering later will give a better idea of the groupings of the data.
The problem which you mention about PCA can be solved in fact it is not an issue at all. You can do regression with the features obtained by the data and only take the coefficients of the reduced feature set.Some other techniques which you might want to look at are regularization in regression.
For example L1-regularization helps in feature selection and helps with correlated variables.


How do we define a linearly separable problem?

1.) Perceptron is a non-linear transformation!
2.) Linear seperable function is only defined for boolean functions, see Wikipedia.
Therefore, yes, the statement is meant only for binary classification.
3.) For general functions, see the universal approximation theorem.


Which machine learning approach/algorithm do I choose for path validation?

This is a good question but it's rather complicated. I can suggest two approaches:

Graphical models; specifically Bayesian networks since your graph is directed.
Recurrent neural networks. Here's a talk on popular recent model: Sequence to Sequence Learning with Neural Networks.


This part calculates a second derivative. it should be slow and time consuming.

# Perform weight update
with tf.GradientTape() as outer_tape:
    # Calculate gradient penalty loss
    with tf.GradientTape() as inner_tape:

Try

# Perform weight update
with tf.GradientTape() as outer_tape, tf.GradientTape() as inner_tape:


PCA first dimension do not not capture enough variance

You don't state how many original features there are? where there 10, 50, 50 million? The above 10 principle components capture 62.3% of the variance and so we can be sure that there are more that ten original features. However we will assume then that there aren't many more (lets say circa twenty), lets consider what is going on.

With PCA you are hoping to separate the useful information from the noise. By taking only the the first two components in you example you are discarding all other information as noise.

It depends upon the application but I would say that 25% is never enough (I think serious questions would be raised as why you had). So the question becomes then what is going on? Well lets consider the two extreme cases: A set of uncorrelated features and a set of fully correlated features. Consider first three uncorrelated features:

1 0 0
0 1 0
0 0 1

The total variance is 3. The eigenvalues are

1.000
1.000
1.000

Each component therefore represents 1/3 of the variance and they are equal. Now let use consider the other extreme, all features are highly correlated, i.e. a correlation matrix of

1.000 1.000 1.000
1.000 1.000 1.000
1.000 1.000 1.000

The eigenvalues are

3.000
0.000
0.000

Therefore the first principle component contains all the information and these three features could be characterised by a single latent variable.

So why is this relevant? Well in your data set your eigenvalues are all of similar magnitude. This suggests that the features in your original data are not very correlated (like the first example above, look at the off diagonal elements your original correlation matrix they are likely close to zero) and therefore it is expect that many principle components would be required since there is little shared information in the original feature space.

This of course assume that we only have a small number of original features. If you have 50 million features then capturing 62.3% in only ten latent variables is pretty good,


why accuracy on training dataset is 100% using random forest caret , does it indicate overfitting?

Yes, those accuracies dont say anything.

You have overfitted to 310 data points which is pretty easy using RF. The fact is that you said you "predicted" on training data. Well, when you learned on training data and you ask prediction for points you already gave the model it simply knows them.

You need to use a cross validation technique (here i propose Leave-One-Out technique as you dont have much data points). RF is more robust than DT towards overfitting, but that is not applied to this setting. Here you getting an exam in which you ask a question you already gave to the student. He just simply knows the answer.


How is the backbone of two neural networks trained?

In General any sort of Gradient-based learning is done on scalar functions, so those functions f: ℝ^n ↦ ℝ. (In fact this is the meaning of a Gradient). Mainly if you want to define any minimization problem, you need a single value to minimize and not more.
This means: Ultimately your loss always has to be a scalar (a single number). Combining the gradients in the middle (so before backpropping into your backbone) would ultimately be equal to just combining the losses. And a weighted loss will be easier to implement.
For reference, you can look this talk by A. Kaparthy on how Tesla does multi-task learning on a single Backbone and how they deal with combining different losses.


How to tell my neural network that I care much more about precision than recall?

I think you have two options that don't exclude each other:

Check precision/recall metrics constantly during training, on Validation data. In this way, during experimentation and hyperparameter tuning, you will choose models that are more favourable for precision rather than recall.
Customize your loss function to correct the model in the right way. The problem is that precision, recall, F1 score, they are all non differentiable functions, i.e. they cannot enter a Loss function as they are. For that reason, some tried to approximate these metrics in a differentiable way. One example is the soft-F1 score, as explained in this article. If you read the function code, you will find differentiable approximations of TP, TN, FP, FN's. You can copy those lines to customize your personalized pseudo-precision Loss function.


Learn (common) grammar / pattern from set of sample strings?

Here are a few ideas:
If the number of strings is not too high, you could consider taking a formal approach and use a finite automata determinization algorithm (I'm very rusty about this stuff but I clearly remember that there is such a thing). The idea is to start from a big automaton made of the union of all the strings, then use the algorithm to find the deterministic automaton, which can then be converted to a regular expression.
A more data-science-y idea is to use character-based similarity/distance measures between all the pairs of strings. Then it should be possible to identify outliers, maybe through clustering based on the distance. Typical character-based measures: Jaro-Winckler, Levenshtein edit distance.
Finally an original (but possibly bad) idea would be to try to train a (character-based) language model on the strings (assuming there are sufficiently many of them). Given an input string, the language model gives you a probability that this string belongs to the "language", so an outlier could be detected by its low probability.

[addition following OP's comment]
Language modeling is normally used for representing the valid sentences in a language, e.g. English, based on the likelihood of sequences of words in this language. It's trained from a large number of correct sentences so that it can estimate the probability of the $n$-grams of words in this language. This is a common NLP task (example) but in your case you would use characters instead of words and strings instead of sentences, so there would be a small adaptation compared to the examples you'll find.


analogy for pearson r statistics for binary classification task

The Pearson Chi-Squared test may give you what you need:
https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test

This is a good resource for analyzing categorical data:  https://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data

Also, confusion matrices can be very helpful:  https://en.wikipedia.org/wiki/Confusion_matrix


The input of an LSTM is a sequence of vectors. In your case, each of these vectors represents a word encoded as a one-hot vector. One-hot encoding is a way to express a discrete element (e.g. a word) numerically. Each one-hot vector is a vector of length $d$, where $d$ is the total number of words we can represent, and where all positions in the vector are 0 except the position associated with the represented word, which contains a 1.
The hidden state passed to the next LSTM cell is not the final binary prediction, but the dense numerical vectors we obtain before computing the binary prediction.


There are many solutions for this task. I suggest one of them. As you know, words have relation and if you choose to give each word a special code, you can't have this relation. Consequently, first try to use an embedding network in order to assign each word a code. Then assign each article a label. Next, for each article, you have a sequence of words, codes, which are now embedded. You can employ LSTM networks for classification.

If you are not very familiar with the concepts I referred to, you may want to look for Word2Vec.


Is correlation needed when building a model?

Not really, no. Sort of. It depends on how complex your model/data is.

It's entirely possible to have a situation where a feature taken in isolation will not be correlated with the target variable, but multiple features considered together will. This is why univariate correlation is unreliable for feature selection. A trivial case that demonstrates this is a bivariate model performing a binary classification where the positive class is bounded by the right upper and left lower quadrants, and the negative class is bounded by the left upper and right lower quadranta (i.e. the "XOR" pattern):

If the input features have the same sign (x>0 & y>0 or x<0 & y<0), it's the positive class, else it's the negative class. But either feature in isolation is completely useless and uncorrelated with the target.

Additionally, modern models like deep neural networks are effectively capable of "learning their own features", i.e. constructing extremely complex features by developing abstractions from the raw inputs. The "final" features learned by such a model will likely be correlated with the target, but the input features need not be.

For example, if you consider the imagenet task (classifying a photo as a member of one of 10,000 classes), I'd be very surprised to learn that there's any correlation between the values of specific pixels and any target class. That's just not how we interpret photos. The value of the pixel in position [25, 10] should not have any correlation with whether or not the picture is a photo of a dog. But, if we think of the entire network before the output layer as a feature engineering system (such that the "classifier" is just the output layer), then the "features" provided by the penultimate layer (the inputs to "The Classifier") probably have some correlation with the target.

TL;DR: If a feature is correlated with the target, it probably contains information about the target that will be useful for modeling. But that does not mean uncorrelated features are useless. Reporting correlation when it's there is a simple way to demonstrate that there's a signal in that variable. But lack of correlation doesn't necessarily mean you should throw those features away. In fact, correlation doesn't even mean you should necessarily use that feature either: you can have multiple features correlated with the target that are highly correlated with each other, in which case you would probably only want one or a handful of that group in your model.


Why there is only one type of artificial neuron?

The example you cited (using x^2 instead of x) is the idea more popular outside deep learning community, called feature engineering. The trend in neural network modeling is instead to,

Play with weights (w) and fine tune them.
Not change the input vector (x) but feed it to the network directly.
If a single layer neural network is not good enough, add more layers.
Introduce non-linearities using activation functions.
And in general, not hand-roll features (like x^2) but let neural networks discover such features.


If image is a numpy object:

image = image.reshape((50000,3,32,32))

and then print:

print(image[:,3,:,:])


How to factorize the Matrix in TensorFlow? (Recommender System)

If R is the rating matrix, U is the user matrix and M is the movie matrix, then note that there is almost certainly no matrix M that satsifes $R = UM$. U and M are too low rank. However you should be able to find the matrix M that minimizes $|R - UM|$.

There is no need to use an optimizer, though you could I guess, because this is a convex problem. You're just solving a large linear system. This is indeed just what ALS does repeatedly.

You've found an ALS solver and if you just need to solve one step and have the user matrix already, I think you just supply it as row_init and run one iteration? haven't used it, but conceptually that's all you are doing. You don't need weights either.


Best way to classify plots which are overlapping?

A quick answer is since you have time series data, LSTM is generally appropriate if you would like to model/exploit time dependencies between values in each series.  LSTMs are now combined with CNNs for even better performance.  You can refer to this Q&A for that: What is the best method for classification of time series data?Should I use LSTM or any other method.  The only problem is that it appears that you have an extremely small dataset, which normally leads to overfitting with neural networks given the number of parameters they need to compute.  And there is actually a lower bound on the number of training samples needed for LSTM (see: Number of parameters in an LSTM model).  Perhaps you can find pre-trained RNNs/LSTMs trained on similar data that you can use.

In general, any classification algorithm (especially those that perform well on time series data) can (and probably should) be tried.  With SVM, I hope you've tried different kernels.  There is at least this one paper that discusses the relative merits of different SVM kernels for time series analysis: http://www-ai.cs.uni-dortmund.de/EVENTS/FGML2001/FGML2001-Paper-Rueping.pdf.

If you have not already done so, I also recommend that you research feature extraction strategies for time series data (and particularly, EEG data) that have worked for others.  There is at least one Q&A on this topic on the Cross Validated site: https://stats.stackexchange.com/questions/66027/time-series-classification-very-poor-results.


How to release datasets with fingerprinting

"Digital watermarking" is a set of techniques that might be useful in this context.

From the wikipedia page:

  "Watermarking" is the process of hiding digital information in a carrier signal... Digital watermarks may be used to verify the authenticity or integrity of the carrier signal or to show the identity of its owners. [Emphasis added]

To address your requirement, you would insert a unique watermark for each client that receives your data. Watermarking techniques address requirements such as robustness to modification and imperceptibility.

As an example, this paper talks about watermarking numerical data: "Watermarking Numerical Data in the Presence of Noise", pdf


No registered env with id: BanditTenArmedGaussian-v0 for the package gym_bandits of OpenAI

The standard installation of OpenAI environments does not support Windows. See the GitHub issue here.

To get that code to run, you'll have to switch another operating system (i.e., Linux or Mac) or use a non-standard installation.


Can I use scipy.optimize module with PyTorch?

Here's a workaround I'm currently using:
# model definition
class MLP(Module):
    # define model elements
    def __init__(self):
        super(MLP, self).__init__()
        self.hidden1 = Linear(2, 200)
        self.hidden2 = Linear(200, 100)
        self.hidden3 = Linear(100, 1)
        self.activation1 = LeakyReLU()
        self.activation2 = LeakyReLU()

    # forward propagate input
    def forward(self, X):
        optimize_flag = False
        if not torch.is_tensor(X):
            optimize_flag = True
            X = Variable(torch.from_numpy(X)).float()
        X = self.hidden1(X)
        X = self.activation1(X)
        X = self.hidden2(X)
        X = self.activation2(X)
        X = self.hidden3(X)
        if optimize_flag:
            return X.detach()
        return X

Just make sure when you are using scipy.optimize.minimize or any other function that uses the x0 flag to input to pass in a detached tensor (i.e. minimize(surrogate, x0=x.detach().numpy()))


Why is activation needed at all in neural network?

Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and where the neuron of interest is. In principle the activation function $f$ would go to the calculation of the outcome

$$ y = f(Wx + b)$$

For output neurons, if you are doing classification, then $f$ should map between 0 and 1, since you'll interpret the outcome as a probability. For regression $f$ could be just the identity.

For hidden (i.e. non-output neurons), you definitely want to use a non-linear $f$. The reason is that the neural network would otherwise be equivalent to a regular linear model. So the non-linear activations are needed to harvest the expressive power of neural networks.

For deep learning the most popular $f$ for hidden neurons would probably be the rectified linear unit (relu)

$$ f(x) = \max(0,x)$$


Why is deep learning used in recommender systems?

"Recommender Systems" is a very broad area and can be approached from different optics: latent variable models, graph models, etc.
"Deep learning" is an umbrella term for gradient-based optimization of deep differentiable models, and has been used to model all sorts of supervised learning problems, including graphs and latent variable models. Therefore, it has been applied to many different areas, with a great degree of success.
Multiple deep learning approaches have been applied to recommender systems because deep learning performed well in many other areas, so why not expect it to perform well for recommender systems.
Neural approaches have proven to be effective also for recommender systems. That is why they are used nowadays.
Also, different inductive biases have been applied to neural networks in other areas and, in some cases, such inductive biases are also useful for recommendation, like the sequential bias for recurrent networks, or the structured bias for graph neural networks.
If you want to explore the academic publications related to neural recommender systems, I would suggest that you start with the article Neural collaborative filtering and the survey Deep learning based recommender system: A survey and new perspectives.
Also, in order to get a broad view of the state of the art in recommender systems, you may have a look at papers with code, where you can see the best-performing systems for different benchmark datasets, with links to the associated articles and source code repositories.


Can Reinforcement learning be applied in image classification?

Yes, it can- if you have a criterion of optimality for your problem. Basically, if you can asses automatically, during the learning process, is your new result better than the result of the previous iteration.

Some examples:

https://arxiv.org/pdf/1802.02904.pdf

https://arxiv.org/pdf/1712.07465.pdf

http://www.maths.lth.se/sminchisescu/media/papers/mps-rlcvpr16.pdf

https://arxiv.org/pdf/1804.04603.pdf


Why does an imbalanced data set badly effect distance measures like Mahalanobis?

Mahalanobis distance is defined as a distance between a point and a distribution. The key is how you define the distribution and I would say the imbalance of classses is not the problem in itself here. What could be a problem is mahalanobis is sensitive to initialization and the sample sizes of your classes are not that big. You could check the covariance matrix if it is reasonable for you task.
About more general question -- the distance between two points obviously does not depend on size of the classes. If we talk about distance between a point and a set, then it could affect the result: i.e. you defined the distance as the distance to the closest point in set, then obviously the more points, the bigger chance to get the closer point.


Array of categorical variables vs one-hot encoding

Yes, it will work. Basically by creating the encoding, scikit-learn's label encoder does the same thing, you are creating such more features each representing the presence or absence of that level in your sample. To represent d levels you need d-1 variables, so in your case that is what is exactly happening. However since you said, the number of levels can be massive it will not be a good idea to use a vector that massive. It would be a good idea to run some preliminary analysis and check if some levels are totally useless and dont relate to the response as well, some might have the exact same effect and hence those levels can be combined.


Having trouble understanding None in the summary of my Keras model

The output of the first layer is (None, 62, 62, 32), but isn't in
  supposed to be (62, 62, 32)? Where does the None come from?

The None is just a placeholder saying that the network can input more than one sample at the time. If your input shape was (64, 64, 3) it would mean that your network could one hold one image at the same time. But you want to train with batches of varying length or predict several images at the same time, that's why it's (None, 64, 64, 3).

As an example, if you would send in a batch of 10 images your shape would be (10, 64, 64, 3) and if you changed it to 25 you would have (25, 64, 64, 3).

  Also since my images are (64, 64, 3) doesn't it mean that the filter
  should have 3 dimensions to cater for the color channels of the image?

Technically you could use a 3D filter with Conv3D on images. However, it is not something people do for RGB images. Probably because it gives worse results than using 2D. Instead you treat RGB as three separate 2D channels rather than one 3D channel. Using three dimensions is usually more relevant in video where you instead use time as your third dimension. Here is a Quora where it is discussed more at length:

Why do we use 2D convolution for RGB images which are in fact 3D data cubes? Shouldn't we use 3D convolution for RGB and 4D for video?


The answer is unfortunately no. There is no handy ctrl-z method.

A tip to avoid these situations: I suggest you always save either the 'environment' or, as I prefer to do, the scripts with the codes for the desired objects, and save them regularly. I never type any commands directly into a work space, but always in a script which I save. This is so I can always look back at which steps I took in my coding. Personally, I save my scripts with date notations in the file name - so when I change a script I'll still have access to the old version of the scripts. It has happened quite a few times that I've changed a code and later realized that the old code was better. It's quite a hassle when you don't have the old file saved.


The first question is very broad, 95% accuracy doesn't say a lot, there are tasks where 99% is standard and there are tasks where an accuracy of 60% is amazing.
It is hard to answer this question without more information about your model, dataset and the affects of its outcome. Basically you should ask yourself if it is good enough for your task.

Regarding the second question, I would suggest to use a weighted loss function, where mistakes on the rarer classes have a higher price (it encourages the model to classify them correctly). For optimal performance, you should retrain your model from scratch.

Note for future reference, it is better to present the results in a confusion matrix, than just the final classification (gives more data, especially when we talk about a skewed dataset).


You need to be very careful with your assumptions about "important" parameters. All the parameters have their function, that is precisely why they are implemented on the first place. If with importance you mean potential to affect your outputs/efficiency, perhaps there are some more relevant than others but most likely this relevance will also depend on the nature of your task and you data.

Having said that, I recommend you navigate through the scikit-learn documentation which is usually helpful and it contains a description including all these parameters:

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Moreover, they usually include nice and simple usage examples that can be helpful: here and here.

It is important that you practice and practice and you will start to get a feeling about the importance of each parameter and its relevance to the task at hand.

Finally, there are hundreds of blogs with nice tutorials that will give you a little bit more of insights about parameter tuning and "importance" ;)

You can try this once for example.

Good luck!


Error when checking target: expected Output to have 2 dimensions, but got array with shape (631, 80, 2641)

Alright, so... I got help from someone. He mentions changing LSTM to return_sequences.

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

encoder_outputs = encoder(embed)

# # Average
# average = Average()([state_h, state_c])

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(encoder_outputs)

The average layer is still wrong, but it's the close thing. I might have to edit it later.


In unsupervised learning,the learning procedure is finding similarity between training samples, and putting similar items into a same cluster,
training phase in unsupervised learning produce some sets with similar items.
Then,in test phase the similarity is calculated for all items of each set, and check if the test item is similar to each cluster's items or not, if it was similar to at least one item, the test item belongs to that cluster.

SO... YES, we need training set and test set, the training set helps to find thresholds to find similarity


On the internet, LabelImg is pretty popular. It's a Python program that you can use to automatize drawing of bounding boxes. Alternatively there is HyperLabel, on their website they say it's free so it.s worth to give it a try.

The options available are so many I completely get lost while searching. I suggest you to read this nice review of tools first.

With these tools, labeling 1600 images is not going to be a problem IMHO. Good luck.


Conceptual questions on MLP and Perceptrons

When the data is linearly inseparable, we use MLP. Here what is meant by "data"--is it the response or the input feature that is linearly inseparable?

This means that a linear function of the input features is unable to separate the response.

To answer your question a bit more directly: Given only a linear function of the inputs, the response is the thing that's inseparable.

  If it is linearly inseparable then does it mean that the mapping function from input to output will always be non-linear? Hence, we prefer MLP or the latest new models such as deep learning?

Yes.  If the mapping from input to output were linear, then the output would necessarily be linearly separable by the input.

  Linear regression fails in the case of linearly inseparable data or can linear regression work for inseparable data but if the function mapping is nonlinear then it fails?

Linear regression will never be able to perfectly separate linearly inseparable data.  Consider the following example, where the input features are x1 and x2, and the output is the color:

It doesn't matter how you draw a line in the 2D space - you'll never be able to separate the colors.  The same idea applies in higher dimensions.

I hope that helps!


Sure, of course we can use TF-IDF or bag of words. The easiest way is to build a separate TFIDF transformer for every group of keywords and then combine them together using FeatureUnion. Just make sure the custom tokenizer only filters out the keywords in each group that you're looking at.

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "china factory activity shrinks some text here",
    "first time some text here",
    "2 years some text here",
    "combining first time and 2 years here.",
    "adding china factory activity first but doesn't match exactly"
]

# custom tokenizers

keywords1 = lambda x: [x for x in x.split() if x in ["china", "factory", "activity", "shrinks"]]
keywords2 = lambda x: [x for x in x.split() if x in ["first", "time"]]
keywords3 = lambda x: [x for x in x.split() if x in ["2", "years"]]

combined_features = FeatureUnion([
    ('kwd1', TfidfVectorizer(tokenizer=keywords1)),
    ('kwd2', TfidfVectorizer(tokenizer=keywords2)),
    ('kwd3', TfidfVectorizer(tokenizer=keywords3))
])

pipeline = Pipeline([('bow', combined_features)])
output_corpus = pipeline.fit_transform(corpus)

For the deep-learning approach, I would recommend a similar kind of pattern so that the embedding learned from each group would have a different semantic meaning from another group. You'll just need to ensure that the tokenizer only considers the keywords from each group when it is processed. If you're using a pre-build word embedding; you'll just have a filter before it goes into each embedding so that it doesn't "leak" into another topic.

Example using Keras:

from keras import layers
from keras.models import Model
from keras.preprocessing.sequence import pad_sequences

def custom_sequence_tokenizer(txt, vocab):
    vocab_mapper = dict([(word, idx+1) for idx, word in enumerate(vocab)])
    text_split = [vocab_mapper.get(x, 0) for x in txt.split() if x in vocab]
    return text_split

keywords1 = pad_sequences([custom_sequence_tokenizer(x, ["china", "factory", "activity", "shrinks"]) for x in corpus], padding='post', maxlen=10)
keywords2 = pad_sequences([custom_sequence_tokenizer(x, ["first", "time"]) for x in corpus], padding='post', maxlen=10)
keywords3 = pad_sequences([custom_sequence_tokenizer(x, ["2", "years"]) for x in corpus], padding='post', maxlen=10)

input_count1 = layers.Input(shape=(10,), name='in1')
input_count2 = layers.Input(shape=(10,), name='in2')
input_count3 = layers.Input(shape=(10,), name='in3')

# input size is reflection of the vocab size
embed1 = layers.Embedding(5, 8)(input_count1)
embed2 = layers.Embedding(3, 8)(input_count2)
embed3 = layers.Embedding(3, 8)(input_count3)

combine = layers.Concatenate()([embed1, embed2, embed3])

model = Model(inputs=[input_count1, input_count2, input_count3], outputs=combine)
output = model.predict({
    'in1': keywords1,
    'in2': keywords2,
    'in3': keywords3,
})


How does ML algorithms treat unseen data, a conceptual discussion

Define two flag variables: flag_is_cat and flag_is_dog, which take on values of 1 if the picture shows a cat or dog, respectively, and 0 otherwise. Define another flag that takes on the value of 1 if the picture contains either a cat or a dog. In a word, label the data.

If you train the model using all of the pictures, even those with neither a cat nor a dog, then the model outputs a probability that the picture contains a cat, a probability that it contains a dog, and another that it contains either. This is the approach mentioned by @marco_gorelli . Dividing the probability that the picture has a cat by the probability that it has either a cat or a dog gives the probability that the picture has a cat conditional on the picture having at least one of them.

Alternatively, if you train a model using only those pictures that contain either a cat or a dog, then the model would output the probability that a cat is contained in the picture and that a dog is contained in the picture conditional on at least one of them being in the picture.


Reframing multilabel classification with imbalance in "both" directions

I don't think there's any obviously best option. I'd suggest trying a few reasonable solutions, evaluate on a development set and then pick the one which performs best. Don't forget to try with the original data as it is, resampling doesn't always work better.

Upsample the non-target class? Downsample the target class?

These two options are very likely to give the same result (assuming you use the same proportion).

Reframe the problem as "not drama" to maintain underbalance for every label?

For the classifier it would be exactly the same.


XGBoost, binary classification: uneven number of observations per user

You may want to consider a stratified cross-validation approach where you specifically undersample the data from users that have lots of observations or oversample from uses with fewer observations.

Another approach which may also work is to duplicate the observations of some of the users with less observations, this would even out the weighting of the different kinds of observations.

-- Adding on to answer the additional questions in the comments

There are a number of ways to undersample the majority or oversample the minority.  Here's an overview of how to handle the situation (in general).

To undersample the majority, simply remove some % of the training data that is in the majority for each run.

To oversample the minority, duplicate the values that you want to have a higher weight.

Additionally, there is a parameter for XGBoost (scale_pos_weight) which allows you to set the weight of the different samples, see here.  Additionally you could leverage SMOTE from the imblearn library.


Splitting train/test sets by an identifier?

In Auriel Geron's book, there is a short description of the approach:

you   could compute a hash    of  each    instance’s  identifier, keep    only    the last    byte    of  the hash,   and put
the   instance    in  the test    set if  this    value   is  lower   or  equal   to  51  (~20%   of  256).   This    ensures
that  the test    set will    remain  consistent  across  multiple    runs,   even    if  you refresh the dataset.
The   new test    set will    contain 20% of  the new instances,  but it  will    not contain any instance
that  was previously  in  the training    set.

While a full explanation of what exactly happens and why is probably best placed on StackOverflow, I can try to answer your questions, first with some background info.
The method uses a cyclic redundancy check, which is a method of checking that the raw blocks of memory have not been damaged/changed. It is a way to ensure data integrity, e.g. in network traffic - checking if a message way altered between being sent and received.
For train/test splits, it is checking the unique identifier of each sample. We have a column that gives each sample an ID - this should never be changed! Don't delete rows, only append to the end with new unique IDs.
In this part: test_ratio * 2**32, the part $2^{32}$ represents the largest integer of a 32-bit system.

0xFFFFFFFF is a large number; it's the hexadecimal representation of $2^{32}-1$

To answer your questions:

What is the 3rd line doing:

crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

Based on the information I gave above, we see the crc32 function finds the checksum value in memory (of the unique identifier). If we know the unique ID has never changed, then We ensure that crc32(np.int64(identifier)) & 0xffffffff will always return exactly the same numeric value, across all Python versions and platforms.
Imagine we give IDs in the range 0-80 for train, and 81-100 for test. No we want to make sure a sample'd s ID falls in the first bucket. We check its ID is simple less that 81, right? Well the numeric value we made above is checked to be less than our test_ratio * 2**32, where 2**32 is the largest 32-bit number. It checks that the sample's ID is within the range of train data, not in the test bucket:: > test_ratio * 2**32.

What is the anonymous function doing in the 2nd to last line?

lambda id_: test_set_check(id_, test_ratio)

This simply applies our test_set_check function to each sample's unique identifiers. Using the apply methd on a Pandas Series object (here it is one column of a Pandas DataFrame).

In practice, do you commonly split datasets by ID in this manner?

Not really... Scikit-Learn's train_test_split is often good enough. I think there are many other ways to remove bias and errors from your models before worrying too much about the impact of random splits.
For example, the snoop bias, whereby you analyse the entire dataset yourself before deciding on a model architecture/pipeline, thereby incorporating knowledge of the entuire distribution, which is inherently biasing our model.
There is also bias in overfitting e.g. in sequential imaging data (think frames of videos) such that the background is consistent, even though the objects you might want to detect are not. Your model will learn what to expect based on the background, which is not robust! Here you might look into using a geographical split (not random at all).

On a side note, there is also a slightly robuster way of setting random seeds in Python (instead of using NumPy's random seed generator). Have a look here for some differences.

Helpful resources:

https://stackoverflow.com/questions/36819849/detect-int32-overflow-using-0xffffffff-masking-in-python
https://pynative.com/python-random-module/
https://stackoverflow.com/questions/30092226/how-to-calculate-crc32-with-python-to-match-online-results
https://stackoverflow.com/questions/49331030/bitwise-xor-0xffffffff/49332291#49332291


Extract imperative sentences from a document(English) using NLP in python

In order to maximize accuracy you would need to use not only a POS tagger but also a syntactic parser. Nevertheless for this task POS tags can probably give you reasonable results indeed, here is a general method:

Segment the data into sentences and tokens
Apply the POS tagger (it predicts a POS tag for every token)
A sentence is (likely) imperative if the following conditions are satisfied:

the sentence ends with a full stop or exclamation mark
the POS for the first token corresponds to a verb

This heuristic is probably all you need, but if you want to go further you could generate instances containing these features (and possibly add a few others)  for every sentence, annotate a training set and train a supervised model.


Why n-split is not possible for a dataframe with KFold?

There is a typo error in the below line. Change KFold to Kfold below and try.

score=cross_val_score(clf,train1,test1,cv=KFold,n_jobs=1,scoring=scoring)


How could a considerable increase in loss leads to an improvement in accuracy?

Check whether padded values get accounted while computing accuracy, which shouldn't be the case. You should create mask for padded values and use that while computing accuracy. For instance, while the y_true value for padded region is 0, and 0 as well for y_pred, then y_true == y_pred counts as a correct prediction and impacts the overall accuracy. This is wrong, and what you should instead do is define a custom accuracy metric that does the following:
correct = y_pred == y_true
mask = tf.cast(tf.cast(y_true, tf.bool), tf.int16)
accuracy = tf.reduce_sum(correct * mask)/tf.reduce_sum(mask)

Hopefully this should give you a correct measure.


You could use R to apply time series analysis to your data, i.e. as described here.


What are helpful annotation tools (if any)

Doccano is an open source simpler alternative to Prodigy. Its native python via Django. I found it suitable for simple implementations.


Recommender system based on purchase history, not ratings

You could use Content based filtering but then you have to intelligently pre process the data to extract all the contents of the products. Also, that might lead to leaving a some features, This article is a great head start after you preprocess all the data.

Also, you could make pseudo ratings for product vs a customer. That would depend on your problem statement. Some few suggestions could be the number of times the customer bought the particular product in last one month or you could also take an index which would define how frequently the customer buys that product which, mathematically would be last_two_purchases/interval_of_purchase or could also take an average of last few purchases and intervals.

After making this pseudo rating you could convert all the content based features into numerical ones and use Latent factor model for collaborative filtering. Refer this video. Python could be used for this.


Is it possible to use TensorFlow inside a python script in Azure Machine Learning Studio?

Let me answer my own question: yes it is possible. You simple open this example in ML Studio and you'll get a working tensorflow_keras.zip into your workspace: https://gallery.azure.ai/Experiment/CNN-on-Keras-with-Tensorflow-backend


Can we use DecisionTreeClassifier of sklearn for continuous target variable?

In the Wine Dataset you linked, the quality column is not a continues variable but a discrete. It takes integer value between 0 and 10.

When you use the DecisionTreeClassifier, you make the assumption that your target variable is a multi-class one with the values 0,1,2,3,4,5,6,7,8,9,10. So, the model tries to predict one of these and only these values.

When you use the DecisionTreeRegressor, the assumption is that any number between 0 and 10 is acceptable. Like the number 4.52356. As a result, the accuracy will be noticeable worst. If you still want to use the Regressor for some reason, you can try to round the outcome and then calculate the accuracy. Keep in mind that RMSA doesn't fit your problem. You have a multi-class and not a regression model.


Finding an appropriate price using thousands of data points

It is not clear from your question if you are trying to predict value for many used cars as a function of different vehicle characteristics (like age of car, model, mileage) or predicting price of a single car.

Assuming the former (prediction using variables), you can set it up as a regression problem, where price is dependent variable and (age, model, mileage) are independent variables. Google "Boston housing price prediction" which is a similar application in different domain.

If you have only prices data (ignoring vehicle characteristics), you are looking for summary statistics. You can start with the mean (average) or median price. You can then it little more complex by modelling the price as some reasonable statistical distribution (normal or Poisson) and then take typical summary statistics on the distribution as estimate (say inter-quantile range).


We can look at the source for guidance.

  How does word2vec handle the input word being in the context?

It is skipped; for both the skip-gram and CBOW models.

  If word2vec encounters the same word multiple times in the same window, what occurs?

The relationship is strengthened.


Predict customer action from previous buying history

You can have a variety of solutions, starting from very simple to a more complex and beneficial ones.

I suggest that you'll start with the simple solutions, jain much of the benefit and continue by need.

Note that though the problem can be treated as a supervised learning problem, it is a multi label problem (a customer can be interested in many services). Therefore, decision tree or logistic regression is not suitable (unless you have a handful of services and you try to predict if the user would like a service, for each service).

The classical method to provide such recommendations are "People that like X, also liked Y". Though very simple, at time 50% of Amazon's revenues where due to that.

You should make the recommandation more sepcific by considering the lift - Harri Potter is very popular and many people like it. However, it isn't the most sutabile recommandation for someone reading browsing books.

You can take such a item-to-item recommendation and use all user history by aggregating using Naive Bayes or similar methods.
You can take care of temporality and dates by giving more eight to recent data.

The next step is moving to recommendation system. You can find on the web some implementation.
The winner in the Netflix challenge used matrix factorization algorithm, so this direction is very popular for recommendation systems.


Orange Import and process spectra time dependent/kinetics

To accomplish this you can use the Orange Concatenate Widget under the Data tab shown below:

This widget takes a primary data set and an additional dataset and concatenates them into one dataset. Once all of your data points are contained in the same dataset, you should have no problem plotting them all together.


You can do:

 t, _ , _ = "01:00:00".split(":")

and then:

int(t)

That will return you: 1.

Putting this together as a function would be:

def time_to_integer(time):
    t, _,_ = time.split(":")
    return int(t)

Please note that you should pass the time variable as string. For example:

time_to_integer("02:00:00")

Would return: 2


Transfer learning VGG16 does not work as expected. (Detect tacos as hamburgers)

You are almost there as you mentioned the addition of third class as random photos. Generally what happens when you train a neural network, it assumes the entire domain of images existing as the training data. For  your case each image can be either cat or burger. Now you have not shown random images to network and hence when a image which is neither cat nor burger, network will try it's best to predict it as either cat or burger. Instead I would suggest you to train network as burger vs not burger. In not burger class have all possible kind of images that are not burger. You can use CIFAR 1000 data to get images of different classes. Now your model will learn to differentiate between burger and not burger properly. Obviously you can still find some cases where miss classification happening and the reason will be because you have not covered such images in training.

Also play a bit with softmax threshold to get high presicion and recall.


y_score can either be probability estimates of the positive
    class, confidence values, or non-thresholded measure of decisions

y_score = model.predict_proba(x)[:,1]

AUC = roc_auc_score(y, y_score) # Above 0.5 is good


How do you ascertain the quality of your data?

This entirely depends on what you wish or aim to do with your data, and what you mean by your tag data-cleaning. This can involve the technical process of sanitising data i.e. fixing broken XML, but also the manual deletion of outliers etc.

Are you going to perform statistics to measure the correlation between certain variables?

Or:
Are you going to include the data into a machine-learning solution?

In any case, you can plot descriptive statistics to get an impression of the quality of your data, such as:

Box-plots to detect outliers
Standard deviation, mean, mode, median to assess if it's normally distributed or not
More, don't know from the top of my head

Remember that those measures give you an impression of the quality of your data, but even good measurements can come from bad data, and possibly vice versa.

In short, the best way of assessing the quality of your data is to assess the quality of your results, which are acquired generally after several experiments. Only then will you see how well your data is suited for the problem you are trying to solve. That - or you need to get your hands dirty and examine the data qualitatively before you use it.

Data (in assumingly large quantities) are generally used to identify certain patterns within the phenomena described by the data. It is very important to describe your task in terms of a (research) question and a proposed solution or hypothesis. Then, you can assess whether the data is suitable for your particular (research) project. Keep in mind that, theoretically, you can try to answer numerous questions on the same data, or answer the same question on different data sets. The goal is the right balance.


Error: An operation has `None` for gradient with categorical_crossentropy

This link might be helpful to you. you might be missing something in your code.

https://github.com/keras-team/keras/issues/12521


Does Convolution kernel size affect number of channels?

Normally a convolutional neural network will get flattened into a single column vector after the convolutions and then maybe be processed by dense layer. In this model, the convolution $1\times1$ is used as an output layer. It will have $C$ channels like every other layer, but it is not dilatated. Hence, you can use this layer as the input to other convolutional neural networks.

The kernel size will not influence the channels. Imagine an RGB image with 4 by 4 pixels. If we have a $2\times 2$ convolution with $2\times 2$ stride we will get an output of dimension $3\times 2 \times 2$ (without padding). Hence the channels do not change. If we have $K$ filters we will get $K$ times a $3\times 2 \times 2$ output. If the kernel size is $4\times 4$ with a stride of $2\times 2$ we will get a $3\times 1 \times 1$ (without padding) output for each of the $K$ filters. The kernel size only influences how large the receptive field of the convolutions are. Hence, it only influences how the layer is scaling the individual dimensions of the width and height (by using RGB images as an example).

If you flatten the output of a layer you will always reduce its dimensionality to $1$. In the dilated convolutional neural network they do not have a layer that flattens the input.


I think the first point to address is how employers will know you are being honest about your logging. Maybe a better use of your time is to not put in number of hours, but projects and code references you've made so that the output is more concrete rather than just a number of hours.  The problem is I know plenty of people who put in 1 hour and learn what a normal person would learn in 5 hours. Think about the WOW factor in this.. what would be convincing to employers?

Also, if its only you that are logging in hours and nobody else is going to see or use that program, you might as well use excel while focusing on whatever project you're passionate about.


xgboost cannot identify perfectly fitting regression line

I think that the reason for this to happen is that tree-based methods have problems with linear problems. This is because tree-based methods do partitions of the variables, and not on combinations of the variables.  To fit a linear regression, a tree-based method will have to do a lot of partitions to obtain low error. However, in principle, using enough deep trees you should be able to overfit your training data, although it might take many trees.

If your concern is to make a perfect forecast, no tree based method is able to do a perfect forecast, and this happens with most kind of data. As your data is linear, you happen to be able to do a perfect forecast with linear regression, but this won't happen in real life.


Technically this is sequence labeling, the most common application being Named Entity Recognition.
However it looks like in this case you're trying to solve a problem of coreference resolution, which is a quite difficult task in general. I think this usually involves a more complex model than simple sequence labeling, but I'm not an expert in this. You might want to search around this topic, there are certainly some relevant papers and tools about it.


Minimizing an upper bound of objective function

In many cases, the real function that you want to minimize is something that you cannot reasonably compute explicitly (e.g. in Bayesian statistics, you oftentimes have a formula integrating over all the possible combinations of something, which are too much for even the best supercomputer), or for which there is no known formula or procedure that can minimize it directly  (e.g. in discrete optimization).

In that case, if you cannot compute or minimize the function itself, it makes sense to minimize something that either correlates highly with the function (e.g. a sampled estimate, or some approximation that replaces true values with other estimations), or an upper-bound (as long as it is a reasonable) - in the second case, you know that if you take this upper bound from something like $10^{20}$ to $10^-3$, the initial parameters would very likely have resulted in a worse value for the function that the ones you obtained by minimizing the upper bound.

In some cases, it can be proved that the optimal value of the function is within some percentage of the solution that maximizes the upper bound (quite common in discrete optimization), but this is oftentimes not the case (e.g. in Bayesian statistics), and what you get is NOT BY ANY MEANS the true global optimum for the original function, but something that is nevertheless better than random and feasible to calculate on your computer.

Now, you might argue that there are black-box methods such as evolutionary search that could minimize pretty much any function that you can evaluate directly, but these oftentimes lead to worse solutions than minimizing the upper bound, require more function evaluations, are much, much slower, and impractical if you have too many parameters, while the upper bounds can oftentimes be reasonably minimized with faster methods that rely on assumptions such as differentiability, smoothness, Lipschitz continuity, etc.

In short: the optimal solution for the original function and for the upper bound are indeed different, but if you cannot minimize the original function directly, you can still get a decent solution (not the best one) by minimizing the upper bound.


How do you know when you are using a multi gpu?

If you are using Linux and Nvidia GPUs, you can do the following in a terminal

nvidia-smi

Which will show you some stats about the GPUs available on your system.

You can run it automatically every 2 seconds like this to see how power/memory usage changes during training:

watch -n 2 nvidia-smi

If you see nothing or the nvidia-smi command fails, you likely don't have the correct drivers installed.

If theya re showing, but Keras/Tensorflow is not finding them, have a look at this thread for more checks for Tensorflow backend.


Metrics to measure imbalanced multi-class problem

I see two maybe 3 options here.

1.) Don't use class_weight = 'balanced'.

Instead use
class_weight = {0 : 0.01447 , 1 : 0.095 , 2 : .890}

2.) over sample the other classes using some like SMOTE.

3.) (possibly) use a model to classify label 2 only in a one-vs-rest scheme, then use a second model classify label 1 or 0. Kinda in a stacked approach. Becareful not to leak information when training though.


K-Means clustering - What to do if a cluster has 0 elements?

If you get an empty cluster, it has no center of mass. You can simply ignore this cluster (set k=k-1 for next iteration), or repeat the k-means run from a new initialization. You can also choose to place a random data point into that cluster and carry on with the algorithm if you must have this specific number of K clusters.

If it keeps happening, there is a good chance that your K parameter is too high.
You should iterate over a few different values for k and pick the best.

Here is how an empty cluster might happen:

Consider the following example with 7 data points. The different shapes of the points are just to show that there are actually 2 natural clusters (but we do not know that before running the algorithm). In this case we choose k = 3.

Three random cluster centers are initialized. At the end of first iteration points 3, 1, 2, and 7 will be in one cluster. 4 and 5 will be in another cluster. And 6 will be in the last cluster. Note here that the distance between 3 and 4 is larger than the distance between 4 and 5 and so 4 is assigned to the cluster represented by 5. Before we begin the second iteration we update the cluster centers and the following picture shows the centers and the clusters at the end of first step.

Now, the cluster center for the red cluster moved closer to point 4 due to 1, 2, and 7. And the cluster center for the blue cluster moved away from 5 due to point 4. In the next iteration point 4 will decide that it is closer to the red cluster and point 5 will decide that it is closer to the green cluster. This will cause blue cluster to be empty as shown below.

The images were taken from the following link: K-Means Empty Cluster Example


How to interpret the output of XGBoost importance?

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees.

The meaning of the importance data table is as follows:

The Gain implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.
The Cover metric means the relative number of observations related to this feature. For example, if you have 100 observations, 4 features and 3 trees, and suppose feature1 is used to decide the leaf node for 10, 5, and 2 observations in tree1, tree2 and tree3 respectively; then the metric will count cover for this feature as 10+5+2 = 17 observations. This will be calculated for all the 4 features and the cover will be 17 expressed as a percentage for all features' cover metrics.
The Frequency (/'Frequence') is the percentage representing the relative number of times a particular feature occurs in the trees of the model. In the above example, if feature1 occurred in 2 splits, 1 split and 3 splits in each of tree1, tree2 and tree3; then the weightage for feature1 will be 2+1+3 = 6. The frequency for feature1 is calculated as its percentage weight over weights of all features.

The Gain is the most relevant attribute to interpret the relative importance of each feature.

The measures are all relative and hence all sum up to one, an example from a fitted xgboost model in R is:

> sum(importance$Frequence)
[1] 1
> sum(importance$Cover)
[1] 1
> sum(importance$Gain)
[1] 1


Interesting question!

Short (but maybe naive) answer

Experiment with both options and see which performs best!

Longer answer

predict the drag coefficient, and then calculate the resistance accordingly

If you do this, your network will try to optimize something different than your actual goal (which is the resistance). This means that your model will not "care" if the resistance you eventually calculate is any good, which can result in strange results.

predict the resistance directly

This would be better from a machine learning perspective as your model's goal will be the same as yours, however, you will lose the advantage that you have by knowing how the resistance is calculated.

Solution A

Predict both and then have a final step to decide what your final resistance will be. With LSTM, this is definitely possible, your target will just become 2 numbers instead of 1.

Solution B

The best solution, in my opinion, would be to have the LSTM output a single number (which would act as the drag coefficient), and then, add a layer which calculates the resistance using the known formula so that you can backpropagate on the entire thing, and you get the best of both worlds. In PyTorch this can be done rather elegantly. The big caveat is that the formula to calculate the resistance needs to be differentiable.


How to run R scripts without closing X11

This looks like a similar kind of a problem.

Solutions: (taken from above source)

Just sleep via  Sys.sleep(10) which would wait ten seconds.
Wait for user input via readLines(stdin()) or something like that [untested]
Use the tcltk package which comes with R and is available on all platforms to pop up a window the user has to click to make the click disappear. That solution has been posted a few times over the years on r-help.

2nd option is better to use for the user.

P.S. Since I did not come up with the answer myself, I tried to put it in comment but my reputation is too low for that.


Validation fluctuating on keras training

Maybe the distribution of the validation data is not similar to the training data, and therefore the training signal does not lead the model to perform well on the validation data.

The key point here is therefore: how did you split the data into training and validation?


Word representation that gives more weight to terms frequent in corpus?

I'm not aware of any standard representation which increases the importance of document-frequent words, but IDF can simply be reverted: instead of the usual
$$idf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \in d|}\right)$$
you could use the following:
$$revidf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \notin d|}\right)$$
However for the task you describe I would be tempted to try some more advanced feature engineering, typically by using features which represent how close the distribution of words in the current document is from the average distribution.


Let's work it out from the ground up. Classification (also known as categorization) is an example of supervised learning. In supervised learning you have:

model - something that approximates internal structure in your data, enabling you to reason about it and make useful predictions (e.g. predict class of an object); normally model has parameters that you want to "learn"
training and testing datasets - sets of objects that you use for training your model (finding good values for parameters) and further evaluating
training and classification algorithms - first describes how to learn model from training dataset, second shows how to derive class of a new object given trained model

Now let's take a simple case of spam classification. Your training dataset is a corpus of emails + corresponding labels - "spam" or "not spam". Testing dataset has the same structure, but made from some independent emails (normally one just splits his dataset and makes, say, 9/10 of it to be used for training and 1/10 - for testing). One way to model emails is to represent each of them as a set (bag) of words. If we assume that words are independent of each other, we can use Naive Bayes classifier, that is, calculate prior probabilities for each word and each class (training algorithm) and then apply Bayes theorem to find posterior probability of a new document to belong to particular class.

So, basically we have:

raw model + training set + training algorithm -> trained model
trained model + classification algorithm + new object -> object label

Now note that we represented our objects (documents) as a bag of words. But is the only way? In fact, we can extract much more from raw text. For example, instead of words as is we can use their stems or lemmas, throw out noisy stop words, add POS tags of words, extract named entities or even explore HTML structure of the document. In fact, more general representation of a document (and, in general, any object) is a feature vector. E.g. for text:

actor, analogue, bad, burn, ..., NOUN, VERB, ADJ, ..., in_bold, ... | label
    0,        0,   1,    1, ...,    5,    7,   2, ...,       2, ... | not spam
    0,        1,   0,    0, ...,    3,   12,  10, ...,       0, ... | spam

Here first row is a list of possible features and subsequent rows show how many times that feature occurs in a document. E.g. in first document there's no occurrences of word "actor", 1 occurrence of word "burn", 5 nouns, 2 adjectives and 2 pieces of text in bold. Last column corresponds to a resulting class label.

Using feature vector you can incorporate any properties of your texts. Though finding good set of features may take some time.

And what about model and algorithms? Are we bound to Naive Bayes. Not at all. logistic regression, SVM, decision trees - just to mention few popular classifiers. (Note, that we say "classifier" in most cases we mean model + corresponding algorithms for training and classification).

As for implementation, you can divide task into 2 parts:

Features extraction - transforming raw texts into feature vectors.
Object classification - building and applying model.

First point is well worked out in many NLP libraries. Second is about machine learning, so, depending on your dataset, you can use either Weka, or MLlib.


Active learning on image classification model

Yes, that is possible. In transfer learning you have at least three options to handle transferred weights:

Freeze transferred weights
Retrain transferred weights from scratch
Fine tune transferred weights (further training but not from scratch)

The 3rd one is what you intend to do. And of course different options can be applied to different layers. Also see this paper:

  The usual transfer learning approach is to train a base network and then copy its first n layers to
  the first n layers of a target network. The remaining layers of the target network are then randomly
  initialized and trained toward the target task. One can choose to backpropagate the errors from
  the new task into the base (copied) features to fine-tune them to the new task, or the transferred
  feature layers can be left frozen, meaning that they do not change during training on the new task.
  The choice of whether or not to fine-tune the first n layers of the target network depends on the
  size of the target dataset and the number of parameters in the first n layers. If the target dataset is
  small and the number of parameters is large, fine-tuning may result in overfitting, so the features
  are often left frozen. On the other hand, if the target dataset is large or the number of parameters is
  small, so that overfitting is not a problem, then the base features can be fine-tuned to the new task
  to improve performance. Of course, if the target dataset is very large, there would be little need to
  transfer because the lower level filters could just be learned from scratch on the target dataset.

I have marked the sentence addressing your question bold.


What should I master better for professional data science in economics and finance?

welcome to the forum. I‘m a trained economist, I do a lot of econometrics, and I work in research. My opinion (and it is only opinion of course) is that you should focus on R in the first place and consider learning Python. Both is not so hard after all.

Why? R is for free and it offers a lot of support for econometrics (it is well regarded in the community). It also is well regarded in the ML community and you can work with high spec things such as Keras, LightGBM etc. So with R you can‘t be wrong. Many top research papers in economics are done with R. Stata is good for some things (like Panel), but there are so many things you cannot do with Stata. R is more powerful.

Why Python? Python and R are „similar“ wrt what you can do in econometrics/statistics. But Python offers more flexibility in many aspects (some might disagree but this is how I see it). In essence, Python is a thing that offers you a lot of possibilities on top of working with data. So have a look at it. Can be very beneficial for your career.

Data science is an extremely broad field. But if you focus on econometrics/statistics, many other things become relevant as well in the moment. Examples are: working with „big“ data and getting a structure into data (e.g. from online sources etc). I also think that neural nets and tree based things such as boosting will become more relevant in economics in the future. Actually outside the academic world, these methods are used quite a lot in the context of economic problems.

Final note: If you think that you cannot do both (R and Python) it would still be fine to have some knowledge/experience in both. You grow with your problems. Even if you have only little experience, you can advance fast if you know where/how to start. This can also be good wrt getting a good job.

P.S. For Python there are also some good packages/applications for economists: https://lectures.quantecon.org/py/


Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

what is the difference between slope of line and slope of curve

It is really a matter of perspective. The slope of a line is the same over the entire span of that line, i.e. until the line changes direction. The slope of a curve is like the slope of millions of tiny lines all connected, so the slope is only the same value over tiny spans. So we can only talk about the slope of a curve at a give point (e.g. a given x value) and then we normally talk about the gradient of the line at that point.

Using your words, the gradient computed by numpy.gradient is the slope of a curve, using the differences of consecutive values.
However, you might like to imagine that your changes, when measured over smaller and smaller distances, become the slope (by your definitions). So when e.g. x2 - x1 almost reaches zero, you have the same meaning as $dx$.
Coarse example with 10 points
Here is a coarse example of numpy.gradient, where $dx$ has s size of 1 (equally spaced values):
In [1]: import matplotlib.pyplot as plt
In [2]: import numpy as np
In [3]: N = 10                                  # Use ten samples
In [4]: x = np.linspace(0, np.pi*2, N)          # Equally spaced x values
In [5]: y = np.sin(x)                           # Corresponding sine values
In [6]: grads = np.gradient(y)                  # compute the gradients

Plotting the values and the gradients - I shifted the gradients by 0.5 to the right, so their values line up with the middle of the segment for which they describe the slope:
In [7]: fig, ax = plt.subplots()
In [8]: ax.plot(x, y, "-b", label="values")                  # the y values
        ax.plot(x + 0.5, grads, "--r", label="gradients")    # the computed gradients
        plt.legend()
In [9]: plt.show()

Fine example with 1,000,000 points
Now we do example same as before, just use one million points: N = 1_000_000.
The blue line would look much more like a true sine wave, but the red line is now measuring the gradient with a much higher resolution than before, giving us the exact value at 1,000,000 points - for each tiny line segment of the blue line.

So the gradient values look like they are all zero! Well this is just because we made x2-x1 become almost zero (1 / 1e6), so the values of y2 - y1 also were essentially zero! We have started to approximate $\frac{dy}{dx}$.
Let change the axis scale to see that the gradients do still match the pattern we might expects, and are indeed very smooth - looking like a curve:

Much better  :)
(Notice the scale difference - 1e-6).

Here are a few explanations of what np.gradient really does.


If you hear from the area under the curve (AUC), you can find that the first classifier is better as AUC of the first curve is more than the second classifier. To know more about AUC you can find this post useful.


RCNN - recurrent convolutional neural networks - what is a time step?

A couple of vocab words:

layer: An entire set of receptive units with shared weights that creates a convolutional feature map

Unit: a single nxn neighborhood filter operator in a given layer of a CNN (say 3x3 or 5x5, whatever given filter size you are using)

stride length: The distance, or padding, in between the center of every unit of a layer

In a convolutional layer, the convolution operation can be thought of like a sliding window, in which the filter from a given layer is applied to the top left corner patch of pixels, then slid to the left 1 horizontal stride length and repeated until the end of the row, then it hops down 1 vertical stride length and starts again, and so on and so forth. This is implemented in a CNN as a 2D grid of units, all sharing the same weights for convolution (this produces the same result as a sliding window).

The RCL takes the sliding window concept of convolution and adds memory to it, so the filter within each unit will be modified based on the response of the previous unit. The time step means how many strides back you want to directly include in the calculation of the current filter. So, if I have an RCL with a filter size of 5x5, and a stride of 1 for an image size of 100x100, a single convolutional layer (assuming it is padded for the filter radius) will have a grid of 100x100 units. Utilizing a time step of N means that you will include response results from the previous N units on the grid to calculate the new unit response (or, more accurately, the unit weights in backprop).


What are some possible reasons that your multiclass classifier is classifying alll the classes in a single class?

I found the reason finally. I was using the unstandardized X_test.
Thanks.

Edit:

previously i defined the y_pred like so;

y_pred = clf.predict (xtest)

and then constructed the confusion matrix like so;

cm = confusion_matrix (y_test, y_pred)

However, i forgot that i previously changed xtest using standard scalar like so;

x_test_std = sc.transform (xtest)

and that the new x_test that should be used is xtest_std not x_test.

When i realized this and used the proper x_test_std, everything worked and made much more sense.


What is the classical way to visualize 3D filters in convolutional neural networks?

With CNNs, a common strategy is to visualize the weights. They are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network.

If you want to have a look at the activation maps, you will have to display them 1 by 1 and they will be grayscale (e.g. you would display 15 AxB grayscale filters for your layer N). Check this site for more ideas: http://cs231n.github.io/understanding-cnn/

Having said that, it is unclear what you can expect to "see" on layers that are looking at a 200*200*10 cube and what action you can take afterwards.

EDIT - Additional link
This article describe a new technique that tries to show the "meaning" included in the neural network layers: https://distill.pub/2018/building-blocks/ "For instance, by combining feature visualization (what is a neuron looking for?) with attribution (how does it affect the output?), we can explore how the network decides between labels like Labrador retriever and tiger cat."


Using multiple machine learning algorithms together

You can train multiple machine learning models with same data and based on accuracy and confusion_matrix response you can decide which one to use.

In theory, you don't need to combine/merge two different ML Models (you can enhance your pre-processing)
but if you still want to use different models there are two approaches:

Ensemble (available with three categories - a) Bagging b) Boosting c) Stacking
Hybrid (This approach allow users to create own models (or use existing) and combine them for better prediction)

Note: You need to be careful with individual algorithm response before combining them together :)


The column you are using must be of string data type. First filter the dataset where it contain "-" and create another dataframe. On this new dataframe perform split string operation and create two different column and convert them to integer. Now create another column which has average or whatever you prefer with two column. Rename column and drop unnecessary columns.  Merge it back to dataframe.


In Keras, how to get 3D input and 3D output for LSTM layers

Add another dimension to represent the number of time steps.

From Keras documentation on recurrent layers:

  if return_sequences: 3D tensor with shape  (batch_size, timesteps,
  units).

Something like this:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 10, 3), return_sequences=True))


You can use ML/DA in almost every sector, but to generalise there are few domains which i would like to list

Broadly it is classified based on data

Text data
Image data
Signal/Periodic Data
Financial data/Numbers

In here we can further classify or give names to the domains. Text data broadly falls under Natural Language Processing (NLP). Here you can work with text generation, text segmentation, document classification, chatbots, topic modelling, Summary extraction & much more
NLP Research,
NLP Application I have added links to 2 articles listing current research & applications by current industry standards.

Image data is broadly falls under Computer vision. Here you work with videos, live streams, images. They can be satellite images, road accidents, machine parts etc. You can do analysis of football match, or even do analysis on plastic pollution in oceans. Sub topics can be classified as

Image tagging
Image based information gathering
Object prediction
Vision based health prediction
Biological researches
Image rectification using NNs

And much more. If you want to have hands-on experience for image based data, please visit this site, here you will get basic idea of what topics one can achieve with image data.

Periodic data consists of measurements from machines, observations of how they work, it also includes biological data. Basically our normal signal processing or risk analysis. It's application is much needed for machinery makers, health sectors & financial brokers, vehicle positioning etc. Anything which has some time based data into it can be used to simplify routes, analyse risks & avoid unnecessary wear & tear.

But generally this domains are used together with each other unless you go for research. Working for a solution firm i have encountered many projects where i had to gather text data from image & use topic modelling for classification. Had a project where i had to analyse data generated from video cam observing human to predict their HR for next few seconds.

Hope this helps you in identifying what domain interests you more.


Azure automl time series forecasting error

The data either has duplicates or you may have some rows with empty values. I removed my rows at the bottom that had empty values separated by comma which resolved the issue