How can I infer psychological intentions from email corpus using text mining?

DISCLAIMER : I'm not sure that this will be the answer you are looking for but I'll give a try.

According to Noam Chomsky

  Language is a process of free creation; its laws and principles are
  fixed, but the manner in which the principles of generation are used
  is free and infinitely varied. Even the interpretation and use of
  words involves a process of free creation.

On one hand, data or text mining the combined large lexical data resources of linguistics and psychology suggests that the linguistic variable of 'word frequency' should be rehabilitated in psychology.

But you have to remember that, correlation does not imply causation.

This said, there is not much way to apply cross text mining with psychological intuition. In other words, use cases are not too many.

You can always try quantitative text analysis. It is a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of, usually, a casual personal text for the purpose of psychological profiling etc.

And by psychological profiling, I'm talking about offender profiling per example.

You can get more information upon that on the following websites :

Criminal Investigative Research & Analysis
http://www.psychlotron.org.uk/ article  about criminal profiling.
Here is a very intersting article about criminal profiling for beginners.

You can also try to take a look at this presentation by Harold Baize who talks about the intersection of text mining in the content analysis of psychology.

I hope this can help!


Grouping domain specific words/phrases with same meaning

I suggest you use word2vec for that task. Word2vec is an unsupervised algorithm that calculates N-dimension embeddings for the words in the corpus used for learning. Basically, it gives you a numerical representation in the form of an n-dimension array for each of the words you use in your inputs.

Once the model is built and the emeddings are available, to find similar words is as easy as calculate the similarity between arrays. This is done in word2vec either by looking for the most similar:

>>>result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])
>>>print("{}: {:.4f}".format(*result[0]))
queen: 0.7699

or directly comparing terms:

>>> result = word_vectors.similar_by_word("cat")
>>> print("{}: {:.4f}".format(*result[0]))
dog: 0.8798

Have a look here for more examples.

To answer this :

  I want to do this for texts that contain domain specific and
  colloquial jargon so existing NLP solutions may not be suitable?

If you train the model with your own domain specific corpus, you should't have any problems; provided of course that it is long and rich enough.


You are specifying the input_shape inside the Conv2D layer, meaning:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))

Instead you should write:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu'), input_shape = input_shape))

or more clearly:
model.add(TimeDistributed(Conv2D(...), input_shape=input_shape))

Furthermore I should note that you are using a TimeDistributed with a Conv2D. This implies that your input_shape should be like this (timesteps, dim1_size, dim2_size, n_channels).

You are using: input_shape=(img_width, img_height, 3)

If you want to take the img_width as timesteps you should use TimeDistributed with Conv1D.

To summarize, always consider that a TimeDistibuted layer adds an extra dimension to the input_shape of its argument-layer.

Lastly, your first LSTM layer with return_sequences=False will raise an error. You must give it a True value.


Understanding Youtube recommender (candidate generation step)

$v_j$ is the learned vector of weights (dimension 256) that connect the last hidden layer (dimension 256) to the corresponding output node for video class $j$. The last hidden layer is the user embedding vector $u$.

The paper uses a vocabulary $V$ of 1M video classes so the deep neural net learns 1M vectors of weights that connect the last hidden layer to each class: $[v_1, \ldots, v_{1M}]$. Negative sampling is used to train such a model with so many classes.

Below softmax function on page 2 section 3.1 with vj and $u$.

$$P(w_t=i|U,C) = \frac{e^{v_i u}}{\sum_{j \in V e^{v_{j}u}}}$$

I believe $v_j$ is "thought of as a separate output video embedding" because $v_j$ can be interpreted as a compressed representation of video j in a 256 dimensional vector space.

Since $e^x$ is monotonically increasing, we just care about the dot product $v_iu$ in the numerator of equation above. This is why "the scoring problem reduces to a nearest neighbor search in the dot product space" where we simply find the nearest neighbors $v$ to $u$.


How to check and correct misspelling in the data of pairs of words?

Another way to reframe the problem is approximate/fuzzy string matching. FuzzyWuzzy is an approximate/fuzzy string matching package for Python. The package can find the nearest element in a set to a user's inputted value with a level of confidence. Something like this:

from fuzzywuzzy import process

ports = {"Houston", "New York", "Newark", "Long Beach"}
user_input = "Strong Beach"
best_guess = process.extractOne(user_input, ports)
print(f"The best match for '{user_input}' is '{best_guess[0]}' which is a {best_guess[1]}% match.")

  The best match for 'Strong Beach' is 'Long Beach' which is a 82% match.


Coding Problem - Extracting values from a column and forming a new dataframe [edited]

I wrote you two small functions which you can use to unpack a dataframe.

The original dataframe looks like this

import pandas as pd
df = pd.DataFrame(data = temp['data']['weather'])
df.head()

The first one is simple, it takes a dataframe and the name of a column, and it will extract the column into a new dataframe.

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]
    df = pd.DataFrame(data = data)
    return df

df_astronomy = extract_col_as_df(df, 'astronomy')
df_astronomy.head()

df_astronomy = extract_col_as_df(df, 'hourly')
df_astronomy.head()

The fact that the extracted tables sometimes have a useless list of dictionaries with a single value bothered me. Of course you can use the same function as above to extract that column as another dataframe, but then you would have a dataframe with a single column, why not just unpack it in place. So I wrote another function which cleans up the extracted dataframe by unpacking a list of dictionaries with a single value.

A better version

This version also takes a dataframe and a column name to extract a dataframe from it. But from the extracted dataframe if a column contains a list of dictionaries with only a single value it unpacks it.

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]

    data = []
    for datum in df[column_name]:
        record = {}
        for i in datum[0]:
            # If the entry in the record is comprised of a list with a
            # dictionary containing a single value then unpack it
            if type(datum[0][i]) is list:
                if len(datum[0][i]) == 1:
                    key_name = list(datum[0][i][0].keys())[0]
                    record.update({i: datum[0][i][0][key_name]})
                else:
                    record.update({i: datum[0][i]})
            else:
                record.update({i: datum[0][i]})
        data.append(record)

    df = pd.DataFrame(data = data)
    return df

df_astronomy = extract_col_as_df(df, 'hourly')
df_astronomy.head()


Does the predict function in machine learning understand categorical data

This depends somewhat on the model and language (implementation).

First please understand that categorical data is not the same as non-numerical data! Many models can handle categorical data (e.g. regression formats) just fine and some can even handle non-numerical data.

Finally and most important for you feature engineering has to be done on the whole data set before the train/test split. All models can only predict on data that has the exact same input formats as the data it has been trained on!

So yes if you one-hot encoded some column it also needs to be one-hot encoded for the prediction.


Is supervised machine learning by definition predictive?

A description as any statistic drawn from your sample data, say the sample mean, quantiles, etc.. Inference is a conclusion drawn from your sample data about the population, e.g., rejecting or accepting some hypothesis or stating that a model is suitable or not for describing your data. Prediction is simply a guess about future observations, which hopefully uses your data and some function/model of the data in a way to formulate that guess.

Both unsupervised and supervised learning methods aim to learn a function of the data that predicts another variable (typically called y) so both are drawing an inference (i.e., a model is well suited to describe your data, see the first sentence here). However, these two methods differ in what data is available. In supervised learning you are able to use an observed sample of y for training your model and in unsupervised learning, y is unobserved.

Hope that helps!


What is a product most frequently bought together with

Two approaches, both based on discrete mathematics. The first one is a set-theoretic approach to frequent item-set mining and association rule mining. For instance A-priori algorithm works here.

the next approach is creating a graph based on orders and do graph mining on them. A-priori might be a better approach as in this case you probably need to deal with hypergraphs which makes life a bit complicated.

I suggest you first extract sets of different orders from your dataframe. As your problem is not general and you already know that there is only association to one product is important, you can prune your data much. Then simply rank other products bought together with Bike Helmet by counting number of their occurences and dividing by number of all (it gives you the probability estimate of each product is bought together with bike helmet).


A rule of thumb, as you go deeper, number of filters increase and the size of filter remains same or increases. You don't follow both of them. This will help your network learn.
Then, consider increasing the number of filters in proper fashion if still your network is not learning.


recommendation system for eCommerce healthcare portal suggestion

Your recommendation system will be designed to tell the customer what product they should choose, however, this doesn't account for what products the customer likes.  A ML method could take all the input parameters from the recommendation system and provide a recommended product based on what similar users liked.

There's no specific ML technique for considering HIPPA constraints.  This would come into play more in the pre-processing stage.  For example, you might not be able to acquire specific birth dates and addresses, but maybe you can get an age range and a zip code (first 3 digits only).

Logistic regression is frequently used in healthcare for binary classification.

There are lots of tutorials online for building ML models.  Kaggle has a good tutorial for Python and R using titanic survival data.


Getting answers to bullets (numbered items) from text via NLP

If we want to search for What are the steps to create a website, all you need to do is the find which set of bulleted points is going to answer that. One thing that you can try is that you can vectorize your question using some pretrained model (w2v, GloVe etc) and also encode the descriptions before the bulleted list ( For eg : How to create a website). Once you have the query vector $q$ and the descriptions vectors $d1, d2, d3,..., dn$ , you can perform cosine similarity of the query vector with all the description vectors and choose the description with the maximum similarity. Following this, the corresponding bullet points will be the answer.


Filter row depending on specific object value and delete those instances

You can replace ? with nan and use dropna(). This will work if you don't already have rows with nan entries that you want to keep.

train = train.replace('?', np.nan).dropna()

Another option is to filter rows where any value is ?.

train = train[~(train == '?').any(axis=1)]

Update:

After looking at your data I found the problem: Your csv file has spaces after the commas, so the rows containing ? have a leading space.

If you use train = pd.read_csv('adult.data', sep=', ', engine='python') to read your data then either of the above methods will work.


Extrapolating GLM coefficients for year a product was sold into future years?

If you suspect your response is linear with year, then put year in as a numeric term in your model rather than a categorical.

Extrapolation is then perfectly valid based on the usual assumptions of the GLM family. Make sure you correctly get the errors on your extrapolated estimates.

Just extrapolating the parameters from a categorical variable is wrong for a number of reasons. The first one I can think of is that there may be more observations in some years than others, so any linear extrapolation needs to weight those year's estimates more. Just eyeballing a line - or even fitting a line to the coefficients - won't do this.


How to predict specific user from session logs?

There are certainly different ways to approach the problem, I am giving you what I think.

The main key is to create as many as features as you can, well it depends on what algorithm you may end up using, but still more features won't hurt. After all you want to differentiate each user, so your feature space should be rich enough to be able to distinct them from each other. I think you are on the right track. For example, there are many features can be extracted from date (weekday, weekend, month etc.), or the browser you mentioned, if it is multiple browser you can create a new feature num_broswers (count number of distinct browser that user uses). You better know your data. Are sessions equally spaced on time or not? For instance, could be that user_one exits a session faster than user_two? Then total time spent for each user on each session could be another feature (engagement factor). You mentioned time in your columns, but I would not know what it is. You can even take time_of_day as another feature (morning, mid-day, afternoon, evening etc.).

You need also to clarify in your question whether number of users are fixed for a specific period of time, or constantly changing?

After addressing key points about data and features, I am thinking of two algorithmic approaches:

Similarity-based Method:
This is exactly what you mentioned, and it is a valid approach and perhaps the fastest and most straightforward one. As mentioned above, you can make new features like num_browsers for multiple browsers entries for a user in those 40 sessions, which was your major concern. I am more concerned that naturally you may have a mix of numerical and categorical features, and I would not be very sure if normal KNN (Euclidean metric) would work! Doesn't hurt trying though after encoding those categorical features or normalizing features are not continuous, but check this post, or another one, or at least be aware of pitfalls!!
Classification-based Method:
You could even turn this into a Multiclass Classification problem, i.e. classifying which user the new incoming user is based on features. The challenge is that the quality of your classifier strongly depends on your feature space. The choice of the algorithms does not matter (even kNN again can be used, but perhaps Decision Trees i.e. RandomForest would be better). If you won't have a good set of distinctive features, classifier can not do magic! This could be also true about kNN, but since kNN doesn't need training, you may get more reasonable results than a classifier.

P.S. This method would be more suitable "if number of users are fixed". Still it would be possible to train a classifier if number of users are NOT fixed. It is ONLY then you need to re-train whenever you have a new user or user_ids are changing, otherwise your algorithm have not seen such user and likely it is going to miss it!

I would personally go with the Similarity-based Method and try to enhance my features as much as I can.


How does backpropagation works through Max Pooling layer when doing a batch?

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way

During back propagation, those activation values are used as one of the numerical sources to calculate gradients, along with gradients calculated so far working backwards and the connecting weights. Like forward propagation, back propagation is applied per example, it does not work with averaged or summed values. Only when all examples have been processed do you work with the summed or averaged gradients for the batch.

This applies equally to max pool layers. Not only do you know what the output from the pooling layer for each example in the batch was, but you can look at the preceding layer and determine which input to the pool was the maximum.

Mathematically, and avoiding the need to define indices for NN layers and neurons, the rule can be expressed like this

The forward function is $m = max(a,b)$
We know $\frac{\partial J}{\partial m}$ for some target function J (in the neural network that will be the loss function we want to minimise, and we are assuming we have backpropagated to this point already)
We want to know $\frac{\partial J}{\partial a}$ and $\frac{\partial J}{\partial b}$
If $a > b$

Locally,* $m = a$. So $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$
Locally,* $m$ does not depend on $b$. So $\frac{\partial J}{\partial b} = 0$

Therefore $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$ if $a > b$, else $\frac{\partial J}{\partial a} = 0$
and $\frac{\partial J}{\partial b} = \frac{\partial J}{\partial m}$ if $b > a$, else $\frac{\partial J}{\partial b} = 0$

When back propagation goes across a max pooling layer, the gradient is processed per example and assigned only to the input from the previous layer that was the maximum. Other inputs get zero gradient. When this is batched it is no different, it is just processed per example, maybe in parallel. Across a whole batch this can mean that more than one, maybe all, of the input activations to the max pool get some share of the gradient - each from a different subset of examples in the batch.

* Locally -> when making only infinitesimal changes to $m$.

** Technically, if $a=b$ exactly then we have a discontinuity, but in practice we can ignore that without issues when training a neural network.


Is it true that Machine Learning is basically the study of Pattern recognition along with Optimization?

Maybe? As Wikipedia says:

  The terms pattern recognition, machine learning, data mining and
  knowledge discovery in databases (KDD) are hard to separate, as they
  largely overlap in their scope.

https://en.wikipedia.org/wiki/Pattern_recognition


Decision Tree gives 100% accuracy - what am I doing wrong?

Machine Learning is one of the few things where 99% is excellent and
  100% is terrible.

Well, I cannot prove this because I don't have your data, but probably:

the test data is included in the training data.

To check this possibility, here's a hint:

print(X_test[X_test.isin(X_train)])

will print all the rows in X_test that appear in X_train. Can you think what to do with that information? :)

Or

you forgot to remove the target variable y from X.

To check this possibility, simply type:

print(X_train.columns)

and check whether one of the columns matches the name of the target variable.


How do SOMs make visualization of high-dimension data possible?

Consider a data set of 1000 patterns, each with 100 features. It is impossible to plot this data. When this data is run through SOM, the network learns the weights such that each of the neurons summarizes a subset of the vectors. After the training is complete, similar patterns get mapped to neighborhood neurons and farther apart neurons represent dissimilar patterns. Visualizing this mapping tells which patterns lie where, in a 2D space.

For example, in the world poverty map[1][2], each pattern is a vector of length 39 and represents one country. When this data set is run through SOM, the countries(patterns) are mapped to nearest neurons and it can be seen that the countries are arranged together based on their economic wellness.

After SOM is run until the stopping criterion is met, each pattern/input vector is mapped to the best matching unit. Hence, for example, Belgium (BEL) is closest to the weight vector of the first node in the map (top-right corner) and hence, mapped to it.

Each node is colored based on the average distance of its weight vector with the weight vectors of its neighbors (U-matrix).

However, it is still not clear how dimensionality reduction is achieved.


Semi-supervised learning and in particular active learning could be considered in cases like this:

The general semi-supervised setting consists in training a model from an initially small training set by applying it iteratively to unlabelled instances. There are various methods to minimize the risk of training the model on wrongly classified instances.
Active learning is a variant of semi-supervised learning where the model queries the human expert for annotations, but the instances are carefully selected in order to minimize the amount of human labor.
There is also bootstrapping, where one would focus on the positive instances: apply the original model to the unlabelled data, the manually annotate only the instances which are predicted as positive (useful only in cases where the positive class is much smaller than the negative one).


You basically have data with a panel structure ($i=$shops, $t=$ weeks). So for a simple clustering problem, you need to "tell" the algorithm that observations for each $i$ (shops) are not independent.

I think you are on the right way to treat each week $t$ as an own variable/feature here. In this case you would have 416 features and 1000 observations. This should be "okay", but especially with non-supervised problems, it is up to a try.

In any case, I would also try to do a little aggregation on the data (over time). Maybe there are good reasons to do this by month or quarter (or some other sort of "season"), meaning to sum up the weekly values over some period of time. Say you do this for quarters, you would have "only" 32 features, which should be absolutely fine with 1000 observations.

See also this post: https://stats.stackexchange.com/questions/88320/cluster-analysis-on-panel-data


what could it be a good descriptor for simple greyscale symbols?

If you can take a quick look with photoshop and get information on the shade of gray and zoom into the objects and count a pixel pattern.

You would be able to build an algorithm around the color 0-255 in 8-bit grayscale and create a simple matrix for each symbol based on the number of pixels you counted.

 110, 0,   0,   0,   0
 110, 0,   0,   0,   0
 0,  110, 110, 110,  0
 0,   0,   110, 110, 0
 0,   0,   110, 110, 0

You can see a shape emerge from the matrix...that is what you train on, but you do so by having the script read pixel by pixel and create a matrix.

This would be a gray (110) on black (0)  shape. If you only have a few, you could go into photoshop and cut the symbols out at the exact shape and feed the shape into your model and train it to look for that specific color and shape.

You might need to build an equation to scale it up or down so that it could account for using the identifier at different sizes or on files with different pixel depth.

you could also do this in R or python by reading a file in and localizing that data, but it would be easier to bring in exactly the shapes as separate files.
You might also need to give the model some leeway as to the darkness like from 100-120 depending on how the images enter the system.

If they are digital and unmodified straight out of CAD this is likely unnecessary, but if they are scanned or grabbed online, you will not have consistent exact numerical color-depth and need to account for the variation in your algorithm.


Is there a Feature selection process for ARIMA model?

In some sense, it is common to do feature selection before you fit the ARIMA model, or at the very least, it is natural (in my opinion).

The problem is that there seems to be little development in automatic feature selection techniques for statistical time series models that can use exogenous variables (like ARIMA). Thus, it is not clear as to how we can do feature selection. To make things worse, auto.arima doesn't do any feature selection on exogenous variables, it just uses AICc to find the most optimal order of your model (in a stepwise fashion in its default setting). If you include exogenous variables in your model, they will always be included in all models in the selection process.

Basically, one way to do variable selection would be to try all possible combinations of exogenous variables, use auto.arima to find the "best" orders based on AICc, record this model's AICc (recall that AICc penalizes models that have large amounts of fitted parameters that do not increase the model's likelihood by a justifiable amount), and then pick the absolute best model out of all combinations of exogenous variables. Kind of a pain, and possibly very time consuming.

I hope this helps.


Python: Plotting equation with the Dirac Delta function

Use the scipy library function
in this example
import numpy as np
from scipy import signal
number_points =100
location_of_impulse= 60
it_theta=signal.unit_impulse(number_points ,location_of_impulse)
you'll have to do your own scaling
By the the way the following is a direct copy from the SCIPY sit and may be more illuminating
..........................................................................
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt
imp = signal.unit_impulse(100, 40)
b, a = signal.butter(4, 0.2)
response = signal.lfilter(b, a, imp)

plt.plot(np.arange(-50, 50), imp)
plt.plot(np.arange(-50, 50), response)
plt.margins(0.1, 0.1)
plt.xlabel('Time [samples]')
plt.ylabel('Amplitude')
plt.grid(True)
'


How to save dynamic files from colab to Google Drive?

Just mount your drive into Colab and create the checkpoint in the mount point location. Let me try to explain this with an example:

Mount the drive as follows:

from google.colab import drive
drive.mount('/content/gdrive')

This will mount your drive and your google drive files should be all under /content/gdrive/My Drive/ location

Now use your location as you would use any other location in bash or create the check-point directory as follows:
You can use the ! to run bash commands in notebooks as follows:

!cp -r <PATH_TO_PNG_FOLDER> /content/gdrive/My Drive/


Can ROC/AUC help model training or just be used for model chosing?

Your answer would depend on the model you have in mind. For something like multi-variate linear regression, you would use it for factor reduction. For something like a neural network, you could use it while you test different settings for any number of things like the dropout rate or the learning rate. It really just depends on what you're working on at the time.


SARIMAX if your "seasonality" is a clear sine/cosine can work OK (as long as you give the sine/cosine wave as the exogenous input).
Also you could just try bunging it all into Facebook's Prophet.


What is the difference between fully observed and partially observed state features in Reinforcement learning ?

A fully observed state means that there is no hidden information. Clear examples of this are chess and Go because both players have all the information. The fact that both these games are deterministic doesn't matter. A game where the state changes are stochastic can still be fully observable. Games like poker, where both players can observe their own hand but not their opponents' are called partially observable. Other examples of this can be real time strategy games like Starcraft where you can only see in the line of sight of your units.


YOLO Pre-trained weights using Open Images Dataset V4

Darknet has a yolo implementation using Open Images check this page (scroll down up to the end, just before the citation)

You can simply use
wget https://pjreddie.com/media/files/yolov3-openimages.weights


Influence of trend on (supposedly) correlated time series

Q1: What impact does a linear trend have on the correlation between non-spurious time series?
The 4 main measures of correlation are Pearson, Kendall rank, Spearman and Point-biserial (the latter of which is not applicable for this type of problem). For simplicity, I'll only explain how it affects measuring Pearson correlation.
Let's assume $X$ represents a sinusoidal time series without trend: $x_t = sin(t)$, $Y$ represents $X$ with an upwards linear trend: $y_t = t + sin(t)$ and $Z$ represents $X$ with a downwards linear trend: $z_t = -(t + sin(t))$. All series have identical timestamps and have the same unit of measurement (for ease of plotting):

One of the assumption to measure Pearson correlation between two time series is called linearity, that when both series are plotted against one another on a scatter graph, there is a linear relationship:

As you can see, $X$ and $Y$ do not satisfy this condition and so Pearson correlation is the incorrect statistical measure to use, whereas for $Y$ and $Z$ it is. Why though?
Pearson correlation measures the degree to which values deviate from the linear line of best fit between the two series provided. If the relationship is not linear, this relationship will not be measured accurately. This can be shown by plotting the Pearson correlation coefficient as $t$ increases:

Notably, $X$ and $Y$ will also violate the assumption of a monotonic relationship for both Spearman and Kendall rank, and so you cannot measure correlation with any of these methods for $X$ and $Y$ unless transformations of the data are performed to satisfy the underlying assumptions - as you do in the question post.
Linear trends, therefore, don't have strictly positive or negative impacts on measuring correlation. You will just have to react accordingly to the underlying assumptions of the correlation measure you need to use.
Q2: Can I trust the findings of a contractor who doesn't take these type of trends into account?
To paraphrase Hanlon's razor:

It is better to assume ignorance than malicious intent.

If you provide your feedback, the analyst will have an opportunity to discuss through why they chose to pursue a certain route, give them a chance to realise what they did was incorrect, or that they misunderstood the requirements and/or limitations of the project.
Hopefully, this leads to a more positive outcome, given you want the best results and the analyst wants to provide the best service.


What Models should i try for this problem?

If I understand correctly, you currently try to predict the next code among 5500 possibilities. I think that's too many options to choose from, you can't expect a good performance with that.

Since the task is about predicting the 15 error codes I would represent the problem as a sequence labeling problem: the label at each step is either "nothing" (often represented as "_" in sequence labeling tasks) or one of the 15 error codes. The label would represent the most likely outcome for the next step, not for the current one. This way the system can use the history of the codes (regular features) without having to predict all of them, only the critical ones. Note that if several error codes can happen at the same time, you might have to select a single one as label (maybe they can be ranked by priority?).

As far as I know, the standard method for sequence labeling is Conditional Random Fields (CRF). In the traditional approach you would have to do some feature engineering: for example try to not give only the codes as features but also features such as how many times a code happened in the last N steps. I think the modern approach consists in training a RNN and using the vector representation as features for the CRF, but I might not be up to date with this.


Pre-processing irregular, high frequency time-series data in python

Keep in mind, downsampling is always lossy. I will give you two hints.

Nyquist-Shannon Theorem. What it says in short is that the max frequency that you may observe in a sampled signal is half of the sampling frequency. This means that if you downsample the data every 1 hour, you can observe dynamic phenomena with period of 2 hours (or more). Another way of looking at the theorem is "you have to sample the signal at least twice as fast as the maximum frequency that you expect to see in it".
Engineering rule of thumb: Sample the signal at least 10 times faster than the expected highest frequency, not just 2 times.

Histogram Representation. Since you have a huge database, it might make sense to convert your variables into histograms! In this way, you will greatly reduce their size. You discretize every real-valued variable into bins and then instead of storing the exact value of the variable you just increase the count of the appropriate bin by 1. There are a lot of Machine Learning algorithms that deal with histogram representations, such as this:
https://pdfs.semanticscholar.org/b8c8/347f9c33935b97703ecd35a67af5c5508487.pdf

Hope it helps :) !


How is cross validation used to prune a decision tree

I don't know how this is implemented in matlab. I know that some packages use cross validation to decide whether to grow the tree or not. Quite simply, they decide whether to grow based on that evaluation.

(Notice what a lot of packages call pruning is: during training, they mark branches with some score criterium and then remove if the user wants afterwards.)

Personally, I use sklearn which does not have this feature. So, I just do a grid search for several values of max_depth and use whatever maximizes accuracy or whatever score I want.


Sentiment analysis for multiple entry in one text

A Ensemble of "Target based sentiment" models worked for a similar use-case that I worked on.

Some of the models that were part of the solution :

http://sentic.net/sentic-lstm.pdf
https://hal.archives-ouvertes.fr/hal-01537896/document
https://arxiv.org/pdf/1802.05818.pdf


predict_proba() on a continuos target made binary

You can use a Sigmoid function on the Force values(Scaled to [0-10] based on a max value)
The Threshold should become 5 after scaling.
def predict_proba(y_pred):

    y_pred = y_pred*10/100000  # scaled to [0-10]
    thresold = 5

    proba = np.exp(y_pred - threshold)/(1 + np.exp(y_pred - threshold))

    return proba

predict_proba(100000), predict_proba(80000), predict_proba(50000), predict_proba(40000), predict_proba(10000), predict_proba(2000)

(0.9933, 0.9525, 0.5, 0.2689, 0.0179, 0.0081)


This is a tentative general answer.
Bias in statistics (and machine learning) is the difference between an estimated value and the true value. ie $\text{bias} = \hat{y}_{\text{estimated}} - y_{\text{true}}$.
Another (losely related) concept of bias especialy in mathematics is any, primarily, constant term that is added to something (and thus makes a difference between the two values and thus introduces bias). Bias in neural networks is meant in this mathematical sense, rather than the statistical one.
Apart from the above, bias as a term has been introduced in other fields, for example in psychology (eg cognitive bias). The term inductive bias comes from machine learning.
This sense of bias refers to the initial assumptions some entity or algorithm takes for granted and tries to learn based on them. So the induction made is influenced by these initial assumptions, and if these are proved wrong, then there will be bias in the usual statistical or mathematical sense, that is the estimation will differ from the true value.
In machine learning and statistics there is what is called Bias-Variance tradeoff meaning that the total Mean Squared Error (MSE) can be decomposed into bias (in the statistical sense), variance and irreducible error. How is this related to generalisation? Well to generalise well variance should be minimised (in most cases), so since irreducible error is out of control, one way is to increase bias, in the statistical sense (one way or another), if MSE is to be held the same.


Overcoming "too many indices for array" error when loading file into Orange

This error basically means that your data is incorrectly formatted and Orange cannot read it. Somewhere there is either a redundant column delimiter or it might be absent. Either way, you should carefully check that all your columns are correctly separated by tabs or any other consistent separator.


The short answer is that you can determine the max and min of your transformed series if you normalize after your PCA step, as your title implies.

Have you tried L2-norm after your PCA step? This might get you what you are looking for by constraining the values between 0 and 1.

You can, of course, customize this to suit your needs.


List value in Pandas DataFrame column makes analysis harder

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.
Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. If you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.


Gazetteer or any other option of intentionally fixed size feature seems a very popular approach in academic papers, when you have a problem of finite size, for example NER in a fixed corpora, or POS tagging or anything else. I would not consider it cheating unless the only feature you will be using is Gazetteer matching.

However, when you train any kind of NLP model, which does rely on dictionary while training, you may get real world performance way lower than your initial testing would report, unless you can include all objects of interest into the gazetteer (and why then you need that model?) because  your trained model will rely on the feature at some point and, in a case when other features will be too weak or not descriptive, new objects of interest would not be recognized.

If you do use a Gazetteer in your models, you should make sure, that that feature has a counter feature to let model balance itself, so that simple dictionary match won't be the only feature of positive class (and more importantly, gazetteer should match not only positive examples, but also negative ones).

For example, assume you do have a full set of infinite variations of all person names, which makes general person NER irrelevant, but now you try to decide whether the object mentioned in text is capable of singing. You will rely on features of inclusion into your Person gazetteer, which will give you a lot of false positives; then, you will add a verb-centric feature of "Is Subject of verb sing", and that would probably give you false positives from all kind of objects like birds, your tummy when you're hungry and a drunk fellow who thinks he can sing (but let's be honest, he can not) -- but that verb-centric feature will balance with your person gazetteer to assign positive class of 'Singer' to persons and not animals or any other objects. Though, it doesn't solve the case of drunk performer.


How can data science teams inside businesses measure costs and efficiency of their technical work?

I don't know what you mean by "measure and improve costs of their technical work" - but I'll answer in response to the question "how do you know how much value your insights are bringing the business":
To structure a successful project it is important to understand the baseline you are competing with. For example: let's say I have a timeseries forecasting problem where I'm forecasting how much cash will be in some corporate account. One baseline might be: "let's assume that tomorrow's net cash flows will be identical to yesterday's". Now that you have this benchmark you can evaluate how performant your solution is relative to this.
This can give you insight: "my solution outperforms the benchmark by some % and increases the detection of future overdraft fees by 90%". Convert this 90% improvement into the total # of new overdraft fees detected by your system relative to the baseline. This can be translated to some dollar amount on an annual basis - and you can say "my solution saves the business $###,### annually"
Understanding the system you are replacing allows you something to compare to and can track results. This is also an effective strategy for communicating successes to management (something that is vital in larger enterprises).


Why we take $\alpha\sum B_j^2$ as penalty in Ridge Regression?

Expanding my comments into an answer.
Ridge regression is a by definition an augmentation of Least Squares method, especialy for problems where the data may be highly correlated (there is what is called multi-colinearity).
Lets assume the dependent variable is $y$ and $x_i$ are the independent variables.
Then assume the true mapping between $X$ and $y$ is:
$$y = \sum_i \beta_i x_i$$.
Ordinary Least Squares (OLS) assumes the mapping is given by $\hat{\beta_i}$ coefficients which solve the least squares problem:
$$\hat{B} = (X^TX)^{-1}X^TY$$
The above formula indeed minimises the square error $RSS = \sum_i(y_i-\hat{y_i})^2$
However, if some columns in $X$ are correlated then ordinary least squares may not provide the best solution.

Least Square Method finds the Best and Unbiased Coefficients
You may know that least square method finds the coefficients that best
fit the data. One more condition to be added is that it also finds the
unbiased coefficients. Here unbiased means that OLS doesn’t consider
which independent variable is more important than others. It simply
finds the coefficients for a given data set. In short, there is only
one set of betas to be found, resulting in the lowest ‘Residual Sum of
Squares (RSS)’. The question then becomes “Is a model with the lowest
RSS truly the best model?”.
Bias vs. Variance
The answer for the question above is “Not really”. As hinted in the
word ‘Unbiased’, we need to consider ‘Bias’ too. Bias means how
equally a model cares about its predictors. Let’s say there are two
models to predict an apple price with two predictor ‘sweetness’ and
‘shine’; one model is unbiased and the other is biased.
First, the unbiased model tries to find the relationship between the
two features and the prices, just as the OLS method does. This model
will fit the observations as perfectly as possible to minimize the
RSS. However, this could easily lead to overfitting issues. In other
words, the model will not perform as well with new data because it is
built for the given data so specifically that it may not fit new data.
The biased model accepts its variables unequally to treat each
predictor differently. Going back to the example, we would want to
only care about ‘sweetness’ to build a model and this should perform
better with new data. The reason will be explained after understanding
Bias vs. Variance. If you’re not familiar with the bias vs. variance
topic, I strongly recommend you to watch this video that will give you
insight.

In order to affect bias-variance tradeoff ridge regression adds a small offset in the estimation process, ie:
The magnitude of the $B$ coefficients must satisfy a criterion (the "ridge") eg:
$$|B|_2^2 \le C^2$$
Then this constraint can be added in the Least Squares formula (eg via a Lagrange Multiplier).  This constraint affects the bias-variance tradeoff and can be helpful in cases such as highly correlated data or data with unequal significance. Then the new $RSS$ is similar to what you ask in the question.
Again the square error is measured along with constraints on the coefficients in order to affect bias-variance tradeoff.
Reference:

Ridge Regression for Better Usage
Bias-Variance Tradeoff


Fitting and transforming text data in training, testing, and validation sets

It's best to leave the test data unseen. You can create a data transformation pipeline for your test data so that your test data will be transformed into a TFIDF vector after being passed through the pipeline. This way you get to know how well your model performs on unseen data.

And it's OK to transform the entire training data into a TFIDF vector and run the k-fold validation.


Calculate correlation between two sensors regarding the time

Your suggested method should work, assuming the two column are in the same dataframe with the same timestamps as the  index. This means that they are already aligned in time and therefore the time is implicitly encoded.

If there are any missing datapoints, you could impute them (fill them) with something like the previous value, which is usually best for time-series data. Replacing with e.g. the mean of the entire time-series might be misleading because there are temporal fluctuations/cycles involved.

df.fillna(method='ffill')

You could also simply plot them to see which kind of correlation value you might expect, doing something like this:

import matplotlib.pyplot as plt    # <-- needed if running in script (not a Jupyter notebook)
df[["sensor1", "sensor2"]].plot()
plt.plot()


What are some good machine learning techniques for developing rules that govern changes in data?

Yes. There are different methods to extract some rules from data. It depends on the type of rules that you want to extract. Anyhow, different types of the decision tree (such as fuzzy decision tree) and association rules to find the rules that have strong support and confidence, could help you in this way.


What are some key difference in functionality between ProM and Apromore?

Below are three key differences.

Apromore is a server-side collaborative tool. It is typically installed on a server (on-premise or on the cloud). Users connect to Apromore via the Web browser. Users can upload process models and event logs, organize them into folders, enrich them with log filters and dashboard designs, and share them with other users. Apromore comes with a process model editor, so you can edit the process models you discover from your event logs, beautify them, and export them in BPMN format (so you can share them with other tools). Meanwhile, ProM is a desktop tool. You typically install it in your laptop. You use it to analyze an event log locally, but you cannot use it in a collaborative manner with say 3-4 team members. You cannot save your analysis (e.g. custom dashboards, filters) in ProM.
A second difference is that Apromore supports BPMN natively. The process discovery algorithm in Apromore is Split Miner, which discovers BPMN models by design. Traditionally, ProM is designed to support an academic notation called Petri nets, which is used in research studies but generally not used by process improvement practitioners.
Finally, Apromore supports both process mining use cases (discovery, conformance), but also process performance monitoring use cases via customizable process performance dashboards.


Typically XGBoost does not require many parameters to be tuned to get good performances.

I would start with optimizing the n_estimators, max_depth and min_child_weight parameters only. This should already bring you close enough.

Another thing you can do to speed-up the process is to prefer Random Search over Grid Search, since in most cases is as or more effective.

You could also have a look at the LightGBM implementation, which is faster (and needs less memory) than XGBoost.


Time-series prediction: Model & data assumptions in AI/ML models vs conventional models

Great question @Maeaex1 !

First of all why do we even need assumptions in models (generally speaking
)?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources

But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

So to conclude: there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.


What's the best way to predict weekly selling data?

There is an ongoing competition on Kaggle right now which is about sales prediction.
https://www.kaggle.com/c/m5-forecasting-accuracy/notebooks

The user kyakovlev have created a series of notebooks where he creates features and creates an lgbm model:

https://www.kaggle.com/kyakovlev/m5-three-shades-of-dark-darker-magic

The notebook linked is the last in the series.
I've had success in applying a very similar strategy on a real-world project for a client.

You probably want to engineer more features and add more lagging variables.

The prophet model could possibly be a problem if the patterns in the data aren't clear. I have had examples where it is completely of. This could be due to the trend. The Prophet model also easily overfit on holidays due to the lack of data. This is due to the nature of the model.
You should definitely inspect what the model is learning.

Models such as XGBoost or LightGBM are great for this type of problem!


Is NN with no hidden layer is behave like a regression?

You would then call it perception model. As in 1945 or so some researcher did this with no hidden layer. They came up with AND,OR,XOR problem where you can solve AND,OR with a single perception model but that model was unable to solve the XOR problem.

There are two kinds of regression: Linear and non linear. The activation functions in neural net makes them non linear regressor.

A Neural Net would be a linear regressor if you don't put any activation function.

Then it would be like : $Wx+b$ (which is $y=mx+b$ equivalent).
But when you add activation function let it be sigmoid then it achieves a non linear space.Like $$\frac{1}{1+e^{-(wx+b)}}$$

"If we have for instance 20 input and 4 output and I have no true label, is it similar to regression?" --- In this you are just deleting back propagation thing. what as usual regression algorithms like linear,polynomial does is they just predict the function but never learns something like neural net learns proper function via optimizing weights. If you don't have true level that means you are not calculating the loss value between [true-predicted] so it would be a simple regression.


Ok... Take a step back. When you're dealing with RNNs your dataset should have a shape that looks like (nb_samples, nb_timesteps, nb_features)
Translating this to your use case means that each account is a sample (what you'll iterate when doing mini-batching), each week is a timestep (what your rnn will iterate over) and your features are.... Features.

Now... Don't use accountID as a feature. That's just an ID with no prediction power. Also try to add some seasonality features, like month or something.
Does that help you out?


Pre-processing mixed data prior to clustering

If it's categorical you can use the presence/absence of each category as a feature.

If it's nominal (ordered) you can normalize their values to be between 0 and 1 (or whatever range you normalized the rest of your data to).


Methodologies for predicting missing data

As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are:

Mean/ Mode/ Median Imputation: Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the data set to assist in estimating the missing values. Mean / Mode / Median imputation is one of the most frequently used methods. It consists of replacing the missing data for a given attribute by the mean or median (quantitative attribute) or mode (qualitative attribute) of all known values of that variable. This can further be classified as generalized and similar case imputation.
Prediction Model:  Prediction model is one of the sophisticated method for handling missing data. Here, we create a predictive model to estimate values that will substitute the missing data.  In this case, we divide our data set into two sets: One set with no missing values for the variable and another one with missing values. First data set become training data set of the model while second data set with missing values is test data set and variable with missing values is treated as target variable. Next, we create a model to predict target variable based on other attributes of the training data set and populate missing values of test data set.
KNN(k-nearest neighbor) Imputation: In this method of imputation, the missing values of an attribute are imputed using the given number of attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.


What does "batch" and "batch_size" mean in word2vec skip-gram model?

A minibatch, is a group of (input,output) pairs that you present to your neural net in one pass (or epoch), without computing the stochastic gradient descent between two pairs (you only do it at the end of the minibatch, summing the errors over the pairs). This improve the speed and prevent over learning on single elements, thus improving learning. It also doesn't need to much memory since your batches are "mini".

Some people call stochastic training when your mini-batch has a size of 1, and batch training when it has the size of your training dataset.

You can get some nice explanations here.


Amount of data needed for deep learning vs support vector machine

When it comes to how much data model M needs to accurately model problem P, there are 3 factors to consider:

What is the dimensionality of model M?

The power of neural networks is that they can model functions up to to a high number of dimensions, which means that they can, therefore, model any function with fewer dimensions. Indeed, a quadratic function can always be estimated by a cubic function, whereas the converse isn't true.

Support vector machines are an interesting type of models because the kernel trick allows them to model arbitrarily high-dimensional relationships. This allows you to eventually tweak your kernel based on the amount of data at your disposal.

What is the dimensionality of the space represented by the data at your disposal?

Not all data are created equal, and the amount of data is only a proxy to estimate the dimensionality of the problem space. Always consider the possibility that your data might be an unrepresentative sample of your space. The amount and type of noise present in the data is also a huge factor. A factor which gets mitigated the more data you have.

What is the true dimensionality of problem P?

Easily the most important factor in terms of how much data you need, and sadly, the only factor that we can never truly know. In a perfect world, the dimensionality of the model should be greater or equal to the dimensionality of the problem. Equal being, of course, the ideal scenario.

Do deep learning approaches really need more data?

Theoretically, the need the amount of data that will appropriately match the true dimensionality of the problem. Because they have a higher number of parameters, more data generally help reduce the chance of overfitting. However, as previously stated, a model with a dimensionality much higher than the problem's is ill-suited, to begin with.

The power behind neural networks is that they have such a large solution space that the chance to converge in a satisfying spot is quite high.


Difference between RMSProp with momentum and Adam Optimizers

(My answer is based mostly on Adam: A Method for Stochastic Optimization (the original Adam paper) and on the implementation of rmsprop with momentum in Tensorflow (which is operator() of struct ApplyRMSProp), as rmsprop is unpublished - it was described in a lecture by Geoffrey Hinton .)

Some Background

Adam and rmsprop with momentum are both methods (used by a gradient descent algorithm) to determine the step.

Let $\Delta x^{(t)}_j$ be the $j^{\text{th}}$ component of the $t^{\text{th}}$ step. Then:

In Adam: $$\Delta x_{j}^{(t)}=-\frac{\text{learning_rate}}{\sqrt{\text{BCMA}\left(g_{j}^{2}\right)}}\cdot\text{BCMA}\left(g_{j}\right)$$
while:

$\text{learning_rate}$ is a hyperparameter.
$\text{BCMA}$ is short for "bias-corrected (exponential) moving average" (I made up the acronym for brevity).

All of the moving averages I am going to talk about are exponential moving averages, so I would just refer to them as "moving averages".

$g_j$ is the $j^{\text{th}}$ component of the gradient, and so $\text{BCMA}\left(g_{j}\right)$ is a bias-corrected moving average of the $j^{\text{th}}$ components of the gradients that were calculated. Similarly, $\text{BCMA}\left(g_{j}^{2}\right)$ is a bias-corrected moving average of the squares of the $j^{\text{th}}$ components of the gradients that were calculated.
For each moving average, the decay factor (aka smoothing factor) is a hyperparameter.
Both the Adam paper and TensorFlow use the following notation:

$\beta_1$ is the decay factor for $\text{BCMA}\left(g_{j}\right)$
$\beta_2$ is the decay factor for $\text{BCMA}\left(g^2_{j}\right)$

The denominator is actually $\sqrt{\text{BCMA}\left(g_{j}^{2}\right)}+\epsilon$, while $\epsilon$ is a small hyperparameter, but I would ignore it for simplicity.

In rmsprop with momentum: $$\Delta x_{j}^{\left(t\right)}=\text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(t-1\right)}-\frac{\text{learning_rate}}{\sqrt{\text{MA}\left(g_{j}^{2}\right)}}\cdot g_{j}^{\left(t\right)}$$
while:

$\text{momentum_decay_factor}$ is a hyperparameter, and I would assume it is in $(0,1)$ (as it usually is).
In TensorFlow, this is the momentum argument of RMSPropOptimizer.
$g^{(t)}_j$ is the $j^{\text{th}}$ component of the gradient in the $t^{\text{th}}$ step.
$\text{MA}\left(g_{j}^{2}\right)$ is a moving average of the squares of the $j^{\text{th}}$ components of the gradients that were calculated.
The decay factor of this moving average is a hyperparameter, and in TensorFlow, this is the decay argument of RMSPropOptimizer.

High-Level Comparison

Now we are finally ready to talk about the differences between the two.

The denominator is quite similar (except for the bias-correction, which I explain about later).
However, the momentum-like behavior that both share (Adam due to $\text{BCMA}\left(g_{j}\right)$, and rmsprop with momentum due to explicitly taking a fraction of the previous step) is somewhat different.
E.g. this is how Sebastian Ruder describes this difference in his blog post An overview of gradient descent optimization algorithms:

  Whereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface [...]

This description is based on the paper GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, so check it out if you want to dive deeper.

Next, I would describe 2 simple scenarios to demonstrate the difference in the momentum-like behaviors of the methods.

Lastly, I would describe the difference with regard to bias-correction.

Accumulating Momentum

Consider the following scenario: The gradient was constant in every step in the recent past, and $\Delta x_{j}^{(t-1)}=0$. Also, to keep it simple, $g_{j}>0$.
I.e. we can imagine our algorithm as a stationary ball on a linear slope.
What would happen when we use each of the methods?

In Adam

The gradient was constant in the recent past, so $\text{BCMA}\left(g_{j}^{2}\right)\approx g_{j}^{2}$ and $\text{BCMA}\left(g_{j}\right)\approx g_j$.
Thus we get:
$$\begin{gathered}\\
\Delta x_{j}^{(t)}=-\frac{\text{learning_rate}}{\sqrt{g_{j}^{2}}}\cdot g_{j}=-\frac{\text{learning_rate}}{|g_{j}|}\cdot g_{j}\\
\downarrow\\
\Delta x_{j}^{\left(t\right)}=-\text{learning_rate}
\end{gathered}
$$

I.e. the "ball" immediately starts moving downhill in a constant speed.

In rmsprop with momentum

Similarly, we get:
$$\Delta x_{j}^{\left(t\right)}=\text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(t-1\right)}-\text{learning_rate}$$
This case is a little more complicated, but we can see that:
$$\begin{gathered}\\
\Delta x_{j}^{\left(t\right)}=-\text{learning_rate}\\
\Delta x_{j}^{\left(t+1\right)}=-\text{learning_rate}\cdot(1+\text{momentum_decay_factor})
\end{gathered}
$$
So the "ball" starts accelerating downhill.
Given that the gradient stays constant, you can prove that if:
$$-\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}<\Delta x_{j}^{\left(k\right)}$$
then: $$-\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}<\Delta x_{j}^{\left(k+1\right)}<\Delta x_{j}^{\left(k\right)}$$
Therefore, we conclude that the step converges, i.e. $\Delta x_{j}^{\left(k\right)}\approx \Delta x_{j}^{\left(k-1\right)}$ for some $k>t$, and then:
$$\begin{gathered}\\
\Delta x_{j}^{\left(k\right)}\approx \text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(k\right)}-\text{learning_rate}\\
\downarrow\\
\Delta x_{j}^{\left(k\right)}\approx -\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}
\end{gathered}
$$
Thus, the "ball" accelerates downhill and approaches a speed $\frac{1}{1-\text{momentum_decay_factor}}$ times as large as the constant speed of Adam's "ball". (E.g. for a typical $\text{momentum_decay_factor}=0.9$, it can approach $10\times$ speed!)

Changing Direction

Now, consider a scenario following the previous one:
After going down the slope (in the previous scenario) for quite some time (i.e. enough time for rmsprop with momentum to reach a nearly constant step size), suddenly a slope with an opposite and smaller constant gradient is reached.
What would happen when we use each of the methods?

This time I would just describe the results of my simulation of the scenario (my Python code is at the end of the answer).
Note that I have chosen for Adam's $\text{BCMA}\left(g_{j}\right)$ a decay factor equal to $\text{momentum_decay_factor}$. Choosing differently would have changed the following results:

Adam is slower to change its direction, and then much slower to get back to the minimum.
However, rmsprop with momentum reaches much further before it changes direction (when both use the same $\text{learning_rate}$).
Note that this further reach is because rmsprop with momentum first reaches the opposite slope with much higher speed than Adam. If both reached the opposite slope with the same speed (which would happen if Adam's $\text{learning_rate}$ were $\frac{1}{1-\text{momentum_decay_factor}}$ times as large as that of rmsprop with momentum), then Adam would reach further before changing direction.

Bias-Correction

What do we mean by a biased/bias-corrected moving average? (Or at least, what does the Adam paper mean by that?)

Generally speaking, a moving average is a weighted average of:

The moving average of all of the previous terms
The current term

Then what is the moving average in the first step?

A natural choice for a programmer would be to initialize the "moving average of all of the previous terms" to $0$. We say that in this case the moving average is biased towards $0$.
When you only have one term, by definition the average should be equal to that term.
Thus, we say that the moving average is bias-corrected in case the moving average in the first step is the first term (and the moving average works as usual for the rest of the steps).

So here is another difference: The moving averages in Adam are bias-corrected, while the moving average in rmsprop with momentum is biased towards $0$.

For more about the bias-correction in Adam, see section 3 in the paper and also this answer.

Simulation Python Code

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

###########################################
# script parameters
def f(x):
    if x > 0:
        return x
    else:
        return -0.1 * x

def f_grad(x):
    if x > 0:
        return 1
    else:
        return -0.1

METHOD_TO_LEARNING_RATE = {
    'Adam': 0.01,
    'GD': 0.00008,
    'rmsprop_with_Nesterov_momentum': 0.008,
    'rmsprop_with_momentum': 0.001,
    'rmsprop': 0.02,
    'momentum': 0.00008,
    'Nesterov': 0.008,
    'Adadelta': None,
    }
X0 = 2
METHOD = 'rmsprop'
METHOD = 'momentum'
METHOD = 'GD'
METHOD = 'rmsprop_with_Nesterov_momentum'
METHOD = 'Nesterov'
METHOD = 'Adadelta'
METHOD = 'rmsprop_with_momentum'
METHOD = 'Adam'
LEARNING_RATE = METHOD_TO_LEARNING_RATE[METHOD]

MOMENTUM_DECAY_FACTOR = 0.9
RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR = 0.9
ADADELTA_DECAY_FACTOR = 0.9
RMSPROP_EPSILON = 1e-10
ADADELTA_EPSILON = 1e-6
ADAM_EPSILON = 1e-10
ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR = 0.999
ADAM_GRADS_AVG_DECAY_FACTOR = 0.9

INTERVAL = 9e2
INTERVAL = 1
INTERVAL = 3e2
INTERVAL = 3e1
###########################################

def plot_func(axe, f):
    xs = np.arange(-X0 * 0.5, X0 * 1.05, abs(X0) / 100)
    vf = np.vectorize(f)
    ys = vf(xs)
    return axe.plot(xs, ys, color='grey')

def next_color(color, f):
    color[1] -= 0.01
    if color[1] < 0:
        color[1] = 1
    return color[:]

def update(frame):
    global k, x, prev_step, squared_grads_decaying_avg, \
           squared_prev_steps_decaying_avg, grads_decaying_avg

    if METHOD in ('momentum', 'Nesterov', 'rmsprop_with_momentum',
                  'rmsprop_with_Nesterov_momentum'):
        step_momentum_portion = MOMENTUM_DECAY_FACTOR * prev_step
    if METHOD in ('Nesterov', 'rmsprop_with_Nesterov_momentum'):
        gradient = f_grad(x + step_momentum_portion)
    else:
        gradient = f_grad(x)

    if METHOD == 'GD':
        step = -LEARNING_RATE * gradient
    elif METHOD in ('momentum', 'Nesterov'):
        step = step_momentum_portion - LEARNING_RATE * gradient
    elif METHOD in ('rmsprop', 'rmsprop_with_momentum',
                    'rmsprop_with_Nesterov_momentum'):
        squared_grads_decaying_avg = (
            RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR) * gradient ** 2)
        grads_rms = np.sqrt(squared_grads_decaying_avg + RMSPROP_EPSILON)
        if METHOD == 'rmsprop':
            step = -LEARNING_RATE / grads_rms * gradient
        else:
            assert(METHOD in ('rmsprop_with_momentum',
                              'rmsprop_with_Nesterov_momentum'))
            print(f'LEARNING_RATE / grads_rms * gradient: {LEARNING_RATE / grads_rms * gradient}')
            step = step_momentum_portion - LEARNING_RATE / grads_rms * gradient
    elif METHOD == 'Adadelta':
        gradient = f_grad(x)
        squared_grads_decaying_avg = (
            ADADELTA_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - ADADELTA_DECAY_FACTOR) * gradient ** 2)
        grads_rms = np.sqrt(squared_grads_decaying_avg + ADADELTA_EPSILON)
        squared_prev_steps_decaying_avg = (
            ADADELTA_DECAY_FACTOR * squared_prev_steps_decaying_avg +
            (1 - ADADELTA_DECAY_FACTOR) * prev_step ** 2)
        prev_steps_rms = np.sqrt(squared_prev_steps_decaying_avg + ADADELTA_EPSILON)
        step = - prev_steps_rms / grads_rms * gradient
    elif METHOD == 'Adam':
        squared_grads_decaying_avg = (
            ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR) * gradient ** 2)
        unbiased_squared_grads_decaying_avg = (
            squared_grads_decaying_avg /
            (1 - ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR ** (k + 1)))
        grads_decaying_avg = (
            ADAM_GRADS_AVG_DECAY_FACTOR * grads_decaying_avg +
            (1 - ADAM_GRADS_AVG_DECAY_FACTOR) * gradient)
        unbiased_grads_decaying_avg = (
            grads_decaying_avg /
            (1 - ADAM_GRADS_AVG_DECAY_FACTOR ** (k + 1)))
        step = - (LEARNING_RATE /
                  (np.sqrt(unbiased_squared_grads_decaying_avg) + ADAM_EPSILON) *
                  unbiased_grads_decaying_avg)

    x += step
    prev_step = step
    k += 1

    color = next_color(cur_color, f)

    print(f'k: {k}\n'
          f'x: {x}\n'
          f'step: {step}\n'
          f'gradient: {gradient}\n')

    k_x_marker, = k_and_x.plot(k, x, '.', color=color)
    x_y_marker, = x_and_y.plot(x, f(x), '.', color=color)

    return k_x_marker, x_y_marker

k = 0
x = X0
cur_color = [0, 1, 1]
prev_step = 0
squared_grads_decaying_avg = 0
squared_prev_steps_decaying_avg = 0
grads_decaying_avg = 0

fig, (k_and_x, x_and_y) = plt.subplots(1, 2, figsize=(9,5))
k_and_x.set_xlabel('k')
k_and_x.set_ylabel('x', rotation=0)
x_and_y.set_xlabel('x')
x_and_y.set_ylabel('y', rotation=0)
plot_func(x_and_y, f)
x_and_y.plot(x, f(x), '.', color=cur_color[:])
k_and_x.plot(k, x, '.', color=cur_color[:])
plt.tight_layout()

ani = FuncAnimation(fig, update, blit=False, repeat=False, interval=INTERVAL)
plt.show()


Interpreting the results of randomized PCA in scikit-learn

Yes, through the components_ property:

import numpy, seaborn, pandas, sklearn.decomposition
data = numpy.random.randn(1000, 3) @ numpy.random.randn(3,3)
seaborn.pairplot(pandas.DataFrame(data, columns=['x', 'y', 'z']));

sklearn.decomposition.RandomizedPCA().fit(data).components_

> array([[ 0.43929754,  0.81097276,  0.38644644],
       [-0.54977152,  0.58291122, -0.59830243],
       [ 0.71047094, -0.05037554, -0.70192119]])

sklearn.decomposition.RandomizedPCA(2).fit(data).components_

> array([[ 0.43929754,  0.81097276,  0.38644644],
       [-0.54977152,  0.58291122, -0.59830243]])

We see that the truncated decomposition is simply the truncation of the full decomposition. Each row contains the coefficients of the corresponding principal component.


Predict the average temperature for next 30 years

Do you have context where this problem originates from? You seem to be mixing up a few things.

Q1: depends on what dataset do you have available, or can you find? It can be just year and average temperature of the year, and then you'd apply a regression or ARIMA model.
But maybe you have average temperatures for various countries and regions, and then you could predict average temperatures per region.

Q2: again, depends on what dataset and columns you have. Typical independent variables are variables whose variation does not impact the temperature. Those typically have near 0 correlation with the temperature.

Q3: depending on the dataset, you probably should not use machine learning at all. Maybe regression/ARIMA is enough. And regression to predict values you should not do on partial data.

When you train a model with 30%, you use 70% for validation or scoring the model.

Your prediction could be just the extrapolation on the fitted curves on your data set.
You also have some information on the variations (R2), that can help give probability scores.
A typical outcome could be a 95% confidence interval that the temperature will be Tx +/- dTx.


What is cohen kappa metric, implementation in Python?

Quadratic Kappa Metric is the same as cohen kappa metric in Sci-kit learn @ sklearn.metrics.cohen_kappa_score when weights are set to 'Quadratic'.

quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). In the event that there is less agreement between the raters than expected by chance, the metric may go below 0. The quadratic weighted kappa is calculated between the scores which are expected/known and the predicted scores.

Results have 5 possible ratings, 0,1,2,3,4. The quadratic weighted kappa is calculated as follows. First, an N x N histogram matrix O is constructed, such that Oi,j corresponds to the number of adoption records that have a rating of i (actual) and received a predicted rating j. An N-by-N matrix of weights, w, is calculated based on the difference between actual and predicted rating scores.

An N-by-N histogram matrix of expected ratings, E, is calculated, assuming that there is no correlation between rating scores. This is calculated as the outer product between the actual rating's histogram vector of ratings and the predicted rating's histogram vector of ratings, normalized such that E and O have the same sum.

From these three matrices, the quadratic weighted kappa is calculated.

Code implementation in Python

Breaking down the formula into parts

5 step breakdown for Weighted Kappa Metric

First, create a multi-class confusion matrix O between predicted and actual ratings.
Second, construct a weight matrix w which calculates the weight between the actual and predicted ratings.
Third, calculate value_counts() for each rating in preds and actuals.
Fourth, calculate E, which is the outer product of two value_count vectors
Fifth, normalize the E and O matrix

Calculate weighted kappa as per formula

Each Step Explained

Step-1: Under Step-1, we shall be calculating a confusion_matrix between the Predicted and Actual values. Here is a great resource to know more about confusion_matrix.

Step-2: Under Step-2, under step-2 each element is weighted. Predictions that are further away from actuals are marked harshly than predictions that are closer to actuals. We will have a less score if our prediction is 5 and actual is 3 as compared to a prediction of 4 in the same case.

Step-3: We create two vectors, one for preds and one for actuals, which tells us how many values of each rating exist in both vectors.

Step-4: E is the Expected Matrix which is the outer product of the two vectors calculated in step-3.

Step-5: Normalise both matrices to have the same sum. Since it is easiest to get the sum to be '1', we will simply divide each matrix by its sum to normalize the data.

Step-6: Calculated numerator and denominator of Weighted Kappa and return the Weighted Kappa metric as 1-(num/den)

More Info


How to predict next visit date based on this data

If you plot days since the first visit vs the visit number, this ends up being a simple regression problem.

import pandas as pd

data = pd.read_csv('data.txt', '|')
data['visit_date'] = pd.to_datetime(data['visit_date'])
data['days_since_first_visit'] = (data['visit_date'] - data['visit_date'][0]).map(lambda delta: delta.days)
data = data.reset_index()

data.plot('index', 'days_since_first_visit')

I would recommend using a library like scikit-learn or statsmodels to do the regression.


I would suggest to look at intermittent demand models, I think for such type data ARIMA type model is an inappropriate model.

Intermittent Demand model first proposed by J. D. Croston and I would suggest to look at that paper too. Here, you can find a short note in intermittent demand forecasting.


Scaling matters. The y values will be ignored because of the x axis.
Parameters matter. You probably used DBSCAN with the sklearn default parameter values which are utter nonsense for anything but 2d normalized toy data.
Asking the right question matters. What are clusters here supposed to be anyway? I don't see any clusters. You show a time series with plenty of missing data, but there are no clusters. You seem to want to split your time series where you have missing values, but that is not clustering. You likely can abuse clustering such as DBSCAN here (just try minpts=1, epsilon=100 or similar) but that is an incredibly inefficient way to solve this, because you abuse a multivariate approach for a trivial one-dimensional problem.

Sort by time. If the next timestamp is much larger than the previous, this is the end of a subintervals (I wouldn't call these clusters). That is easily done with a for loop.


Performance of Triplet loss network vs multiclass classification

For me triplet loss function (as mentioned by Neil Slater as well), is used for object recognition i.e. identify the similar object. Face-recognition is one of the use case.

  This function is based upon Siamese Network, which will provide
  us the feature vector as an output. During recognition, we compare the
  feature-vector of the new image with the feature-vector of the
  training data. Select the one which has closer matching.

In order to improve Triplet-loss function, you can try different similarity approaches or different threshold value's. Please do share the output.

But again for me it's not comparable with object-classification use case.


Different available packages in TensorFlow virtualenv?

I can't offer any suggestions as to why your virtual environment can't find the packages (I'm not familiar with anaconda). The only thing that comes to mind is a PATH problem.

Perhaps try installing the package with the following command:

conda install -n yourenvname [package]

or, in your case:

conda install -n tensorflow h5py

  To install additional packages only to your virtual environment, enter
  the following command where yourenvname is the name of your
  environemnt, and [package] is the name of the package you wish to
  install. Failure to specify “-n yourenvname” will install the package
  to the root Python installation.

source: https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/


Are mini batches sampled randomly in Keras' Sequential.fit method()

If you set shuffle=True as an argument of the model.fit method, Keras will shuffle the dataset before splitting it into batches (source), otherwise the dataset will be processed sequentially.


How does Batch Normalization in Machine Learning address covariate shift and speed up training?

So for the first question, you are almost correct. Your description of covariate shift is correct, but batch normalization isn't focused on fixing input layer covariate shift, but instead internal covariate shift. It's not a huge difference, since the internal covariate shift of a hidden layer acts as the input to another layer, but internal covariate shift can occur even if there isn't significant input layer covariate shift.

The internal shift happens when weight updates occur while learning, after a couple epochs the output distribution of a hidden layer will likely change. This slows learning, because now the next hidden layer will have to learn the new distribution. Batch normalization tries to minimize this change in distribution so that less of the learning is focused on trying to learn the changes in hidden layer distributions.


Why do trained RL agents still display stochastic "exploratory" behavior on testing data?

The PPO approach directly generates stochastic policies. Its output is some probability distribution function over the action space.

This is not the case for all RL algorithms, but is common for many Policy Gradient methods.

In some cases you may want this behaviour to continue. Paper-Scissors-Stone is a classic example of a game where a stochastic policy is optimal, and there are other examples from game theory.

Also, the agent many have learned on-policy (I'm not sure in case of PPO without looking it up), in which case the stochastic behaviour should match with expected value predictions. This is similar conceptually to using SARSA to generate an $\epsilon$-greedy policy. The value functions match following that policy, although if $\epsilon$ is low enough you may take a reasonable guess that a fully greedy policy is optimal.

  To what degree is the trained agent stochastic (will it follow its model predictions 90% of the time and guess the other 10%)?

To the degree that the output of the policy is stochastic. It will always "follow its model predictions".

Switching deterministic on actually stops the agent from following the model, and typically will select the mode (highest probability density) of the action distribution.

Unlike SARSA, with Policy Gradient methods, there is not always access to a "greedy" policy that picks according to action values. So instead, your deterministic flag is likely picking greedily according to action probability.

In some cases (e.g. Actor-Critic) you may also have a value based estimator for V(s) or Q(s,a), and it could be used instead, but that is usually considered secondary to the policy function (in addition, using this on a large or continuous action space is very inefficient compared to a real-valued policy function)

Also note in some cases e.g. DDPG (which stands for Deep Deterministic Policy Gradients), a policy gradient method can use a deterministic policy with exploration added as a behaviour policy, making the algorithm off-policy. DDPG should not exhibit exploration during testing as PPO is for you. You may find DDPG behaves closer to your original expectations because of this.


Help with replacing values in a data frame

Assuming that the data is small enough that efficiency is not an issue, I would use the Pandas data frame and do something like:

white['quality_code'] = 'high'
white.loc[quality < 5, 'quality_code'] = 'low'
white.loc[(quality > 4) & (quality < 8) , 'quality_code'] = 'med'


This is what Machine Learning models are used for..(to predict what they think will happen in the near time based on the data they have been fed into....)

A simple answer is this

The first couple days of data are the most important , you need to gauge and have a watch over your model..

As with any incremental learning, you can learn more recent stuff but also underfit the past significantly more than without learning the new stuff.
Therefore, it's why there's a monitoring to perform on models which use incremental learning (especially on production systems).

Provided that other things remains similar with time..(same Preprocessing, same scaling etc..)

But let's say if you have tons of data everyday, then nearly at the month end, the model can be completely trained on the last months data or else checkpoints needs to be kept


Why does a machine learning model of English can not be directly applied on right to left languages, like Arabic Hindi, and Urdu

We would need to know more about how your model actually works. Logistic regression means that the model transforms an input in numbers to output in numbers using a sigmoid function. But your input is not numbers, it's text.

That means that you're doing something already to transform the text to numbers before you're doing Logistic Regression. That's why your question is a little confusing.

Also, are you training a model on English text, and evaluating on a separate English test set, and then training a second model on Arabic text and evaluating that on Arabic test data? If you try to train a model on one language and evaluate it on another it won't work.

What library are you using? NLTK? Sklearn? All of these have simple text classifiers.

It's possible you're using a bag of words model, where each word is given an ID and transformed to a sparse vector, and the entire document is transformed to a long vector whose size is the length of your vocabulary. You might be using a tf*idf stage which weights words like "the" with a lower weight than content words.

A model like this doesn't need to break necessarily if you change language. However there are a few important points you need to know:

there will be a tokenisation stage. Arabic tokenisation in particular is different from English because some prefixes and suffixes are attached to the word without spaces (e.g. article al- ال, preposition "to" li- لِ, etc.)  You need to have replaced the tokenisation stage correctly.
If there is a stopword removal stage you should replace the stopwords with the list of the new language.
If there is a lemmatisation stage you need to find an Arabic/Hindi/Urdu lemmatiser or stemmer. Otherwise you will have a problem with sparsity, e.g. an inflection of a word will occur in the test text and your model won't recognise it because the word appeared in a different inflection in the training text.
Some languages don't use spaces between words like English, the obvious example is Chinese, however I don't think this is an issue for the languages you mentioned.

Whether the language is RTL or LTR makes no difference for the classifier since that only affects how the text is displayed on screen by a browser or word processor. The bytes are in the same order in an Arabic text file as an English text file.

However any model architecture would work fine with any language provided the preprocessing is adapted correctly to the language. If you used a character based model with neural network you could even make a model that works for any language without worrying about adapting the preprocessing, for example BERT has a multilingual pretrained model which can be run directly on almost any language. However these models tend to be bigger and require a powerful computer and even a GPU.

If you wanted a model that would work on text in any of the four languages you have mentioned, you could train four separate models and have a language identification component which first identifies the language and passes the input to the correct model.


Clustering efficiency in a discrete time-series

Let me start with an important point; what other criteria than classification success can U use to identify my cluster separation performance?:

Classification as an indicator for clustering performance has an internal paradox. If you have the classification the clustering question does not apply anymore. These two concept are coming form two totally different philosophies so I would say be careful if you have already understood the concepts (what you say may make sense in semi-supervised learning which is not in your tags so I assume there is a misunderstanding).
Clustering has no performance evaluation! this is the problem I see most of data scientists struggling with. In practice you may define a good criterion (but honestly, what is good?!!) and deliver your solution, but in research schema there is no evaluation for clustering as the question itself is not well-defined i.e. you never have label to be sure who is who so you need to define closeness of points from which the problem starts; how close is called closeness?!!
Be careful about Curse of Dimensionality while using k-means for time-series clustering (I'm not sure how you do it).

After these points let's have a look at your question.

What are time-series? If time-series are pretty non-stationary or simply speaking is the dynamic behind variation is complicated enough, then there is not a one-to-one map for time-series characteristics and time points (imagine a simple ECG signal. It's pretty simple but if you explore research community you will find super sophisticated methods for feature extraction on ECGs. I'm pretty confident finding a time point at which ECGs differ is almost impossible.). You may extract features from your time series or embed it into some n-dimensional manifolds and look at it. In best case you may find some time-related features which describe your time-series and you may find some time-related criteria at which time-series differ (however I'd say it's not likely to find them).

Assuming time-series are pretty well-behaved (!!) with a simple dynamic (should be super simple). Then a solution might be to define a distance function of time-series which outputs the pair-wise distances of all time-series as a single score. Then the maximum of this function returns the time-point at which these time-series are pretty distinguishable.

If you provide more information on your data I might be able to give a more detailed precise answer.

Good Luck!


How to retrieve tweets from all cities using Twint package in python?

You are only seeing the last one since you overwrite your object df at each iteration and when the loop is finished it will store only the last value store which happens to be mumbai
If you want to create a DataFrame with all the results you should do something like this:
ldf = list()
india = ["chennai","pune","mumbai"]
for city in india:
    print(city)
    c = twint.Config()
    c.Search = "phone"
    c.Lang= "en"
    c.Pandas = True
    c.Since = "2021-01-01"
    c.Near = city
    c.Limit = 10
    twint.run.Search(c)
    tmp = twint.output.panda.Tweets_df(["date","tweet","near"])
    ldf.append(tmp)
data = pd.concat(ldf, ignore_index = True)
data.head()


Bias-variance tradeoff and the uncertainty principle

No, the uncertainty principle describes a property that is specific to electrons. That electrons don't display their wave and particle properties simultaneously. Here from Wikibooks:

  The Heisenberg uncertainty principle states that the exact position
  and momentum of an electron cannot be simultaneously determined. This
  is because electrons simply don't have a definite position, and
  direction of motion, at the same time!

.

  This is due to the fact that electrons cannot exhibit both their wave
  and particle properties at the same time when being observed to
  interact with their surroundings. The momentum of an electron is
  proportional to its velocity, but based on its wave properties; its
  position is based on its particle position in space.

The bias-variance tradeoff describes a property of predictive models. That error from bias and variance usually exist in a tug-o-war where you cannot decrease one without increasing the other and that you have to find the perfect middle ground. You can observe it here from this Medium:

I agree that there are some similarities. There is a sort of tradeoff between measuring the momentum and position of electrons. But they are not the same thing.


Is it wrong if I cluster numerical attributes and categorical attributes separately?

Generally, clustering on separate categorical and numerical features is wrong since it could lead to merging the otherwise separate clusters. Here is a visual example of why this may fail (drawn by myself):

If we cluster only on the categorical feature, clusters C1 and C2 would be merged. If we cluster only on the numerical feature, all three clusters would be merged. Therefore, clusters C1 and C2 could not be found separately.

As a side note, this blind separation is different than a careful feature selection (mentioned in this answer) which could end up with both categorical and numerical features.


How to develop a good model for predicting electricity usage?

Adding as a answer instead of comment due to insufficient reps.
Consider developing a model that can identify the pattern of electricity and water usage per house throughout the year. And predict the upcoming requirements. It will be a time series problem. This will help in maximising resource utilisation and better availability of resources among everyone.
Later, this can help in smart power and water distribution which is not yet applied.
If you want a bigger community working into this data set, you can share it online or provide it to Kaggle. Many people are out there ready to help in it.


Pass the panda series directly to pandas.to_datetime() function. It works faster than using apply function and passing pandas.to_datetime as a parameter.

In[20]: %timeit -n 1000 pd.to_datetime(mdp)
96.1 µs ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In[21]: %timeit -n 1000 mdp.apply(pd.to_datetime)
5.59 ms ± 91.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Incase of dataframe, looping through the series works fine as well.

In[45]: %timeit [pd.to_datetime(df[x]) for x in df]
230 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

In[46]: %timeit df.apply(pd.to_datetime)
1.22 ms ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


How keras.layers.embedding learn word embeddings?

The keras embedding layer is initialized with random weights and will learn an embedding for all of the words in the training dataset. So,the output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all possible vectors.
You could also use pre trained word2vec embedding.
Refer this to know more.


Automatic question categorization when we know important words in each category

You said you have a bunch of keywords for each category extracted from the data. It is better to explain how did you get them as your data does not contain labels i.e. how do you know the word $w_i$ is a representative of category business?

You might have assigned them based on semantics which is not bad but does not give you a level of confidence that they are really the representatives of category. The whole point about clustering is to find this representative words via the algorithm. An example is the famous concept of Stop Words. Everyone thinks that words such as $The$, $is$, $an$ and the rest of common lists are stop words but empirically you always get better classification result including them! That's because in NLP the only reference to indicate a stop word is the data itself. So be careful about it.

Assuming your words are really representatives (however I would rely on LDA results), I would design a score for similarity of each question to each category representation list. It could be the category which has the maximum number of representatives in the sentence. It most probably falls in overfitting problem however you will not see that (your problem is unsupervised) but if you insist using them it could be a starting point.

In the direction of this idea, another one could be to extract the very extreme words for each category (e.g. "Manchester united" and "Neymar" are more probable to represent sport) then do the same clustering with only extreme words and based on found categories find more representatives and feed them to the algorithm again (iterative manner) and continue till some stopping criterion is satisfied. This looks like above but the difference is that the least input (first set of words) are fed manually and the rest will be found through algorithm.

You also may have a look at embedding algorithm such as word2vec or doc2vec and use them with some similarity measure (cosine similarity for example).


How many features can we input for a SVM to classify?

There is no restriction on the number of features.

The syntax is exactly the same. For your tutorial, rather than

X = iris.data[:, :2]

That takes only $2$ columns, just change it to a bigger number if those columsn area in your dataframe. It is a common practice that the last column is the $y$ values of which case just to remove that column, just use $-1$.

For some online tutorials, sometimes we use $2$ features for the convenience of visualization only.

However, note that we do not want to use too many features as we want to build a model can be generalized. A simple model that works is better than a complicated model.


How to Transpose dataframe with custom output?


CSV is a format meant to machine and human-readable. Most programming languages have support to use CSV's, and Python is certainly one of them.

There is a standard library that would help, you will find it here: https://docs.python.org/3/library/csv.html#module-csv

I would recommend however to install the pandas package, to read the CSV into a structure made for analysis. There is a tutorial here: https://www.datacamp.com/community/tutorials/pandas-read-csv


It sounds like you have good initial data but you might need to develop a meta data set. I would start by taking all this info and building behavior profiles. How often do they query per session? When do they query? How many sessions per hour/day/week/month? And so on, I think you get the idea. It just doesn't strike me like the raw file is the way to go, you need a little bit more to work with but you can get there with what you have now. Get down to various behaviors, one ID per row, and you'll be in a much better position to model this out.


Your target y can be whatever you need.

If you want to do sequence to sequence for example, y will have the same number of time steps than the inputs (you predict something for each timestep).

But you can also define an output with one timestep only, for text classification for example.
Your input data has 10 timesteps, but the output is only a prediction at the last timestep.

It depends on the problem you’re trying to solve.


How large of a value should a weight have in a neural network?

One of the problems that can occur when training a neural network is known as the exploding gradient problem. A poorly initialised network could lead to a large increase in the norm of the gradient during training. These larger values will basically run the weights out of the number precision of the computer, resulting in NaN values.

This post gives more information on the exploding gradient problem and how to solve it. A related post discusses different initialization strategies.