Some feedback/tips/tricks/opinions here:
Problem setup
Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there.
Model evaluation.
It is very desirable to have a quantitative way to evaluate our model performance.
For that we want some labeled data. It is very quick to count the number of ON events (given assumptions) for each house. House 1, 4 times. House 2, 3 times. House 3, 2 times.
Then at least we can use that as an evaluation metric.
Could also mark the times in the dataset to compute a percentage of ON time. That would be a more granular/precise metric to evaluate by. Strongest evaluation would be to check that each event time was also correct (within some tolerance).
Generalization
A model trained on N TVs/houses will also work on unseen houses. Very desirable, as allows the same model to be used for all houses, including future ones.
For this reason one should do possibly do Cross Validation grouped by house.
Might also want to use Time-Series splitting, but a bit hard on such a tiny dataset.
System behaviour assumptions
We assume that that Power(ON) >> Power(Standby) > Power(Off).
However each TV might have different Power levels, as well as differences/ratios between the three levels. Different houses may also have different behavioral patterns in turning on/off.
Minimum on time.
It is reasonable to assume/set some minimum time for TV on/off. Can be informed from data, or set based on what you know about TVs. Maybe 10 seconds is OK?
Technique feedback
EDA
Really missing a histogram over power level!
Denoisning
KNearestNeighbours is too complex/opinionated for this. A running mean or running median would be more suitable.
Classifier
GradientBoostingClassifier wayy to complex. You have a single feature and tiny amounts of data.
GaussianMixtureModel is quite relevant, as this is a classic case of Hidden States and transitions. However it is still not clear if the complexity is justified - ie if the problem cannot be solved with just state-independent thresholding
Report feedback
Your notebook indicates heavy work-in-progress, it does not have the qualities of a report. Of course if you did this live, on-the-spot or with a very tight deadline, that is natural. But if you had several hours on this task, I would expect a much cleaner notebook.

The intro should define/re-state the problem(s). So that the report is self-container
An Exploratory Data Analysis section should be before models. Question -> Data Exploration -> Findings.
All non-interesting findings or explorations should be removed!
Low use of global variables, proper use of functions. Models should use pipelines where appropriate
Use pandas time series functionality. All time-series plots should have time on axis!
After the EDA and models, Results, and then a Discussion/Conclusion
You may want to also have the most important findings/conclusions summarized on top


shifting the last column in the dataframe to the first place

cols = list(df.columns)
cols = [cols[-1]] + cols[:-1]
df = df[cols]


Previous work Replication and Research ethics Ask Question

I wouldn't think so. If they're publishing their methodology, they want other people to see how well it works and apply it to their work. You'll probably want to explain why you think this method works best for your dataset and compare the performance results to other methods commonly used.


I want to run PCA on a data set that will be aggregated by country. Should I aggregate the data before or after I standardize the data, and why?

Data should be standardized for calculating principal components. If data is standardized before aggregating by country, the resultant data after aggregation will no longer will have mean 0 and standard deviation 1, hence, the results of PCA will be erroneous.
Therefore, data should be aggregated and then standardized in order to obtain correct principal components of the data.


Which type of regression has the best predictive power for extrapolating for smaller values?

OK, so you apply linear regression to create a model for your data, and when you use that model to predict new values, the output values don't satisfy a constraint (namely, being positive). I can think of only a few different things that might be going on here:

The new input values you are giving to the model are not within the allowable range for the problem. This is unlikely given your setup - presumably you know your situation well enough to determine what input values are allowed - but if this does happen to be the case, you need to change the way in which you obtain inputs.
A linear model is not accurate over the full range between your original input data and new input data. Without more domain knowledge, it's impossible to tell what sort of model would be a better substitute. If this is a situation where you can come up with some sort of theoretical prediction about what general form of model should describe the data, you should use that form. If not... try an exponential I guess?
A linear model is accurate, it's just not this particular one. You might have to do something like forcing the regression to include a particular point (e.g. the origin), for example. That would mean that you're not actually doing a full linear regression, but rather fitting a model with one fewer free parameter - e.g. in slope-intercept form, you're basically forcing the intercept to zero and only fitting the slope. Or the equivalent if using another point. This should be doable with some regression library that you can use. But if you use a linear model, it will predict negative values somewhere. You'll have to live with declaring those regions of parameter space to be outside the domain of the model (or possibly finding a way to make sense of negative values).

Based on the little information you've provided here, there's really not much more I can say. Maybe if you described your situation in more detail, I could offer more specific advice.


How to handle overfitting in the following classification case

I recommend including some more information in your post - you'll be able to get more help that way. What is the size of your dataset? How many epochs are you training for? What is the accuracy and loss value after training?

Nonethless, I recommend taking a look at the early stopping callback.

It will stop training when your model stops improving according to a given metric. I've found this useful when using loss as the metric. You should also definitely monitor the loss and accuracy of your model during each epoch and stop training when they stop improving. Tensorboard, another callback, can help with this.

You might also want to try reducing the number of nodes you have per layer, as a model with more parameters has a greater capacity for memorization, reducing your model's capability to generalize. 300 nodes is a lot to have in the first layer of your model for data with just 11 features, but to be honest everything really all depends on the data.


This basically asks for a recurrent network, like the LSTM. But if you only have 2 properties that are dynamic, I don't think you will have as much luck because they might be affected from external factors as your boss said. However, this will happen regardless of the model you're using.

You should not throw away static properties, unless they are the same for every test case. E.g. farmer/politician/baker category should always be included, but you call this 'static' but it's not completely static as it is not the same for every sample in your test cases.


Well, I think it is better to watch a course about it.
But, for instance, when people comment on different products on Amazon, there is a way to understand are their comments positive or negative, therefore their comments are used as data, analyzed and then labeled as positive or negative. Further, using this approach, one can find whether an email is spam or not, etc.

In order to analyze a text, we need to split it into its elements, meaning, its words (usually) each of these elements is called a token.

after splitting the text to words, useless words that do not represent anything, some signs, etc (the, ".", "my",...) are omitted, then we create bags of words, that are the most important words in the whole dataset (all comments), then each word is a column in the table and each row of the table is a comment and has 1 in a column if it contains that specific word. there is also a label column that shows if the comment is positive or not. then having this dataset, we can fit a classification model to the dataset.


Convert .cbf to lossless image format with defined "dynamic range"

import os
from glob import glob
import fabio

path = '/Users/myuser/Documents/Crystals'

result = [y for x in os.walk(path) for y in glob(os.path.join(x[0], '*.cbf'))]

i = 0
command = ''
while i < len(result):

    path = os.path.dirname(result[i])
    filename = os.path.splitext(result[i])[0]+'.tif'

    command = '/Applications/Adxv.app/Contents/MacOS/Adxv -smv32bits -sa -overload 500 '+result[i]+' '+filename
    os.system(command)

    i += 1

In the end I figured it out using Adxv in stand alone -sa and -overload N as dynamic range.


Identifying most informative (sub)words/vectors that help classify a sample

Note 1: I believe the technical term for what you are looking for is '(sub)word salience'.

First thought (which is not the best method, but could be worth a quick try if your examples are smallish) - Run the text with all words through fastText, and get the baseline most likely labels with probabilities (predict-prob function). Then remove each word that you are interested in testing, (maybe removing common words like 'a' 'the' etc), and compare the predicted probabilities of each obtained set of predicted_probs. The ones that give the greatest difference between baseline and missing wordX can be interpreted as the most informative.

A permutation of this idea is to get the embedding for the interesting words, and then get a distance measure (I assume cosine distance) to the labels you have. Then you have a problem, as they may be distributed evenly across labels in terms of distance, but if there is a group of words that are close to a particular label, that can be interpreted as informative.

I am sure there is a better technique to do this, so I am curious as well :). You may want to check these out (but they use additional modeling to get salience)

Learning Neural Word Salience Scores.

A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION - this deals with the exact problems of how to identify the useful bits of an input text


Split into test and train set before or after generating document-term matrix?

I gave a lot of thought about the question. I agree with you. But the slight difference might come if there are any random variable operation happens during the training. What model are you using for training?


Input changing row count matrices in an MLP

First, I made the matrices (x, 129) to have the same shape by padding in with the rows filled with zeros.
Second, I transformed my large matrices into dimension 3 by the following process:
train = list(train_df['spec'])
np.array(train).shape
>>> (50395, 71, 129)

X_train = list(train['spec']);     X_valid = list(valid['spec'])
y_train = train['label'].values;    y_valid = valid['label'].values
model.add(Dense(12, input_shape=(50395, 71, 129), activation='relu'));
model.fit(np.array(X_train), y_train, validation_data=(np.array(X_valid), y_valid), epochs=500, batch_size=1);


Calculating correlation between two time variables

I don't think you can use Pearson's correlation because it is used for continuous variables. Your variables are ordinal, so a test like Spearman's would be more appropriate. However I don't think that tests for ordinal variables would be appropriate either, because your variables are also cyclical, in the sense that Hour_ofDay=23 and Hour_ofDay=1 are really just 2 hours apart, but for the Spearman's test they would be considered 22 hours apart.

I think in this case it would be more appropriate to look at the distribution of the distance (measured in hours) between the two variables.
The appropriate distance metric in this case is defined as follows (distance originally defined in the accepted answer to this other question)

import numpy as np
distance = np.sign(a1-a2)*(12 - abs(abs(a1 - a2) - 12))

Where a1 and a2 are your App and Email opening time variables. Note that the variables need to be in the range 0 to 23 for this distance to work.

Compute this distance for each row, add it as a column to your dataframe and plot it with an histogram.
This histogram will tell you quite a lot about the "correlation" between the two variables.

For example

No correlation: the histogram will be uniform between -12 and 12
Instantaneous correlation, i.e. user opens email and app at the same time: the histogram will have a peak at 0
Anticipated delay, i.e. user opens email 1 hour before app: the histogram will have a peak at 1
Delayed correlation, i.e. user opens email 1 hour after app: the histogram will have a peak at -1

This visualization will allow you to draw rich conclusions about the relation between App- and Email opening times.

Note: if your variables also include minutes and seconds in a date format you have to convert the variables to numerical. E.g. 01:30 (hour 1 and 30 minutes) becomes 1.5 . Also pay attention to the date format in case you have time expressed in the 12-hour clock (e.g. 6 PM, 1 AM)


BERT : text classification and feature extractionn

You'd need to apply a tagger, either a generic NE tagger or a custom-trained one. The tagger works with each token as an instance, so that you can extract a particular sequence of tokens, e.g.:

$15.00     Begin_Billing_rate
hour       Billing_rate
customer   _
service    _
,          _
open       _
to         _
industries _

Of course in order to train a custom tagger you will also have to annotate your data token by token.


Categorical Variable and Target Variable

Reading from your comments, it appears you can use more that one predictor for your target variable.

If you want to understand whether Gender (M/F) has a significant association, you should run a logistic regression using it together with other predictors. This would let you control for the impact of Gender all things equal, i.e. after you controlled for the effects of other variables.
Logistic regression will return significant scores and standard errors for each variables.
Alternatively, you could look at relative importance scores from tree-based models such as Random Forest or XGBoost. (However, Logistic regression is better for hypothesis testing.)

BUT: If you still want to stick with bivariate statistics, i.e. using just one predictor at a time, you can use simpler tools such as T-test or one-way ANOVA to understand whether belonging to one gender or the other makes a difference (statistically speaking).

Hope this helps, otherwise let me know.


What is the purpose of a 1x1 convolutional layer?

First of all, Conv2D(16, kernel_size=(1,1)) applied to 28x28x1 will produce 28x28x16 as it doesn't change the number of dimensions. Secondly, I took a look at the paper you refer to, and haven't found that they apply 1x1 filters to an input with just one channel as in the example you provided. In case of just one channel in the input your reasoning is correct, but it cannot be transferred to input with several channels.

Conv2D by definition, assign weights between all channels of a selected area (in case of 1x1 kernel to a whole column of nodes) and corresponding column of nodes in the output grid. Then it applies these weights subsequently to all valid areas according to stride and padding parameters. Thus applying 1x1 kernel on 1-channel input doesn't make sense as all output channels will be proportional to each other and to the input channel as well. In case of multiple channels, as per definition Conv2D each node of the output will be a weighted (non-trivial) sum of a corresponding input column and won't be proportional to either channel of the input in most cases.


How to get non-normalized feature importances with random forest in scikit-learn

It seems you are normalizing the Gini dip for every tree and then summing those values.
 You should do the normalization at the end after the sum.
Result might vary depending upon the overall numbers in the two scenarios.
See this dummy example
import numpy as np
# Three Tree and 3 Features. F1 sum across Trees = 50, F2 = 48, F3 = 20
gini_dip = np.array([[20, 15, 5], [20, 15, 5], [10, 18, 10]])

# Normaliation before sum
gini_dip_n = np.array([((elem-min(elem))/(max(elem)-min(elem))) for elem in gini_dip])
fi_1 = gini_dip_n.sum(axis=0)

# Normaliation After sum
gini_dip_sum = gini_dip.sum(axis=0)
fi_2 = (gini_dip_sum - gini_dip_sum.min())/(gini_dip_sum.max() - gini_dip_sum.min())

fi_1, fi_2

(array([2.        , 2.33333333, 0.        ]),
 array([1.        , 0.93333333, 0.        ]))

F2 is best in scenario #1 and F1 is best in scenario #2


There is an easy way to use one hot encoding in pandas and you can read about it in the following link:

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html

The get_dummies method is easy and optimized for use in pandas Data Frames.

Good luck!


How to make model for Multivariate timeseries using tensorflow probability structural model?

You just need to ensure that the model you forecast with has a design matrix covering both the observed and forecasted timesteps. That is, you'd build a model including a component along the lines of

sts.LinearRegression(
  design_matrix=tf.concat([temperature_for_observed_timesteps,
                           temperature_for_forecast_timesteps], axis=-2),
  name='temperature_effect')

(ignoring any centering and reshaping logic) and then pass that model to the forecast method.

If you don't have access to future values of the external regressor when you first build the model, a useful pattern is to encapsulate model building in a method def build_model(observed_time_series, design_matrix) that returns a StructuralTimeSeries model object. Then you can build an initial model with just the observed time steps in order to fit parameters, and then remake the model later once you have the regressors for the forecast steps on hand.


Machine learning for state-based transforms?

If i get it correctly:

You have an input polygon
As a first step you want to "match" that against a list of previously seen templates. If this is successful, you pick it's corresponding output and move on.
If not, you wish to find some optimal transformation, in order for it to satisfy some constraints that you have (your "objective function"). Then add the original+transformed shape to the templates list and move on.

Is this correct? I'll risk an answer anyways:

For the first part, I believe that there is a slew of literature out there. It's not my expertise, but first thing that comes to mind is measuring the distance in feature space between your shape and each template, and picking the closest one, it the distance is below a threshold that you set. "Feature" here would be either some low-level polygon property, e.g. x and y coordinates of vertices, or an abstraction, e.g. perimeter, area, no. vertices, mean side length/side length variance, etc.

For the second part, it really depends on the nature of your constraints/objective functions. Are they convex? Uni- or multi-modal? Single or multi-objective? Do you want to incorporate some domain knowledge (i.e. knowledge about what "good" transformations would be?)? One can really not tell without further details. Evolutionary algorithms are quite versatile but expensive methods (although some argue on that). If you can spare the possibly large amount of function evaluations, you could try EAs as a first step, and then refine your approach.

Finally, while not exactly related to what you describe in your process, I believe you may benefit by taking a look into auto-associative networks (and models in general); these are models that are able to perform constraint-satisfaction on their input, effectively enforcing learned relationships on input values. I could see this being used in your case by inputing a shape, and having a transformed shape as an output, which would be "legal", i.e. satisfying the constraints learned by the auto associative model. Thus, you would eliminate the need for a template matching + optimization altogether.


If you are looking to use US Census data, the American FactFinder website: https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml allows you to search for Census data in different ways and then download tables in csv format.

For example, on the Community Facts site, you can enter a state, county, city, town, or zip code and obtain the population of the geography that you entered.  Then you can click on the subjects listed on the left side of the page (Age, Business and Industry, Education, etc.) to see other interesting information about your geography, or click the links under "Popular tables for this Geography" to see tables containing other data for your community.

Or you can use the Advanced Search feature to search all available data using all of the search features provided by the site.

Also, I don't know much about this myself, but there exists at least this one Github repository with "SQL scripts and other supporting files for importing recent American Community Survey releases from the US Census Bureau datasets".  The scripts are released under the GNU General Public License.


XGBoost: how to adjust the probabilities of a binary classifier to match training data?

The first (and easiest) option is to make sure that your model is calibrated in probabilites. In Python, it means that you should pass the option binary:logistic in your fitting method.
The alternative is to transform the output of your model into probabilities. There are different approaches for that.
This could be achieved with some sort of regression techniques to find the relationship between probabilities and your output. Python's isotonic regression should work for that purpose. However, without more information on your score ditribution it is possible it doesn't work well.
This can also be achieved with platt scaling : transforming your output into binary prediction (0 and 1) with a threshold, then calibrate a logistic regression on those new variables. It is relatively easy to do, but in my experience doesn't necssarily work well with unbalanced problems with non-linear relationships.
Finally, there are some approaches that just correct the output depending on your model. For logistic regression that would mean changing your bias variable so that the overall predicted proportion match the one of your data-set. This also can be used to counter the effects of rare events (see this). I have found this to work wuite well with logistic regression. However, I am not sure if it would directly be appliable to XGBoost, but it could be worth a try.


Do I need a next for loop in order to get all values?

Don't forget to open the file first..

Py 2.6

For item in list:
    the_opened_file.write("%s\n" % item)

Py 3.x

with open(filepath,'w') as fileptr:
    for item in list:
        fileptr.write("{}\n".format(item))

If you want to save your list itself,
Convert it to numpy format using np.assarray(list) and then use np.save(file,array,....(other opts) and reload it whenever required using np.load()


How to add additional information in a neural network other than putting as input data?

If your inputting data, it will always be an input :). But it is definitely possible to add in certain inptus after the first hidden layer. But what is your reasoning behind this? Why would you want this?

You should just input all your data in the first layer, let the neural network figure out how to use the data effectively itself. The neural network is like a black box, you put something in and something comes out - don't interfere too much by adding seperate inputs to the hidden layers.


What is difference between one hot encoding and leave one out encoding?

They are probably using "leave one out encoding" to refer to Owen Zhang's strategy.
From here
The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, excluding the row itself. This gives you the advantage of having a one-column representation of the categorical while avoiding direct response leakage
This picture expresses the idea well.


K-Means clustering for mixed numeric and categorical data

The standard k-means algorithm isn't directly applicable to categorical data, for various reasons.  The sample space for categorical data is discrete, and doesn't have a natural origin.  A Euclidean distance function on such a space isn't really meaningful.  As someone put it, "The fact a snake possesses neither wheels nor legs allows us to say nothing about the relative value of wheels and legs." (from here)

There's a variation of k-means known as k-modes, introduced in this paper by Zhexue Huang, which is suitable for categorical data.   Note that the solutions you get are sensitive to initial conditions, as discussed here (PDF), for instance.

Huang's paper (linked above) also has a section on "k-prototypes" which applies to data with a mix of categorical and numeric features.  It uses a distance measure which mixes the Hamming distance for categorical features and the Euclidean distance for numeric features.

A Google search for "k-means mix of categorical data" turns up quite a few more recent papers on various algorithms for k-means-like clustering with a mix of categorical and numeric data.  (I haven't yet read them, so I can't comment on their merits.)

Actually, what you suggest (converting categorical attributes to binary values, and then doing k-means as if these were numeric values) is another approach that has been tried before (predating k-modes).  (See Ralambondrainy, H. 1995. A conceptual version of the k-means algorithm. Pattern Recognition Letters, 16:1147–1157.)  But I believe the k-modes approach is preferred for the reasons I indicated above.


Re-bucket weekly sales data and calculate descriptive statistics

If the week ID is given as you state, calculate a bucket variable $w_x=int(weekID/x)$.  Then use a SQL statement to summarize the volume to levels of $w_x$.


Pattern Recognition, Bishop - (Linear) Discriminant Functions 4.1

No, $y(\mathbf{x})=\mathbf{w}^T\mathbf{x}+w_0$; it is a scalar.  The dot product $\mathbf{w}^T\mathbf{x}$ is $\|\mathbf{w}\|$ times the length of the projection of $\mathbf{x}$ onto $\mathbf{w}$.  $w_0$ in your figure would be negative, and has the property that $y(\mathbf{x})=0$ whenever $\mathbf{x}$ is on the decision boundary.
No normalization appears to be necessary.  Certainly $r$ can be arbitrarily large (either positive or negative), when $\mathbf{x}$ is far away from the decision boundary.
As mentioned previously, the first term is actually a length from the origin; the bias serves to shift this so that $y$ itself is the orthogonal (scalar) component [up to scaling by $\|w\|$...But that's what we're out to show when we pass from Eq(4.6) to (4.7)].

The text's approach is to decompose $\mathbf{x}$ into components relative to the decision boundary: $\mathbf{x}_{\perp}$ on the boundary, and something perpendicular to the boundary.  Being perpendicular to the boundary, it is in the direction of $\mathbf{w}$, but we don't know how far, so they introduce its length as the unknown $r$.  (There's some standard geometry stuff here that could also get us to the conclusion, but I'll explain their approach.)

Now, as mentioned before, $y$ is zero on the boundary, so they have $y(\mathbf{x}_{\perp})=0$.  And now, just to fill in some of the details of what they say,
$$\begin{align*}
\mathbf{x} &= \mathbf{x}_{\perp} + r \frac{\mathbf{w}}{\|\mathbf{w}\|}\\
\mathbf{w}^T\mathbf{x}+w_0 &= \mathbf{w}^T\mathbf{x}_{\perp}+w_0 + r \frac{ \mathbf{w}^T\mathbf{w} }{ \|\mathbf{w}\| } \\
y(\mathbf{x}) &= y(\mathbf{x}_{\perp}) + r \frac{\|\mathbf{w}\|^2}{\|\mathbf{w}\|} \\
y(\mathbf{x}) &= 0 + r \|\mathbf{w}\|,
\end{align*}$$
and so $r=y(\mathbf{x})/\|\mathbf{w}\|$ as desired.

[I'm not sure what you meant in your last few lines; at least some of it seems to have been a typo?  Feel free to follow up.]

EDIT: Regarding your addition about substituting $r$, you should get $$\mathbf{x}=\mathbf{x}_{\perp}+y(\mathbf{x})\frac{\mathbf{w}}{\|\mathbf{w}\|^2},$$
but $\|\mathbf{w}\|^2$ is not equal to $\mathbf{w}$; the former is a scalar, and the latter a vector!  Rewriting, we have
$$\mathbf{x}=\mathbf{x}_{\perp}+\frac{y(\mathbf{x})}{\|\mathbf{w}\|}\frac{\mathbf{w}}{\|\mathbf{w}\|}.$$
This now looks correct: from the origin, go to $\mathbf{x}_{\perp}$, then along the unit vector $\mathbf{w}/\|\mathbf{w}\|$ for a distance of $y(\mathbf{x})/\|\mathbf{w}\|$ (which, per your bubble3, is the correct distance).


ax = sns.regplot(x="Value", y="dollar_price", data=merged_df, fit_reg=False)

You have the fit_reg set to False here. Just set it True, you should see the regression line.


http://en.wikipedia.org/wiki/Named-entity_recognition#Formal_evaluation :

  To evaluate the quality of a NER system's output, several measures
  have been defined. While accuracy on the token level is one
  possibility, it suffers from two problems: the vast majority of tokens
  in real-world text are not part of entity names as usually defined, so
  the baseline accuracy (always predict "not an entity") is
  extravagantly high, typically >90%; and mispredicting the full span of
  an entity name is not properly penalized (finding only a person's
  first name when their last name follows is scored as ½ accuracy).

  In academic conferences such as CoNLL, a variant of the F1 score has
  been defined as follows:

  Precision is the number of predicted entity name spans that line up exactly with spans in the gold standard evaluation data. I.e. when
  [Person Hans] [Person Blick] is predicted but [Person Hans Blick] was
  required, precision for the predicted name is zero. Precision is then
  averaged over all predicted entity names.
  Recall is similarly the number of names in the gold standard that appear at exactly the same location in the predictions.
  F1 score is the harmonic mean of these two.

  It follows from the above definition that any prediction that misses a
  single token, includes a spurious token, or has the wrong class,
  "scores no points", i.e. does not contribute to either precision or
  recall.


Linear Discriminant Analysis, which parameters can be tunned in cross validation set up?

LDA has a closed-form solution and therefore has no hyperparameters. The solution can be obtained using the empirical sample class covariance matrix. Shrinkage is used when there are not enough samples. In that case the empirical covariance matrix is often not a very good estimator.


Is it possible to identify different queries/questions in sentence?

The basic thing, you can do in that situation, is to split your query into N simple sentences each of which should be processed in order to receive YES/NO answer considering if the sentence is a query. That way you will receive following results:

Input: Gandhi is good guys, Where he was born?
->
Gandhi is good guys - not query
Where he was born?  - query
===
1 query

Input: who is MGandhi and where was he born?
->
who is MGandhi     - query
where was he born? - query
===
2 queries

This approach will require anaphora resolution (in order to convert he into Gandhi in first example) and a parser to correctly divide complex sentence into simple ones.


What are recommended ways\tools for processing large data from Excel Files?

pandas loads a csv upto 10x faster than excel. So, if you can, please convert these files to csv, which ever ones that are being loaded several times.

I have been using cython, which seems to speedup read functions and processing. Please follow this link for more details. It says, that using cython speeds up the processing 10x.

Suggestion: If you can, I would recommend to store data in database and use an orm like django/peewee for the data processing.

Please lemme know if you have any other queries.


On which step should use SMOTE technique for over sampling?

If you are using python, you can't use SMOTE in the presence of null values.

In this case:

Remove Outliers
Smooth Data
Impute null values (there are some smart options for that in R: using random forests to impute)
SMOTE

Removing outliers first let you do better smoothing and imputing.


How to compare the topic coherence between models of different number of topics?

After doing some more digging -- it appears that gensim's functions output difference calculations

This below actually does what I am describing above:

cm = CoherenceModel(model=model, texts=docs, dictionary=dictionary, coherence='c_v')

While this below is actually only the summation across the topics of the model not divided by the total number of topics in the model.

top_topics = model.top_topics(texts=docs,dictionary=dictionary,coherence='c_v')


Using a past time series to predict how a present time series will pan out?

Here you go: https://otexts.com/fpp2

But my experience is you should start using GBM based models for time series forecasting to get really good forecasts.

Check out XGBoost, Prophet, BSTS models for time series forecasting.


Is it feasible to train a single model on 150+ classes?

A Classifier's quality depends from the number of available observations for each class. If you have 300 images for each class, as you said above, it might be enough (it depends from the quality of your data).
In any case, it would require a massive amount of data augmentation.

Training a Network on 240 classes could be very challenging, if you don't have a proper infrastructure. That all depends on training times, but I'd suggest you to break down the problem in smaller subproblems. That would also make debugging and improvements easier to implement.


What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way?

They're two different algorithms, but there is some connection between them:

Gradient descent is an algorithm for finding a set of parameters that optimizes a loss function.  Given a loss function $f(x, \phi)$, where $x$ is an n-dimensional vector and $\phi$ is a set of parameters, gradient descent operates by computing the gradient of $f$ with respect to $\phi$.  It then "descends" the gradient by nudging the parameters in the opposite direction of the gradient.  This process is repeated for different points in the space of inputs (i.e. different $x$s) until a minimum of $f$ is found.

Gradient boosting is a technique for building an ensemble of weak models such that the predictions of the ensemble minimize a loss function.  I think the Wikipedia article on gradient boosting explains the connection to gradient descent really well:

  . . . boosting algorithms [are] iterative functional gradient descent algorithms. That is, algorithms that optimize a cost function over function space by iteratively choosing a function (weak hypothesis) that points in the negative gradient direction.

So the connection is this:  Both algorithms descend the gradient of a differentiable loss function.  Gradient descent "descends" the gradient by introducing changes to parameters,  whereas gradient boosting descends the gradient by introducing new models.


what make lightGBM run faster than XGBoost?

First of all both the GBM methods are great and superiority of each algorithm is dependent on the data.
Major Differences between the two is that LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split.
GOSS assumes that data points with small gradients tend to be more well trained. This means that it is more efficient to concentrate on data points with larger gradients.To attenuate the problem of biased sample it also randomly samples from data with small gradients.
To read more you can refer:

Paper
LightGBM_Documentation


There is a k-medoids clustering for PySpark at spark-packages.org/package/tdebatty/spark-kmedoids and the source code is github.com/tdebatty/spark-kmedoids.

It can be installed with:

> $SPARK_HOME/bin/spark-shell --packages tdebatty:spark-kmedoids:0.1.2


The added layer must be an instance of class Layer. Found:

How did you import "BatchNormalization" layer?
I had the same error and the solution was changing the way I was importing:

from tensorflow.keras.layers import Activation, Dropout, Dense, Conv2D, Flatten, Dropout, MaxPooling2D, BatchNormalization

I put the BatchNormalization with all the other layers, and it work in my case.

Hope it is useful.


It happens often, that some model only predicts one class. The reason usually is, that the model is unable to distinguish the two classes well, and resorts to one (often the majority) class.
With your data, you may have a hard time to fit a NN with reasonable results. I suggest you check boosting which usually works okay with small data. Make sure you tune parameters well.
Here is an application of lightgbm to the iris data.
You could also check Logit with L1 regularization.


Adding extra variables to XGboost model is worsening the train and test accuracy

This is something you can try, remove the parameter early stopping rounds and the number as I see is relatively low. It could be that xgboost stopped training before it is supposed because it may not have seen any improvement. Also you might consider setting random seed to get consistent result between attempts.


It means your logistic classifier is biased towards one class, this could be because of below reasons that I can think of.

Class Imbalance:  This article explains how to identify and overcome the class imbalance problem.
Overfitting: This article explains how to tackle over fitting.

Logistic classifier works better if data is linearly related, if you find non-linear relationships in data I would suggest use better algorithms like GBM/SVM/Random forest, which will give you much better and accurate results.


Statistics + Computer Science = Data Science?

Well it depends on what kind of "Data Science" you wish to get in to.  For basic analytics and reporting statistics will certainly help, but for Machine Learning and Artificial Intelligence then you'll want a few more skills

Probability theory - you must have a solid background in pure probability so that you can decompose any problem, whether seen before or not, into probabilistic principles.  Statistics helps a lot for already solved problems, but new and unsolved problems require a deep understanding of probability so that you can design appropriate techniques.
Information Theory - this (relative to statistics) is quite a new field (though still decades old), the most important work was by Shannon, but even more important and often neglected note in literature is work by Hobson that proved that Kullback-Leibler Divergence is the only mathematical definition that truly captures the notion of a "measure of information". Now fundamental to artificial intellgence is being able to quantify information.  Suggest reading "Concepts in Statistical Mechanics" - Arthur Hobson (very expensive book, only available in academic libraries).
Complexity Theory - A big problem many Data Scientists face that do not have a solid complexity theory background is that their algorithms do not scale, or just take an extremely long time to run on large data.  Take PCA for example, many peoples favourite answer to the interview question "how do you reduce the number of features in our dataset", but even if you tell the candidate "the data set is really really really large" they still propose various forms of PCA that are O(n^3).  If you want to stand out, you want to be able to solve each problem on it's own, NOT throw some text book solution at it designed a long time ago before Big Data was such a hip thing.  For that you need to understand how long things take to run, not only theoretically, but practically - so how to use a cluster of computers to distribute an algorithm, or which data structures take up less memory.
Communication Skills - A huge part of Data Science is understanding business.  Whether it's inventing a product driven by data science, or giving business insight driven by data science, being able to communicate well with both the Project and Product Managers, the tech teams, and your fellow data scientists is very important.  You can have an amazing idea, say an awesome AI solution, but if you cannot effectively (a) communicate WHY that will make the business money, (b) convince your collegues it will work and (c) explain to tech people how you need their help to build it, then it wont get done.


is the logit transform ever actually computed in modeling process of logistic regression?

Basically the Logit function is a series of transformations from ordinary regression. Transformations from ordinary regression are often called "link functions" It goes:

Ordinary Least Squares (OLS) -> Logistic Function -> Logit Function

You probably already know these equations but I am going to write them out anyway:

OLS gives you a single predicted value

OLS = β_1 X_1+β_2 X_2+⋯β_i X_i

The Logistic Function gives you the probability that Y = 1

Logistic Function = 1/(1+exp[-1(OLS)])

The Logit Function gives you the log odds that Y = 1.

Logit Function = log_e(Logistic Function / (1-Logistic Function))

You are asking whether or not the logit function is ever actually used and the answer is yes, but you probably don't see it because a model summary will display the coefficients (betas) but not the equation that they're in. Your preferred language will certainly use these equations when estimating your coefficients, but there is no reason to show them to the user because the user is the one who specified the link function in the first place.

glm(y ~ x1 + x2, data = my.df, family = binomial(link = "logit"))


First order vs Second order online machine learning algorithms

First order and second order algorithms are algorithms which use respectively at least first derivative / second derivative.

This post explains well the definition.


How to cluster label (in a multilabel classification problem) which mostly appear together in a class

The difficulty here is the design: if one just represents every "class" as an instance with a vector of its co-occurrences with other "classes", then the clustering will only group the full group of instances/classes by how similar they are with other groups.
Instead I think that an instance should represent a pair of "classes", using the PMI as a measure of how strongly they are associated.
An advanced option would be to represent each "class" as a node in a graph and add an edge between any two classes which occur together, weighted by their PMI value. Then use a graph-clustering algorithm to group the "classes".


What are some ways to transition from additive to multiplicative(or vice-versa) changes in time-series quickly?

The problem with your request is that both ways are always possible. For any pair of values x and new_x you can choose to represent the increase either as an addition or a multiplication:

additive increment: new_x -x
multiplicative coefficient: new_x / x

So you need a criterion to decide which one to choose, for instance a threshold: if x is lower than 100, use addition otherwise use multiplication.
Additionally if any kind of change can happen at anytime it's completely arbitrary, there's no way to represent this in a systematic way.


How to normalize technical skills of IT using machine learning?

In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a few API.
Regex, Levenshtein distance, and other old school metrics are great, but in general you are never certain as to what you want to consider "as similar" besides the fact they words have a few common concepts (such as agile methodologies vs scrum vs kanban).
However, if you have a good pretrained encoder, this similarities will be captured.
In the second and next iteration, you can now use the encoder to check if similarities between skills are flagged properly, using the for example cosine distance. You can then automatically label the similarities and eventually correct the labels. Once you have sufficient label, you can now retrain the encoders on your own (smaller) dataset.


It cannot persist the structure/organisation of the document.

To tackle this, you can look at pdftotext (which comes by default in case you are on a linux system). It preserves the exact layout of the pdf page. You will have to specify the layout option.

For now it is ok to not read images and table although if there is a library to do that would be great.

Table extraction is a different(and more complicated) problem. However if the tables are not that complicated, you can use Camelot/Tabula to extract those. I would suggest you to go with Camelot as it also provides a lot of tunable parameters that you change for your case specific tables. (More details :Camelot  )


Grouping similar rows to detect duplicates

I did this when i had duplicate values and multiple other variable which are unique.

adding unique variable to remove duplicates

"i have added variables as per above data please change as per your requirement"

Df <- DF%>%
  mutate(Path = paste(id,property_type,NEW_BUILD,SURFACE, sep = ">"))

retaining Unique values in the data

Df <-
  distinct(DF,Path, .keep_all= TRUE)


CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0

CUDNN and Tensorflow require a GPU which has a compute capacity of 3.0 at least: not only the CUDA version must take account of this CC, but also these both programs.

Indeed: https://stackoverflow.com/questions/38542763/how-can-i-make-tensorflow-run-on-a-gpu-with-capability-2-0/38543201#38543201


Fully-Connected DNN: Compute the numbers of free parameter in a DNN

Weights at first layer will depend on the Dimension of the input too.
Let's assume, This is your Network. Input has 5 Features

$\hspace{6cm}$ Image credit - ML Visuals by dair.ai
Per layer weights and biases
\begin{array} {|r|r|}
\hline
 &Weights &Biases\\
\hline
Layer-1 &5*3 = 15 &3\\
\hline
Layer-2 &3*3 = 9 &3\\
\hline
Layer-3 &3*4 = 12 &4\\
\hline
Layer-4 &4*2 = 8 &2\\
\hline
\end{array}
Check with Code
import tensorflow as tf
from tensorflow import keras
model = keras.Sequential()

model.add(keras.layers.Dense( 3, activation="linear",input_shape=(5,)))
model.add(keras.layers.Dense( 3, activation="linear"))
model.add(keras.layers.Dense( 4, activation="linear"))
model.add(keras.layers.Dense( 2, activation="linear"))

model.summary()
#model.get_weights() #Can check weights matrix with this

Output


Document classification - optimal classifier & embedding

I would suggest to have a look at fasttext, it will learn word embeddings for you from your corpus (advantage over word2vec embedding is that it learns representations based on character ngrams, so if a word is there, but its plural is not, it will still have a similar representation) and then in the supervised learning mode it uses something very simple like regression for classification, provided you have examples of labelled data.
I tried it with trivial 20newsgroups and it works well for distinct categories. Maybe not a production-friendly solution (it's in c++, but can be stretched to python, albeit not without pain), but can give an idea.
Given the nature of the algorithm, it's quick, compared to nets.


How many epochs to run during hyperparameter search?

The general rule of thumb is to run the number of epochs until validation error starts to increase.
Sometimes fast initial learning will not lead to the best performance later.


Percentage of smaller dataset with respect to bigger dataset

Naive approach: define a similarity or distance function, say for instance cosine similarity.

Calculate the similarity score between any pair $(x_i\in A, y_j\in B)$
Define a precision level, say $\epsilon=0.000001$. The assumption is that it's extremely unlikely that two vectors would be this close by chance in $A$.
For every $y_j\in B$, find the set $c(y_j) = \{ x_i\in A\ |\ sim(x_i,y_j)\geq 1-\epsilon \}$
Obtain the union: $C(B)=\{x_i\in A\ |\ \exists y_j\in B: x_i\in c(y_j) \}$

The proportion of elements of $A$ which are "equal" to an element in $B$ is:
$$\frac{|C(B)|}{|A|}$$


I'm surprised you got 80% accuracy and think you will have to do some labeling if you want to improve. Even people have problems with this! For example:

Sales Executive - executive can be misleading, this might have been called "Salesman" 50 years ago
VP at a technology company vs VP at a bank. In the latter case, VP is more of an honorific and might not include any actual management.

If you don't want to mass label data, I would try to assign weights to common words that are featured in titles. For example, replace senior/junior with +1 and -1 in a "seniority" value. If you do that for the top 100 words, I bet it will help over a totally unsupervised approach.


model predicting probability close to 50 for positive classes in imbalanced training dataset

More than the probability, what you can consider is the confusion matrix you get after applying the threshold on the probability. If, for example, your threshold is 0.5 and your model is perfectly classifying your validation examples, you shouldn't have a problem.

If you take average log loss as your training metric, with equal weights for positive and negative classes, your average probability values will get shifted towards the dominant side in the training set.

Summarizing, probability values in themselves may not be a cause of worry. Focus more on imbalanced data specific metrics like precision, recall(sensitivity) specificity, F1 score or Area Under the ROC curve.


What is the use of fit method in sklearn.preprocessing.Normalizer()?

In scikit-learn, many preprocessing operations follow the Estimator API (i.e. having fit and transform methods).  The benefit of conforming to the Estimator API is that the object can be included in a data transformation pipeline.  Some of the benefits of pipelines are described in the docs:

  Pipeline can be used to chain multiple estimators into one. This is useful as there is often a fixed sequence of steps in processing the data, for example feature selection, normalization and classification. Pipeline serves multiple purposes here:

  Convenience and encapsulation:
  You only have to call fit and predict once on your data to fit a whole sequence of estimators.

  Joint parameter selection:
  You can grid search over parameters of all estimators in the pipeline at once.

  Safety:
   Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.

Because the Normalizer estimator is stateless, its fit method is a no-op.  But if it was missing the fit method, then it could not be used in scikit-learn Pipelines.


How do I perform K-Means clustering of the Olivetti Dataset

I have a couple of observations about this:

Before you run clustering, I'd run some dimensionality reduction model. Pixel data are too noisy to be fed as they are into a model. With dimensionality reduction you get both noise reduction and lower computational costs.
Images usually come as 2D or 3D arrays (depending whether they are in grayscale or coloured). On the other side, k-Means and other sklearn models require your whole dataset as a 2D array. You should process each image and turn it into a 1D array that can be fed as a single observation.

Could you share your clustering code, just to make sure?


Calculating Confidence Interval at a certain confidence level

The CI is defined as the interval which contains your mean with a $\alpha$% of probability.

Given that you are using a model which its subjacent assumption is normality (amongst others), the interval is to be obtained comparing the probability in the context of a normal distribution.

The function stats.norm.cdf returns the probability of $loc - \frac{mean}{sd}$ being zero.

The value returned is the $(\bar{X} - Z_{\alpha}\sigma,\bar{X} + Z_{\alpha}\sigma)$.

We can't return $(-2 sd+mean,2 sd+mean)$ because it would not be general in terms of probability, we use tables which say $0.95 \rightarrow 2$ because we (humans) cannot calculate the number for every probability, but the stats.norm.cdf function can.

$Z_{\alpha}$ is difficult for us to have all the possible values (what if we need 90%, 95%, 99%, 99.9% or 70% probability?)


removing special character from CSV file

from pandas import read_csv, concat
from ast import literal_eval
df = read_csv('file.csv',header=None,names=['name','value'])
split = df.value.apply(literal_eval).apply(Series).set_index(df.name)
part1 = split.ix[:,:2]
part2 = split.ix[:,3:5]
part3 = split.ix[:,6:]
part2.columns=part3.columns=range(3)
stacked = concat([part1,part2,part3])

Note that this yields a different order than what you requested:

aad    1  4  77
bchfg  4  1   7
cad    1  2   7
mcfg   0  1   0
aad    4  0   0
bchfg  8  0   0
cad    6  0   0
mcfg   0  0   5
aad    0  0   3
bchfg  0  1   0
cad    0  0   3
mcfg   0  1   1


Using training data generated with pure regular expressions - Can machine learning surpass the accuracy of your regular expression?

No- not if your regex is actually generating the data. If this is the case, the regex (or a complementary one) should be able to classify the resulting text perfectly. That is, you know what the rules for creating a 'happy' or 'sad' piece of text are within the regex, and so you simply apply those rules backwards to classify it.

So, if you take what the regex says when it creates the data as ground truth, then there's no reason to create a machine learning classifier to classify it- the creation mechanism itself already does so.

On the other hand, if you have created a regex that will produce "happy" or "sad" tweets, and you want to know if the regex itself or a machine learning algorithm will better conform to the opinions of, say, human observers, you would have a very interesting question to evaluate and a relatively straightforward test case for sentiment analysis.


How is the salary for Data Scientist different among countries?

The biggest one I know is O'Reilly's survey (here the last available from 2016). There are also polls on KDNuggets (older version) and other websites. But, in general, most of them are not very representative.

You can also go look at the national averages on glassdoor.com (example India) or other job sites.


How to do Machine Learning the right way?

There is not really a "right way" to use machine learning outside of running the algorithms, any more than there is a "right way" to use a sorting algorithm, or any other kind of complex automation.

Some guidelines/thoughts:

You can start using your model to make predictions, once those predictions are useful to you. This may be once they pass some threshold in performance that would make the predictions the best way of achieving some other goal. That may just mean "better than the last ML model", but could also mean "cheaper and faster than having some people doing it, even if it is less accurate".
Depending on how your training data is collected, and how the model works, it may be possible to both use your model for predictions, and to continue training it on new data as it arrives.
However, there is no requirement for continuous learning. Learning systems that continuously adapt to new data are useful in some situations (example might be recommendation systems), but might be inappropriate in others (example might be automatic driver for a vehicle, where you don't want to risk reduction in performance whilst in use).

One thing that is considered very standard practice is to demonstrate how the machine learning system is performing after a period of learning by using a test metric. The simplest variation is to keep some of your precious data to one side, deliberately not training on it, then measure the performance. Often two such sets are kept aside - a cross validation set is used to help decide automatically between variations of your learning algorithm, and a test set is used to assess the end result of that selection.


How can I fix this "convex" problem ? Is it just a matter of overfitting?

Yes, what you are seeing is a classic case of overfitting.

You stated that you use a linear model such as logistic regression. To regularize these types of models, usually L1 and/or L2 regularization is applied. L1 regularization is simply $||W||_1$ and L2 is $||W||_2^2$ usually.

Another method is to alter the labels of the model in a specific way, which is a method of regularization I created (shameless plug). Here is the link to the paper: https://arxiv.org/abs/1609.06693

Hope this helps.


Is there a person class in ImageNet? Are there any classes related to humans?

You can also take a look at here for the labels in the imagenet. I guess you are right, there is no label for human in the data-set but there is something to notice. There are labels in imagenet like cowboy or some specific hats and other related things to human like shirt and t-shirt. You can take a look at here and also here. In the latter link Yosinski et al, has tried to show that the popular AlexNet has learned to recognize human faces although there is no label as human face in the imagenet data-set. In their paper, they have investigated that Convolutional neural networks may try to learn things that are distributed among layers or maybe not and they may not have special label in the training data. As an example, the face of cats and humans can be referred to. Moreover, as you can see here maybe the aim was attribute learning in large-scale datasets, as quoted in the last line of the page, as the reference.


Can preprocessing the whole population cause data leakage?

If you have the entire population, there is no need for inference. Thus data leakage is not an issue. You can fit any transformation on the data without a concern for its effect on prediction because there is no prediction step.


CNN for image classification with two outputs

The view classification (front, back, ...) is the easier part as you have the correct labels in your dataset.
I would do it using transfer learning : pick a pre existing image classification model (such as VGG or Res-Net) and freeze it (parameters.require_grad = false in pytorch layer.trainable = false in keras), remove the last classification layers and replace them with your architecture with the correct number of outputs (6 classes in your case). Then train the network : it should only train the last classification part as we froze the convolutional part. And it should give good results depending on how complicated the initial CNN model is.
Transfer learning is only useful if you do not have much data in your dataset, i'd say it is not necessary to use a pre existing model if you have a lot of data (>100 000) in the dataset. If you have a big dataset, you can create your architecture from scratch and build your convolutional layers with an encoder-classifier architecture as represented below (blue + red are the convolutional layers and green is the classifier part) :

I'm not sure to understand what you mean by type of cars part(door, window ...), it seems to me you want to have another classifier that gives the car parts on the image, so the output should be something like : there is window, a door, but no trunk. This is also possible, but requires a dataset labeled with such information (which does not seem to be the case). Maybe you can find correlations between the view and the parts on the image and deduce it from there. Or if you have such a labeled dataset, just create another CNN and train it on this dataset (output may need to be slighty different if you want to predict different classes for one image).
Anyway, CNN is definitly the way to go for your problem as it is the most accurate models we have at the moment for image classification, and they are likely to perform way better than fully connected layers or other architectures.


That depends on the latent representation the model has learned. Typically it won't be rotational invariant because there are no inbuilt mechanism in vanilla CNN or constraints enforcing that!

If you want to learn rotational invariant representation then either you can use a Rotation Invariant CNN or you may set up the training objective of the Siamese network such that when you pass random rotated versions of the same image of the same input, you classify as positive or else negative. The 2nd method is enforcing the constraint that your representation must be rotational invariant.


How can I calculate a rolling window sum in pandas across this MultiIndex dataframe?

If anyone else comes looking, this was my solution:

# find last column
last_column = df.shape[1]-1

# grab the previous 11 columns (also works if there aren't that many in the df)
wanted = df.iloc[:, (last_column-11):last_column]

# calculate the rounded moving annual total
mat_calc = round(wanted.sum(axis=1)/len(wanted.columns), 2)

Probably not the most pandastic solution, but it works well.


I don't completely understand your example, but:

if you want to put your model into operation/production environment at some day ... then you should not allow 'looking into the future' at training time.


Clusering based on categorical variables?

I assume your data set is sth like

A0 A1 A2 B0 B1 B2
1  0  0  1  1  0
0  1  0  0  0  1
0  1  0  0  1  1
      ...

where A0 refer to the answer of question A, choice 0 and question A is a single choice question,
B is a question allowing multiple answers.
Each row represents one record.

For this data format, each column can be considered as a dimension (if some columns can be quantified to a value, you may group them into one column, ex "Like = 1, Neutral = 0, Dislike = -1") Then applying clustering algorithms, such as K-mean to cluster it.

Please tell if your data set format is different from my assumption.

Updated on Jun 30:

The idea is to quantify the variables,
For column 2-5, the values are already quantified to [1, 5]
For column 5, it allows ("poor"/ "nice"). Although they are strings, they are representing the preference of different magnitude. They can also be converted to "poor" = 0, "nice" = 1.
For column 7-9, because it is a binary option, they can also be easily converted to ex. "true" = 0, "" = 1.

Then, you will have a matrix:

1    4    5    4    1    1    1    1
2    4    3    1    0    1    0    1
1    5    2    1    1    0    1    1
3    2    1    5    1    1    0    1

Remark1: For binary options, such as column 9, only allows "False" or "null", choosing "False" = 1 or 0 does not matter. In classification, only the distance matters.

Remark2: If you have a column allows different strings, such as "apple", "banana", "orange", you can convert it to an one hot vector.
Ex.

A
apple
orange
orange
banana

can be converted to

A0   A1   A2
1    0    0
0    0    1
0    0    1
0    1    0

where A0, A1, A2 represent "apple", "banana" and "orange".

Remark3: if there is a column allow storing multi answers, ex.

A
apple
apple orange
orange
orange banana

can be converted to
    A0   A1   A2
    1    0    0
    1    0    1
    0    0    1
    0    1    1

Remark4: before applying some classification algorithms such as K-mean, it is better to normalize the magnitude of each dimension. For example, value of column 9 is [0, 1] can be normalized by times 5 to [0, 5]. The factor/ the range of the dimension reflects the importance of that factor.

I hope my answer can help.


Difference between tf.keras.backend.one_hot and keras.utils.to_categorical

The difference is the input and output.
The utils.to_categorical function takes a vector as input and returns a matrix with one hot encoded rows.
The  backend.one_hot can be called as a transformation step inbetween a compute sequence of tensors.
It takes nD tensor as input and returns n+1 D one hot encoded tensor.


If I understand your question properly. In first example you have 4 input neurons and they are connected to 4 neurons. In second, 4 inputs are connected to 35 neurons. That's it, you simply add more neurons in hidden layer. Btw, what do you mean saying "learns it with an accuracy"?


Evaluating model with categorical target variables

You could convert your categorical outputs back into their numeric equivalents, and then use the confusion matrix and normal


Cost function for support vector regression

Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left< w, x \right> + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.


Multicolinearity & accurate weights of predictors

If you build separate models, you are making the internal predictors/features independent of each other. This will cause a lot of your internals predictors to get really high weights which would probably not be the case if you added them together in the same model. An obvious method is to remove correlated predictors and then see what weights you get.


Research in random forest algorithms able to switch data sets

Semi-Supervised Learning

The combination of unsupervised learning and supervised learning is referred to as semi-supervised learning, which is the concept that I believe you are searching for.

Label propagation is often cited when outlining the heuristics of semi-supervised learning.  The essence is to employ clustering, but to use a tiny set of known cases in order to derive (or propogate) the labels of the clusters.  Hence one is able to use a small set of labeled cases to classify a much larger set of unsupervised data.

Here are some references:

Wikipedia has an entry on the semi-supervised learning.
The scikit learn User Guide is often a useful starting point and has a label propogation routine.
There are, in fact, papers treating semi-supervised random forest models.
Another one here

Hope this helps!


How to rename columns that have the same name?

You can use this:

df.columns = ['Goods_1', 'Durable goods','Services','Exports', 'Goods_2', 'Services', 'Imports', 'Goods_3', 'Services']

or if you have too many columns:

cols = []
count = 1
for column in df.columns:
    if column == 'Goods':
        cols.append(f'Goods_{count}')
        count+=1
        continue
    cols.append(column)
df.columns = cols


What's the best way to approach a regression problem with a lot of features (over 300), many of which are categorical?

Try doing a Principal component analysis to get the reduced feature set and then apply regression techniques in loop to see which one predicts better. SVM may take sometime to execute so better exclude it. Random Forest should be a good fit in this case.


Looking for animals dataset for Deep Learning classification

Unfortunately, a single dataset with all animals does not seem to exist (perhaps you can make one :D ), but there are plenty of datasets with a subset of animal species. Here are a few I can think of:

There are many datasets on Kaggle. Searching for "species", "animal", or some other smarter keyword should give some options. I found the 10 Monkey Species, STL-10, bird species classification, and Animals-10 datasets fairly quickly, for example.
Most large-scale datasets like OpenImages, CIFAR, ImageNet, the Visual Genome, and COCO have animals as some of the categories (among non-animal ones).
The iNaturalist dataset is a large scale species classification dataset (see the 2018 and 2019 competitions as well).
The Nature Conservancy Fisheries Monitoring dataset focuses on fish identification.
The Caltech-UCSD Birds-200-2011 is a standard dataset of birds.
The animals with attributes 2 dataset focuses on zero-shot learning (also here).
The Snapshot Serengeti dataset covers 40 mammals from the African Savannah.
The Oxford-IIIT Pet Dataset is good for fine-grained cat and dog classification.
A survey on image-based insect classification by Martineau et al has references to a number of image datasets for insect identification (though I am not sure how many are public).
The group at the Labeled Information Library of Alexandria: Biology and Conservation (LILA BC) track a large set of animal conservation and ecology datasets, including many involving image-based species recognition here.


Difference between coefficient of determination and least squared error?

R squared is one of many model diagnostics. One calculation of it is (1-var(residuals)/var(response variable)). Since the response variable is constant, you want to minimize the average squared residuals (the variance), which is identical to least squares.

The issue with using only R-squared is that if you have a lot of variables in a model, you will likely get a high R-squared, but that doesn't mean the model is good.  In fact, if you have 100 rows of data, 100 independent variables and 1 response variable, there's a good chance you will get an R-squared of 1, but the model will be completely overfit.

There are 2 common solutions to this problem:

Add a penalty term that reduces (or increases, depending on what is optimal) the "score" of a model when it has more variables.
Use a form of validation to see if the model performs well on data it wasn't trained on.

Oftentimes both of these are used to varying degrees, but here are a few examples:

AIC, or Aikaike Information Criterion, is -2*log(P(Y|X)) + 2*p, where P(Y|X) is the probability density of Y given X under the current model and p is the number of predictor (X) variables.  The goal is to minimize AIC.
In K-fold cross-validation, where K is the number of subsets you divide data into, you train the data on K-1 of the sets combined and then test it on the remaining set and report the error.  You repeat this process for each combination of K-1 training sets and 1 remaining set and combine the K results (usually by averaging error).

The first method is usually a safe bet for linear regression models, but oftentimes, especially when you are using more complex algorithms, you will want to validate the algorithm to make sure it isn't overfitting the training data.  Also, using validation allows comparisons between different models (especially non-parametric ones) to be made much more easily.

For more ways of evaluating models, I would suggest reading into different regularization techniques and different validation techniques.


TypeError: object of type 'Bidirectional' has no len()

That code instantiates model over and over. model should be instantiated once and all other layers are added to that instance. Something like this:

from tensorflow.keras.models import Sequential

model = Sequential()
model.add(Embedding(input_dim=n_words, output_dim=MAXLEN, input_length=MAXLEN))
model.add(Dropout(0.2))
model.add(Bidirectional(LSTM(units=LSTM_N, return_sequences=True, recurrent_dropout=0.1)))
model.add(TimeDistributed(Dense(n_tags, activation="softmax")))


Looking for a classification (?) algorithm for linearly separable but unlabeled data points

I got a partiall solution which migth work for this particular case but it is not easely generalizable: Since all your data converges to the same point: (1400, 500), you can set the center of coordinates there and then cluster in the space of angles. i.e. use the transformation $$\alpha = \arctan\left(\dfrac{y - y_0}{x - x_0}\right)$$ and cluster $\alpha$ (be carefull! you might divide by 0)

I have prepare this synthetic example in R and it works!,  but again: this is not easely generalizable

library(data.table)
library(ggplot2)

x <- c(1,2,3,4,5)
y1 <- x + 1
y2 <- -x + 11
y3 <- -3*x + 21

# creates a data.table with 3 lines converging at point (5,6)
dt <- data.table( x = rep(x, 3)
                , y = c(y1,y2,y3)
                )

The data.table looks like this:

Now is time to transform it and make the clusters

dt_transform <- dt[, .(x, y, a = atan((y - 6) / (x - 6)) )] # avoid x - 5 == 0

km <- kmeans(dt_transform$a, 3)

dt_cluster <- cbind(dt_transform, cluster = as.factor(km$cluster))

The clustered data.table looks like this:

If you go this way please, share the results with me. I'm interesting to know if it works with real data


Why is xgboost so much faster than sklearn GradientBoostingClassifier?

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate.

My guess is that the biggest effect comes from the fact that XGBoost uses an approximation on the split points. If you have a continuous feature with 10000 possible splits, XGBoost consider only "the best" 300 splits by default (this is a simplification). This behavior is controlled by the sketch_eps parameter, and you can read more about it in the doc. You can try lowering it and check the difference it makes. Since there is no mention of it in the scikit-learn documentation, I guess it is not available. You can learn what XGBoost method is in the their paper (arxiv).

XGBoost also uses an approximation on the evaluation of such split points. I do not know by which criterion scikit learn is evaluating the splits, but it could explain the rest of the time difference.

Adressing Comments

Regarding the evaluation of split points

  However, what did you mean by "XGBoost also uses an approximation on the evaluation of such split points"? as far as I understand, for the evaluation they are using the exact reduction in the optimal objective function, as it appears in eq (7) in the paper.

In order to evaluate the split point, you would have to compute $L(y,H_{i-1}+h_i)$ where $L$ is the cost function, $y$ the target, $H_{i-1}$ the model built until now, and $h_i$ the current addition. Notice that this is not what XGBoost is doing; they are simplifying the cost function $L$ by a Taylor Expansion, which leads to a very simple function to compute. They have to compute the Gradient and the Hessian of $L$ with respect to $H_{i-1}$, and they can reuse those number for all potential splits at stage $i$, making the overral computation fast. You can check Loss function Approximation With Taylor Expansion (CrossValidated Q/A) for more details, or the derivation in their paper.

The point is that they have found a way to approximate $L(y,H_{i-1} + h_i)$ efficiently. If you were to evaluate $L$ fully, without insider knowledge allowing optimisation or avoidance or redundant computation, it would take more time per split. It this regard, it is an approximation. However, other gradient boosting implementations also use a proxy cost functions to evaluate the splits, and I do not know whether XGBoost approximation is quicker in this regards than the others.


Identifying outliers in an unknown distribution

If the high volume and low volume data both appear in non-negligible probability, you may view the problem as a clustering problem with two clusters. Gaussian mixture model or K-means algorithm may help you in this case.


Owen Zhang's slides: what does the "time" mean?

Here's my understanding:

I think it's the explanation of page 10, and time is the same as the "time" in page 10, which means the time axis in real life.

In page 10, it shows that:

hold out data should be out-of-time, we can not do a random shuffle because it will not represent the real situation our model will be faced.
unless data is extremely small

As the slide has said before:

  When you looked at your validation result for the Nth time, you
  are training models on it

In this case: validation data = hold out data = Public LB (if my understanding is correct)

The first figure of page 12 is the normal condition, we shouldn't check our Public LB to tune our model(or we're training model on it). So the answer is no.

The second figure is the small data condition, we have to train model using the feedback from Public LB, because of the data limitaion. So the answer is yes.

The third figure shows that Public Data is the same as Private Data, they both represent something under same time period, sharing the same "reality", thus we can use the feedback from Public LB.

The forth figure shows that Public Data is reality, and Private Data is more "realistic" than Public Data. It's the "hold out data should be out-of-time" case, the answer is yes.


Optimization problem: Given Beta Bounds Maximize sharpe

Well, you didn't give a whole lot of information, so I'm going to make some assumptions here, but I think this is pretty darn close to what you want.  Keep in mind, Beta = (Covar/Var), so you need to calculate that with what's given below, and then you will have everything you need.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.optimize as sco
import datetime as dt
import math
from datetime import datetime, timedelta
from pandas_datareader import data as wb
from sklearn.cluster import KMeans
np.random.seed(777)

start = '2019-02-28'
end = '2020-02-28'
# N = 90
# start = datetime.now() - timedelta(days=N)
# end = dt.datetime.today()

tickers = ['MMM',
'ABT',
'ABBV',
'ABMD',
'ACN',
'ATVI',
'ADBE',
'AMD',
'AAP',
'AES',
'AMG',
'XEL',
'XRX',
'XLNX',
'XYL',
'YUM',
'ZBH',
'ZION',
'ZTS']

thelen = len(tickers)

price_data = []
for ticker in tickers:
    prices = wb.DataReader(ticker, start = start, end = end, data_source='yahoo')[['Adj Close']]
    price_data.append(prices.assign(ticker=ticker)[['ticker', 'Adj Close']])

df = pd.concat(price_data)
df.dtypes
df.head()
df.shape

pd.set_option('display.max_columns', 500)

df = df.reset_index()
df = df.set_index('Date')
table = df.pivot(columns='ticker')
# By specifying col[1] in below list comprehension
# You can select the stock names under multi-level column
table.columns = [col[1] for col in table.columns]
table.head()

###################################################

def portfolio_annualised_performance(weights, mean_returns, cov_matrix):
    returns = np.sum(mean_returns*weights ) *252
    std = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights))) * np.sqrt(252)
    return std, returns

def random_portfolios(num_portfolios, mean_returns, cov_matrix, risk_free_rate):
    results = np.zeros((3,num_portfolios))
    weights_record = []
    for i in range(num_portfolios):
        weights = np.random.random(thelen)
        weights /= np.sum(weights)
        weights_record.append(weights)
        portfolio_std_dev, portfolio_return = portfolio_annualised_performance(weights, mean_returns, cov_matrix)
        results[0,i] = portfolio_std_dev
        results[1,i] = portfolio_return
        results[2,i] = (portfolio_return - risk_free_rate) / portfolio_std_dev
    return results, weights_record

returns = table.pct_change()
mean_returns = returns.mean()
cov_matrix = returns.cov()
num_portfolios = 10000
risk_free_rate = 0.0178

###################################################

def display_simulated_ef_with_random(mean_returns, cov_matrix, num_portfolios, risk_free_rate):
    results, weights = random_portfolios(num_portfolios,mean_returns, cov_matrix, risk_free_rate)

    max_sharpe_idx = np.argmax(results[2])
    sdp, rp = results[0,max_sharpe_idx], results[1,max_sharpe_idx]
    max_sharpe_allocation = pd.DataFrame(weights[max_sharpe_idx],index=table.columns,columns=['allocation'])
    max_sharpe_allocation.allocation = [round(i*100,2)for i in max_sharpe_allocation.allocation]
    max_sharpe_allocation = max_sharpe_allocation.T

    min_vol_idx = np.argmin(results[0])
    sdp_min, rp_min = results[0,min_vol_idx], results[1,min_vol_idx]
    min_vol_allocation = pd.DataFrame(weights[min_vol_idx],index=table.columns,columns=['allocation'])
    min_vol_allocation.allocation = [round(i*100,2)for i in min_vol_allocation.allocation]
    min_vol_allocation = min_vol_allocation.T

    print("-")
    print("Maximum Sharpe Ratio Portfolio Allocation\n")
    print("Annualised Return:", round(rp,2))
    print("Annualised Volatility:", round(sdp,2))
    print("\n")
    print(max_sharpe_allocation)
    print("-")
    print("Minimum Volatility Portfolio Allocation\n")
    print("Annualised Return:", round(rp_min,2))
    print("Annualised Volatility:", round(sdp_min,2))
    print("\n")
    print(min_vol_allocation)

    plt.figure(figsize=(10, 7))
    plt.scatter(results[0,:],results[1,:],c=results[2,:],cmap='YlGnBu', marker='o', s=10, alpha=0.3)
    plt.colorbar()
    plt.scatter(sdp,rp,marker='*',color='r',s=500, label='Maximum Sharpe ratio')
    plt.scatter(sdp_min,rp_min,marker='*',color='g',s=500, label='Minimum volatility')
    plt.title('Simulated Portfolio Optimization based on Efficient Frontier')
    plt.xlabel('annualised volatility')
    plt.ylabel('annualised returns')
    plt.legend(labelspacing=0.8)

display_simulated_ef_with_random(mean_returns, cov_matrix, num_portfolios, risk_free_rate)

Result:

Maximum Sharpe Ratio Portfolio Allocation

Annualised Return: 0.2
Annualised Volatility: 0.16

             AAP  ABBV  ABMD   ABT   ACN  ADBE   AES    AMD   AMG   ATVI  \
allocation  3.29  5.51  0.59  4.29  2.89  9.09  8.42  11.22  3.01  10.03

             MMM    XEL  XLNX  XRX   XYL   YUM   ZBH  ZION   ZTS
allocation  0.12  11.39  2.95  6.7  2.89  5.91  3.02  0.79  7.88
-
Minimum Volatility Portfolio Allocation

Annualised Return: 0.05
Annualised Volatility: 0.13

             AAP  ABBV  ABMD   ABT   ACN  ADBE   AES   AMD   AMG  ATVI   MMM  \
allocation  6.35  8.14  2.55  4.32  8.71  2.18  5.62  1.99  2.08  4.86  6.89

             XEL  XLNX  XRX  XYL   YUM   ZBH  ZION   ZTS
allocation  10.1  1.93  1.9  5.5  7.92  5.57  6.01  7.39


Before anything, draw a graph.

You'll notice that you have at least three process: one around 20-25, one around 60-90, and one around 250-450. A closer look tells you that the first one is on Saturdays, the second on Sundays and the last one for other working days.

Further notice that the Sundays series is different since 2016-12-26. And have a close look to Christmas and Banking holidays to decide if they behave like Saturdays, Sundays or ordinary days.

Also notice a set of outliers between 2017-12-21 and 2018-01-02. Remove them before to fit. You also have overproduction by the end of November 2016. Remove them also.

Then make a 7-day moving average, which as you will see is rather linear, except for an underproduction in January 2017, for which you may have an external explanation (following the change of end 2017).

With this linear fit, you may have a look to a weekly seasonality. And reach a 70-80% accuracy.

But forget a 90% accuracy. There is much more fluctuation in your data.


How data are prepared during training, testing and in production?

The principle in supervised ML is quite simple: the "method" which is going to be used to predict the response variable must be fully determined from the training set and only from the training set. In other words, anything which doesn't belong to the training set cannot be used.
As a consequence, feature engineering, i.e. choosing how to prepare/represent/normalize features must be done using only the training set. This includes any feature selection/extraction step.
Note that once the final data preparation process is fully determined, it can and should be applied exactly the same way on the test set or in production. This means that for instance normalization does not involve recomputing any parameter, it uses the ones calculated on the training set.
See also a few related questions:

Data normalization before or after train-test split?
Why you shouldn't upsample before cross validation
Using PCA as features for production


What is wrong with my logistic regression implementation?

What is the ratio between 0's and the other numbers? If the ratio is too low, let's say 2%, if you classify everything as not 0 you will have high accuracy.

EDIT AFTER COMMENTS

Imagine there are 98 non zeroes and 2 zeroes, you don't need any classifier to have a 98% of accuracy...

Accuracy is not the best metric for unbalanced data-sets, let's try precision and recall and take a look to the confusion matrix


Predicting labels which are not independent of one another

In this case the easiest way is to use a simple single-label classifier with three labels:

A0B1
A1B1
A0B0

As long as the number of combinations is reasonable, that's what will give the best results while satisfying the constraints.

By the way "If label B is false, then label A is false" is logically implied by "if label A is true, then label B is also true". (contrapositive).


Any mini-batch implementation of Random Forest?

Check out the CART algorithm. This is essentially a bootstrapping method with subsampling that I am sure you can generalize for batch processing.


How to deal with count data in random forest

Some thoughts:

Your data is highly imbalanced. This is a critical issue which should be dealt with. Possible solutions include simple under-/over-sampling to more complicated synthetic approaches like SMOTE.
Decision trees and random forests do not require feature scaling - this means that normalisation is not needed (unless perhaps you plan on using some other modelling technique which uses regularisation).
Just because this data set only shows a particular relationship between values of V1 and target does not mean it is always the case - especially if your model is to be deployed for a period of time. The relationship may change over time so do not rush to artificially curtail your model.


I am afraid if you had saved the model in your drive rather google collab directory as it will lost after run-time session interrupt.

model_new = load_model('BestModelWeights.hdf5')  # will load model , if BestModelWeights file  exist in collab directory --which i feel lost in your case that could be the reason you are getting "where it saved" error.

I would advise use google drive to save weights and directly load weight file when you need.For this you need to follow two steps

Mount google drive and authenticate collab.

from google.colab import drive
drive.mount('/content/drive')
Load model

model_new = load_model('/content/drive/My Drive/BestModelWeights.hdf5')

and you can play with your new loaded Model.

hope it helps.
JSH


Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF

Do I need to split data to TrainSet and TestSet?

It depends:

It is acceptable to cross-validate on the whole dataset to obtain the performance of a particular model, since in this case the model is always tested on unseen data.
In your case, the selection of the best generalized model/hyperparameters is part of the training stage, so yes it makes sense to cross-validate only on your training set and then measure the performance of your final model on some fresh data. This is because the selection of the best hyperparameters (especially from a large set of possibilities) can still be partly due to chance despite CV (especially if the dataset is small), so testing on unseen data avoids the risk of overestimating the performance.

  Then randomly select hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

Usually the number of folds k for k-fold CV is not really part of the hyper-parameters, so it's a quite unusual to do this (as far as I know). The risk is that your final model might be selected partly because a particular k happens to produce a higher performance on the training set. To see this intuitively: a larger training set is more likely to produce a better model, and a higher k makes CV use a larger training set for every fold. So a higher k is likely to artificially improve the performance, just because the training set is larger. This is why it's safer to use the same k for comparing different hyper-parameters, possibly repeating the whole process for different values of k.  Also for a particular value of k you can randomly reshuffle the partitions to minimize the effect of chance in the CV partitioning.


Match a two items from two different receipts

You can try some approximate string matching which gives a confidence score. For example, you can try out with Levenshtein distance, but adjusted with the length of the strings using a probabilistic model; or, you can try out with Jaccard similarity on 3-grams and special treatment on word boundaries, and then calibrated into probabilities. Now you have an n by n matrix of probabilities.

After this you need to perform a matching. The final likelihood is the product of each single probability. To maximize it, you can try to maximize the sum of logs of individual probabilities. Having taken logarithms of each probability, this now becomes an (additive) assignment problem which has implementations in R or Python.


What is the difference between Perceptron and ADALINE?

The Adaline (Adaptive Linear Element) and the Perceptron are both linear classifiers when considered as individual units. They both take an input, and based on a threshold, output e.g. either a 0 or a 1.

The main difference between the two, is that a Perceptron takes that binary response (like a classification result) and computes an error used to update the weights, whereas an Adaline uses a continous response value to update the weights (so before the binarized output is produced).

The fact that the Adaline does this, allows its updates to be more repesentative of the actual error, before it is thresholded, which in turn allows a model to converge more quickly.

Have a look at this really interesting history of neural networks, which contains a small section on Adalines, along with memistors - resistors with memory, as the neurons where figuratively perceived back in the 1960's.

There are also some other answers to a similar question here.


Adding hour data to Date feature in time series prediction

I am a bit confused by the setup of this question but I'll try my best.

First, if your time series has YYYY-MM-DD, and then you also have an hour column with ranges 0:23, then what you have is a hourly time series. That is, the natural index of the time series is by the hour, not by any other frequency of time, and therefore, you would have no need to include either of these two columns in your "dataset".

If the hour of the day is a strong "predictor", then this is just another way of saying that your time series exhibits hourly seasonality and therefore, including hour of the day may help if you see that the residuals of the time series are not consistent with white noise (an assumption of ARIMA models) and perhaps have a superimposed pattern on the ACF plot. A seasonal ARIMA model might be of interest to you, or perhaps an ARIMAX model with an exogenous hour of the day variable. Furthermore, certain particular days might have noticeable and consistent effects, or even months. There may also be one time effects, etc. which might be useful to include as exogenous variables.


How do I assess whether two time series change together?

One option is a Granger causality test which a statistical hypothesis test for determining whether one time series is useful in forecasting another. One time series is said to Granger-cause another time series if the lagged values of one times series provide statistically significant information about future values of another time series.


Change values of consecutive repeating letters in pandas column

For this particular case in which difference between consecutive block elements is 0 or unitary, you could use diff() and something like this:

import pandas as pd, numpy as np
df = pd.DataFrame({'id': [1, 2, 3, 4, 5, 6], 'block': [1, 1, 2, 2, 3, 4]})
values = ['a', 'b', 'c', 'None']
indexs = np.insert(np.cumsum(np.diff(df.block)), 0, 0)
df['value'] = [values[np.asscalar(x)] for x in indexs]

When value is already part of your dataframe, you could simply replace the values list given above by the following:

df = pd.DataFrame({'id': [1, 2, 3, 4, 5, 6], 'value': ['a', 'a', 'a', 'a', 'b', 'c'], 'block': [1, 1, 2, 2, 3, 4]})
values = np.append(df['value'].unique(), 'None')
indexs = np.insert(np.cumsum(np.diff(df.block)), 0, 0) # unchanged
df['value'] = [values[np.asscalar(x)] for x in indexs] # unchanged

which conducts to the same result.


Data preprocessing for time series prediction

So the question is about how to represent this dataset so you can use this your classification task.
Firstly, we need to group the features for each product, if you are classifying on a product-by-product basis.
Secondly, a couple of things:

For categorical features (e.g. specific products), we normally represent these as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).
For numerical features (e.g. time of stay on the shelf [minutes/hours, etc.]) we can represent these as the value themselves.

Then, for the input into your model, then you simply concatenate these features together to form a "long" n-dimensional vector.


Bidirectional Encoder Representations from Transformers in R

You might be interested in the open-source R package RBERT:
https://github.com/jonathanbratt/RBERT

It's a work in progress, but the goal is to be able to use BERT directly in R.


Machine Learning based Multivariate Time Series Prediction - How to create supervised data format

Regarding how to handle multivariate time series problem, I believe that GitHub link Timeseries multivariate will be helpful.

You have to change n_inputs and n_outputs as 12 and 1 respectively.


What are suitable datasets for univariate time series forecasting with RNNs, LGBM, TBATS, SARIMA models (topic, frequency, sources)?

A dataset I used when checking the ARIMA methodology is the one you can find here about monthly air passengers history data since 1949. It shows both trend and seasonality, and it is nice to play with searching for hyperparameters (p, d, q) when implementing ARIMA.
You can find below the results I got sometime ago as a quick check (being blue the true data and red the predictions):


How to reduce time R takes for model building

Depends on the models you are trying to run. Your data isn't that big. For example using a support vector model from the kernlab package, you run into problems. Not every model is fast or has a fast implementation.

Without more information on what you are doing it is difficult to say what causes the bottleneck. But if you just want a speed boost in running models, have a look at the xgboost package, the h2o package (GLM, GBM, rf, deeplearning), ranger for a faster implementation of a randomforest model.


Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?

Close. Each gate is an activation function (typically sigmoid for actual gates, but tanh is used for other functions within the cell) over a weighted sum of all inputs to the cell's layer. Although spome reading of the diagrams may imply LSTM cells are processing a few single inputs in isolation, recombined later, in fact the inputs to each cell are the whole previous layer and the whole previous timestep cell states for the whole layer. This is similar to a view of a single neuron in a fully-connected feed-forward network.

Each gate in each cell has its own set of weights - one weight for each input from previous layer plus one weight for each cell state in the same layer, plus a single bias. When you combine these cells into a layer of multiple cells (e.g. when you choose to have an LSTM layer with 64 cells), this results in a separate matrix plus bias for each gate. In descriptions of LSTMs, these different matrices are often named for the type of gate they parametrise, so e.g. there will be a matrix plus bias for the layer's "forget" gate, which might be noted as $\mathbf{W}_f$.

If you have $N_i$ inputs, and $N_c$ LSTM cells in a single LSTM layer, then $\mathbf{W}_f$ will be a $N_c \times (N_i + N_c)$ matrix of weights, and so will all the other parameter matrices describing other gates and value calculations for the combined cells.

In practice the calculations don't need to be handled per cell, but calculations over a layer of cells can be vectorised, and it is more like having a few parallel fully connected layers that combine in various ways to generate output plus next cell states.

There would be nothing stopping you extending this architecture and making any single gate or value calculation deeper by giving its own hidden layers. I suspect this has been tried by researchers, but cannot find any references. But without this customisation, the gates in standard LSTM are more like logistic regression over concatenated input and cell state.


How to define person's gender from the fullname?

That dataset looks like a good starting point. Keep in mind that when you make your own dataset from those datasets you'll want to keep the male to female ratio balanced if you want it to predict both well.

It should not matter what machine learning software you use (Apache Mahout, scikit-learn, weka, etc.). Pick one that fits your language of choice since speed will probably not be too much of a concern with the smallish dataset size. As for features, you'd generally use ngrams as your baseline for NLP classification tasks. If you use ngrams here you won't end up with anything very interesting because the model won't generalize to any unseen names. I'd suggest as a feature baseline that you try character ngrams, and maybe something like syllable ngrams for something slightly more advanced (for syllable tokenization see https://stackoverflow.com/questions/405161/detecting-syllables-in-a-word).