I'm running the script now.   It's just creating bottleneck files for the flowers images, i.e. the old bottleneck files are not present.

You can see that the bottleneck files are all created in the /tmp directory.

This makes sense as the point of the script is only to retrain the final layer of the network.   The idea is that the previously trained layers will work for an arbitrary set of images.


More weightage to a categorical feature for an Autoencoder model

Do you mean giving more weights to a selected part of the input? If so, then, I believe this is possible with neural networks. Each neuron is assigned a random weights at the beginning of training. You can just initialize the first layer of the neural network model with some values representing the desired weights and then freeze those neurons.

An automated way of doing this is by using attention mechanism which automatically provides more weights (attention) to a portion of the input data.


How do I arrange my data to predict 6 weeks of daily sales

I guess it depends on exactly what you want - do you want a forecast per store? If so you would need to leave the data set aggregated only to a store level. If not, and you want the overall sales by day, then you are OK to leave it aggregated.

Another consideration is whether there are many store-level features e.g. location, weather, local demographics. These features may be useful for the model to learn from and would lose resolution if store level data is aggregated, therefore leaving it un-aggregated would be better. The counter point to this is whether you have enough data for each store to learn representative trends! So it does get kind of circular and depends on the size of your data set.


How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

One normally uses Grid Search for calculating the optimum parameters in these situations:

from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

n = 30 # Max number of neighbours you want to consider
param_grid = {'n_neighbors': np.arange(n)}
grid = GridSearchCV(KNeighborsClassifier(), param_grid)

Then given this grid, you can fit it to your data to compute its optimum values (from those you provided, they may not be global optima (or even local if the returned value is one of the extrema of your input range)):

grid.fit(X_train, y_train)

You can view the optimum parameters from your input by calling:

grid.best_params_
>>> {'n_neighbors': ?}

You can automatically select an estimator with said optimum parameters by calling:

model = grid.best_estimator_
y_pred = model.fit(X_train, y_train).predict(X_test)

Note: you can find the optimum values of other parameters by adding them to the input dictionary param_grid.


Implementing a CNN with one convolution layer

I guess you should change the following line to solve the problem:

model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='valid'))

instead use this code:

model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='same'))

this will keep the dimension of your input. If it does not work, let me know to help you more.


Model with rank as the dependent variable?

Leaving aside any discussion of algorithms and statistics, you cannot predict a rank of an object without context. Are you supposed to rank it relative to your current rows, or in relationship to other new rows? For ranking algorithms based on features, you would usually want to assign an arbitrary score and then sort by it and assign ranks accordingly.


Chinese restaurant process vs Dirichlet Process

A sample from a Dirichlet Process, or DP, is a distribution over a sample space $S$. Here the DP is defined based on a base distribution $H$ over $S$. For instance, in the Wikipedia example from your page, the sample space is all real numbers $\mathbb{R}$ and the base distribution is the standard Normal.

The Chinese restaurant process, or CRP, defines a partition over integers $1,2,...,n$ at each time $n$, and $n$ can go to $\infty$. In this metaphor each block in the partition is called a table. Notice that the CRP itself has nothing to do with the original sample space $S$ or the base distribution $H$.

To associate draws from the base measure H with every table, it means for each table $i$, you independently draw one sample $s_i$ from $S$ according to the base distribution $H$. You repeat the sample $s_i$ for $b_i$ times, where $b_i$ is the number of "customers" seated in the $i$'th table in that Chinese restaurant. The distribution of all those repeated samples for all tables, is your sample distribution from the DP.


Plotting Weibull distribution on Wind Speed

Tried a few things in the OP comments that didn't work.  However, you may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

import datetime

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start the Wolfram session

wolfSession = WolframLanguageSession()

I will collect my own WindSpeedData from a WeatherData weather station closest to the "Country" Entity of Bermuda during October 2016.

weatherStation = wolfSession.evaluate(
    wl.WeatherData(wl.Entity('Country','Bermuda'))
);
print(weatherStation)

Entity['WeatherStation', 'TXKF']

windData = wolfSession.evaluate(
    wl.WindSpeedData(
        weatherStation,
        [
            datetime.datetime(2016,10,1),
            datetime.datetime(2016,10,31)
        ]
    )
)

windData is a TimeSeries object with observation count comparable to your problem.

print(wolfSession.evaluate(windData('PathLength')))

1058

The result can be visualised with DateListPlot by Exporting in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.DateListPlot(windData,
            PlotTheme='Detailed',
            PlotRange=wl.All,
            FrameLabel=wl.Automatic
        )
    )
)

A hurricane passed over in that month with is the cause of that spike in wind speed.

Now that there is data Histogram on the PDF scale gives.

hist=wolfSession.evaluate(
    wl.Histogram(windData, wl.Automatic, 'PDF',
        PlotTheme='Detailed',
        ChartStyle=wl.ColorData('Crayola','Silver'),
        PlotRange=wl.All
    )
);

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        hist
    )
)

Use EstimatedDistribution for the WeibullDistribution.

windDistrbution=wolfSession.evaluate(
    wl.EstimatedDistribution(
        wl.QuantityMagnitude(windData('Values')),
        wl.WeibullDistribution(wl.Global.alpha, wl.Global.beta, wl.Global.mu)
    )
);
print(windDistrbution)

WeibullDistribution[1.883495945177254, 28.34295076324276, -0.7675654467340361]

Or use FindDistribution restricted to the continuous distribution functions to auto find and fit a distribution.

autoDistribution=wolfSession.evaluate(
    wl.FindDistribution(
        wl.QuantityMagnitude(windData('Values')),
        TargetFunctions='Continuous'
    )
);

print(autoDistribution)

ExtremeValueDistribution[18.436141153779957, 10.204118744677338]

Check for goodness of with DistributionFitTest.

print(
    wolfSession.evaluate(
        wl.Map(
            wl.Function(wl.DistributionFitTest(windData,wl.Slot(1),'PValue')),
            [windDistrbution,autoDistribution]
        )
    )
)

[0.0002862317707539308, 0.1802410209978217]

The Weibull fit is not very good and Extreme Value not that strong.  In any case the Plot the PDF functions.

pdfs=wolfSession.evaluate(
    wl.Map(
        wl.Function(wl.PDF(wl.Slot(1), wl.Global.x)),
        [windDistrbution,autoDistribution]
    )
);

pdfPlot=wolfSession.evaluate(
    wl.Plot(
        pdfs,
        [wl.Global.x, 0, wl.QuantityMagnitude(wl.Max(windData))],
        PlotTheme='Detailed',
        PlotLegends=wl.Placed(
            wl.LineLegend(wl.Automatic, [windDistrbution,autoDistribution]),
            wl.Below)
    )
)

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        pdfPlot
    )
)

Combine the plots with Show.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.Show(hist, pdfPlot)
    )
)

Terminate the session

wolfSession.terminate()

Hope this helps.


How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?

I just noticed that none of the answers contained the most important instruction. So here is how it's done since i already resolved this a long time ago.
from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
                ('normalization', MinMaxScaler()),
                ('classifier', RandomForestClassifier())
            ])
pipeline.fit(X)

The pipeline.fit() was missing from other answers which is very important. Once scikit function has learned the parameters of your training data X , you can dump that pipeline into a file and do the rest like this.
joblib.dump(pipeline, 'transform_predict.joblib')
pipeline = load('transform_predict.joblib')
transformed_data = pipeline.transform(new_data)

This will apply the same transformations to a dataset (even if containing 1 sample) that was done on your training dataset.
Update
for newer version, I believe one has to read. I hit the memory limit and the following change helped.
with open(conf['le_embedder_file'], 'rb') as f:
    pipe_le = joblib.load(f)

and write
with open(conf['tecv_embedder_file'], 'wb') as f:
    joblib.dump(pipe_tecv, f, compress='zlib')

I had to read


neural network to find a very simple linear model (scikit-learn)

The problem is relu which is the default activation function. It zeros the inputs smaller than 0. Problem will be solved by using other functions. For example:

C = NN.MLPRegressor(activation="identity", hidden_layer_sizes=(1,1),max_iter=25,random_state=1)

Note that even 1 hidden layer with 1 neuron is enough to find the $X_2 = X_1$ decision boundary.


A possible solution is to calculate the information gain associated to each attribute:
$$I_{E}(f) = - \sum \limits_{i = 1}^m f_ilog_2f_i$$
Initially you have the whole dataset, and compute the information gain of each item. The item with the best information gain is the one you should use to partition the dataset (considering the item's values). Then, perform the same computations for each item (but the ones selected), and always choose the one which best describes/differentiates the entries from your dataset.
There are implementations available for such computations. Decision trees usually base their feature selection on the features with best information gain. You may use the resulting tree structure to find these important items.


Do you have some sort of a dataset where you have labels of images with good visuals and bad visuals?
Without it, the definition of finding bad visuals seems under-defined to me.

Tracing ads/subscriptions could probably be done by using Hough transform (I think that all of them have rectangular bounding boxes).
Overexposed images can found by comparing the image brightness histogram with some threshold value.
"Images which don't contain subject in detail" - you need to use some detection model for this.


How to estimate machine learning training speed on different hardware

Is there any good way to estimate the general training speeds of computer hardware?

Probably not in general, because "training speed" means something completely different if you train  a kNN model (virtually no training) or a complex DL model, what kind of data and what kind of task.

However estimations exist for fixed settings, for instance I found this MLPerf Benhchmark or this paper.

Btw desktop computers are not ideal for very computer-intensive ML, cluster or cloud computing offer better options.


Inferring Relational Hierarchies of Words

Look up taxonomy/ontology construction/induction. Relevant papers:

Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees
Topic Models for Taxonomies
OntoLearn Reloaded. A Graph-Based Algorithm for Taxonomy Induction
Ontology Population and Enrichment: State of the Art
Probabilistic Topic Models for Learning Terminological Ontologies


[After Update1/2]

A learning curve shows how error changes as the training set size increases. One basically change the size of training data points and measure a desired score and compare it against a fixed test set to see how it generalizes. For you the utmost important part to note is the fixed test set. If you were to change the test set, that you could theoretically, how could you evaluate the performance of the same model by changing the size of the training size (because you are changing both datasets at the same time)?!

That is why you pass the the whole X,y in The Sklearn's learning curve, and it returns the tuple containing train_sizes, train_scores, test_scores. As far as I understood, it does a nested validation cross-validation. It basically takes the whole X,y, and split into train/test  (keeping the test data strictly independent and fix in size), depending on how you pass the cv parameter, and keep increasing the training size and record the performance scores for plotting the learning curve. This ensures a proper measure of the optimal model's performance.

Concerning your Update 1: Having just discussed the details, then your Update 1 example scenario would be wrong. We train a model (no matter how), and would like to evaluate its performance on a hold-out dataset. If you change your hold-out dataset, you wouldn't know whether changing the training size led to the change in test score or the variability of the hold-out dataset itself!

Concerning your Update 2: This is exactly what we have been discussing so far. No matter what train_size is, the test_size would remain 1914, as long as the same cv method is used.

Example: To further clarify let's imagine we have the following scenario:

train_sizes = [1, 50, 100, 200, 400, 600]
# X.shape: (1000, 5)
# len(train_sizes)=6

If we were to use the following settings in learning_curve class in sklearn:

cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
learning_curve(estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)

The learning_curve returns the train_sizes, train_scores, test_scores  for six points as we have 6 train_sizes. And for these points the train_sizes and test_size would look like this:

Point 1: train_size: 1,   test_size:200
Point 2: train_size: 50,  test_size:200
Point 3: train_size: 100, test_size:200
Point 4: train_size: 200, test_size:200
Point 5: train_size: 400, test_size:200
Point 6: train_size: 600, test_size:200

I think train_sizes are clear (taken from the list we provided). The test_size remains fixed, and it is decided because we chose test_size=0.2 in the cv (recall that we have had 1000 data points, so 20% of it would be -> 1000 * 0.2 = 200), and it has nothing to do with our train_sizes!! This is the hold-out set that we keep fixed during this experiment. Hope this last example clarifies it altogether. ;-)

I would still keep that blogpost here as reference for others.


I can't speak to the dogbox algorithm,  but the sigmoid only has range (0,1), so fitting to your example data is sure to be bad. The paper you reference presumably scales the input first.

The first example you give has a best fit that's a step function  which can be approximated by the sigmoid with parameters going to infinity; so it's no surprise the algorithm wouldn't converge.

EDIT: Maybe you should try increasing the tolerances (passed as kwargs through curve_fit to least_squares); your error message mentions gtol specifically:  https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html#scipy.optimize.least_squares
Or, if things are converging enough for your purposes, just catch and handle that error?


Yes, some alternatives are feedback-alignment (FA), Direct Feedback Alignment (DFA) and Indirect Feedback Alignment (IFA).


Are there ML Libs in Python robust to missing data?

In general Weka can handle missing values, which are marked as ? in the .arff format. However this doesn't mean that every learning algorithm in Weka has a specific mechanism to deal with them, and unfortunately the documentation is a bit poor about this point. So as far as I'm aware, one has to study the source code and/or do some experiments in order to find out exactly what a particular class does with missing values.
Weka is in Java but there is a wrapper for Python.


How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?

Singular Value Decomposition is a linear algebraic technique as a result of which the notion of normalization is hard to define. In principle, you can do this normalization by dividing each element A(i,j) of the matrix by the sum (or max) of the elements in that particular (ith) row, i.e. A(i,j) = A(i, j) / \sum_{k=1}^{n} A(i,k)

However, a more elegant way to achieve this would be to apply a probabilistic dimensionality reduction technique such as PLSA or LDA. These dimensionality reduction techniques ensure that you always end up with probability values strictly between 0 and 1.


Network structure: k-cliques vs. p-cliques

In graph theory a clique indicates a fully connected set of nodes: as noted here, a p-clique simply indicates a clique comoprised of p nodes. A k-clique is an undirected graph and a number k, and the output is a clique of size k if one exists.

Clique Problem


You can train a LightGBM Regressor. LightGBM Regressor feature selection methods embbed in it. You can direct plot them to see which features are important.
See this link.
LightGBM Plot Importance


How to create custom Keras metric using multiple functions with numpy arrays and matrices?

Solved it using tf.py_function() as a wrapper function around the angular_error() function and added it to callbacks.


What math topics are most needed to become data scientist?

Linear Algebra, Calculus, Probability and Statistics are fundamental topics.

Lots of great books out there, Khan Academy is a good resource as well:

https://www.khanacademy.org/math/linear-algebra

https://www.khanacademy.org/math/calculus-1

https://www.khanacademy.org/math/statistics-probability


Markov Chains: How much steps to conclude a Transition Matrix

I expect you have, or can make, a matrix of transition counts. Consider the data in each row to be draws from a multinomial distribution. Then you should be able to use sample size calculations for the multinomial to get off the ground.

It is also possible that your data is not well described by a simple Markov chain. There are some available techniques for this, e.g. multistate modelling, but which may or may not fit your particular problem.


Best approach for detecting human in a picture

I had a similar problem with skin disease classification. You can see the question and its answers too here.  Gathering a dataset of human pictures could be easy but that of non-human could be nearly impossible ( non-human objects are a huge set of objects )

Solution:

We can use a Siamese Convolutional Neural Network. I have written a blog on it. See here. They perform one-shot learning and are best when we have smaller datasets.

  A standard CNN tries to classify whether the pictures belongs to any one of the classes -> (HUMAN, NON_HUMAN ). The Siamese CNN outputs a similarity score rather than class probabilities. A standard CNN learns to classify and on the other hand, a Siamese CNN learns to differentiate images.

How does it work? ( In the context of your task )

Suppose I have an input image $x$. This is our input for the Siamese CNN. I have two more images, say $f$ and $g$. Image $f$ belongs to class 1 ( HUMAN ) whereas $g$ belongs to CLASS 2 ( NON-HUMAN ). The Siamese CNN takes in 2 images as input and outputs how similar they are.

$ Siamese( x , f ) = 0.678$

$ Siamese( x , g ) = 0.987$

$Clearly, \space Siamese( x , f ) < Siamese( x , g )$

Therefore, the image belongs to CLASS 2. You can find its other advantages on the blog. Hope this helps.


How does Pytorch deal with non-differentiable activation functions during backprop?

The function value is never exactly equal to those exact point because of numerical precision error.And again those functions in torch calculate left or right derivative which is defined in every case.So non-differentiability doesn't pose a problem here.


How to Visualise a CNN model using Python

Add this two lines below of your code.

from keras.models import Model
model = Model(inputs=input, outputs=output)
print(model.summery)


does xgboost's eval_metric changes the loss function being optimized?

Chaging eval_metric does not affect the fitted model in any way. You have to interact with objective to do this.


Predicting time series, with few historical samples based on similar series

I'd suggest statistical forecasting techniques such as ARIMA or ES, but those models usually cannot generalize well across timeseries, so you'd need one for each timeseries

A good starter for using LSTM for forecasting is here - https://www.tensorflow.org/tutorials/structured_data/time_series. But if you don't have enough data, NNs would like result in poor test results.

For your case, I'd suggest you trying a regression approach- structuring your timeseries into a regression features format. After that you can use regression models from sklearn, starting with linear and moving to more complex ones. Because you have less data, you might want to explore less complex models first to prevent overfitting. For the features, for example, you could create lag features (what was the value of your signal two timestep back, what was the mean/std of the past 2 timesteps, what was the maximum in the last 6 timesteps and so on). Look up any kaggle competition on timeseries forecasting if you want to refer code or get specific ideas on feature extraction


Dimensionality reduction to correlate large number of variables

Yes, in scikit-learn, you can find the correlation between the elements using LedoitWolf Estimator. For dimensionality reduction, I assume you will use PCA but then, you want to backtrack the reduced to the original data, for that I don't have a solution, since, PCA transforms your data and computes the largest variance direction. But, yes both of these won't require any labels.


Clustering of sparse matrix with many co-variates

You need to apply PCA and reduce your data to lower dimensions, then applying a classic clustering technique (e.g. k-means or DBSCAN) works depending on how your samples are distributed. I strongly recommend you visualize data in 2 or 3 dimensions and have a brief visual inspection. Gives you an insight about what is going on there. However the final number of dimensions you get out of PCA might be chosen to be more than 2 or 3 (it is usually).

Steps

Normalize features if needed
Apply PCA and take the number of PCs which explains 85% of variance
Optional: Visualize embedded data in 2 or 3 dimension to get a feeling about distribution of samples (it does not anything more than just an intuition! Even that intuition can not be validated)
Apply a clustering algorithm on the result of (2)


The current approach use 70/30 or 80/20, the most used is 80/20 (train/test). However there is other things you should check, for example if you data is balanced. If your data is not balanced you might want to use undersample or oversample.


How to adaptively sample n-dimensional data and build an optimum training set

Take a look at active learning.

https://en.m.wikipedia.org/wiki/Active_learning_(machine_learning)

In this approach you can train with a subset of samples and add more depending on what samples are most informative for the model.

Alternatively you could try reducing the number of samples by clustering and choosing a small number of samples from each cluster.


How to know for sure if we can learn from a given data or not?

how can we know for sure

We can't.
A toy example to show why even humans can not do this for sure:

Assume you get the number 2, 4, 8, 16, 32, ?? and want to extrapolate to the next number ??. A natural extension of the series would be 64, but we can not take this for granted. The next number can just as well be 0. You can not be sure.

Only given the data and without additional assumptions about what you would expect to see, you can not learn a correct model per-se. You always have to be critical about your data.


Converting Json file to Dataframe Python

I once ran into a situation like this where i wanted a complex dataframe due to the original source having a complex data structure.

I solved my issue by simplifying my structure into multiple separate dataframes instead of one big complex multi structure dataframe.

With simple separate dataframes i was better positioned to apply complex algorithmic operations.

Looking at your specific data, you could get rid of userIdentity which results in a simple 2d dataframe. This should position you to do any complex operation.

I understand this doesn't answer your specific dataframe structure requirement.

But i hope this answers the spirit of your objective.


Regularization is used to help smooth multi-dimensional models.
Take this example,
y = x_1 + eps*(x_2 + ... + x_100)

Let's say eps is very small. It  doesnt seem very useful to store those 99 coefficients, isn't it? How do we manage to fit a model in such a way that we drop negligible coefficients? This is exactly what L1-regularisation does!
Each other type of regularisation has another geometric intuition.


Fitting Gaussian to data: Density-Estimation vs Regression

The difference comes about because the sample mean and sd estimators are assuming the data is a random sample from the distribution on the full range of the distribution, in this case -Inf to +Inf. In your second case above there's no data below zero. The sample mean estimator can't see that you've only got half a Gaussian there, it will always be a positive number.

On the other hand, the curve-fitting approach of the least-squares fitting procedure (which is presumably possible via EM, or any of a dozen other algorithms) can see the data as part of a curve that has a Gaussian density shape. If the data is just curving upwards, it will fit to the left side of a Gaussian. If its curving down it will fit to the right side. If its flat it will either fit it to the peak with a very large variance, or a tail and might even go degenerate and divide by zero.

So you are doing parametric estimation of a distribution, or curve-fitting, which are categorically different things.


how to reshape xtrain array and what about input shape?

Keras requires you to set the input_shape of the network. This is the shape of a single instance of your data which would be (28,28). However, Keras also needs a channel dimension thus the input shape for the MNIST dataset would be (28,28,1).

First we load the data as you did,

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

  Training data shape:  (60000, 28, 28)
  Testing data shape :  (10000, 28, 28)

Then we reshape the examples in the MNIST dataset to have the additional channel dimension

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

And now we can define our model as you did. Do note the necessary Flatten layer between the Convolutional layers and the Dense layer.

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.fit(xtrain, ytrain, epochs=10, batch_size=60000)


What kind of RAM to choose for data analysis?

It's not easy to compare two RAMs with different frequency and latency, since both of them affect your performance with not the same way.

The short answer is:

If you have two RAMs with same capacity and frequency, choose the one with lower latency.
If you have two RAMs with same capacity and latency, choose the one with the higher frequency.

From lifehacker.com

  Essentially, you have two things to worry about when it comes to RAM "speed": frequency, which deals with how much data can be transferred to the stick at one time, and latency, which is how quickly it responds to requests. In the current market, as you get to higher frequencies, latency tends to increase, so in many cases, they tend to balance each other out. Buying RAM with a higher speed doesn't matter a ton.

The long answer:

Don't worry about RAM. Yes, if you have a crappy RAM, you will also have a problem with your performance. But, the most important thing in Data Analysis is the performance of your code. An example from my personal experience:

I had a text dataset, about 30GB. I needed to create a corpus for a simple TF-IDF. The brute-force code, spent about 15 hours, whereas a map-reduce algorithm spent about 1.5 minute for the same dataset.

I work on a MacBook Pro with 16GB RAM. I didn't have any problem with performance. However, if you have huge datasets and even code improvements and RAM upgrades cannot help, you may reconsider to move to Hadoop or similar.


Finding parameters of image filter using classified pairs

Your parameter $\alpha$ has fairly low dimension.  Therefore, I recommend that you apply optimization methods directly to try to find the best $\alpha$ (without trying to use convolutional neural networks and regression for this purpose).

Define a distance measure on images, $\|I-J\|$, to represent how dissimilar images $I,J$ are.  You might use the squared $L_2$ norm for this, for instance.

Now, the loss for a particular parameter choice $\alpha$ is

$$L(\alpha) = \sum_{k=1}^N \|F_\alpha(I_k)-J_k\|.$$

We can now formulate your problem as follows: given a training set of images $(I_k,J_k)$, find the parameter $\alpha$ that minimizes the loss $L(\alpha)$.

A reasonable approach is to use some optimization procedure to solve this problem.  You might use stochastic gradient descent, for instance.  Because there might be multiple local minima, I would suggest that you start multiple instances of gradient descent from different starting points: use a grid search over the starting point.  Since your $\alpha$ is only three-dimensional, it's not difficult to do a grid search over this three-dimensional space and then start gradient descent from each point in the grid.  Stochastic gradient descent will allow you to deal with fairly large values of $N$.

This does require you to be able to compute gradients for $L(\alpha)$.  Depending on the filter $\alpha$, it might be possible to symbolically calculate the gradients (perhaps with the help of a framework for this, such as Tensorflow); if that's too hard, you can use black-box methods to estimate the gradient by evaluating $L(\cdot)$ at multiple points.

If $L_2$ distance doesn't capture similarity in your domain, you could consider other distance measures as well.

I expect this is likely to be a more promising approach than what you sketched in the question, using convolutional networks and a regression model.  (For one thing, there's no reason to expect the mapping from "features of $I_k$" to "features of $J_k$" to be linear, so there's no reason to expect linear regression to be effective here.)


The description of the needed data is quite general, but I have found this dataset that describes sales during Black Friday; it is possible to identify products bought by the same customer using customer_id column.

Also, this dataset describes transactions from a bakery. The products that were bought in the same transaction (and so by the same customer) are identifiable as they share the same #Transaction value.

Generally below are 2 sources where someone can find free datasets that can be used for training/ educational purposes:

Kaggle: https://www.kaggle.com/datasets (both datasets that I suggested were found here)
EU Open Data Portal: https://data.europa.eu/euodp/en/data


It is better to formulate this problem in terms of classification, not regression. For example, you can apply sign transformation on your labels (e.g. $y = numpy.sign(y)$) before training the model, then fit the model with classification loss (like binary cross-entropy), and then you can use accuracy for measuring the performance. Using accuracy with regression setup outputs zero because comparing real numbers usually outputs False (like 1.2 != 1.19).


First, for your problem you can read https://www.sciencedirect.com/science/article/abs/pii/S0031320310004619
Unfortunately there are no closed answers for these questions... One of the major tasks of a ML engineer is to decide what algorithm to apply and what features from the original data to use to solve a specific problem... Both decisions affect the other. Almost every single decision in ML is strongly related to others... Other important things to bear in mind are the amount available data, hardware constraints and so on...

how do I know what data to provide a machine learning algorithm?

In classical ML, the engineer has to decide which features to compute. My advice is consulting expert's papers and contributions that solved similar problems and take advantage of their expertise. Features are dependent on the specific problem, domain and data. In audio processing there are quite well known: MFCC, Spetrograms...
In audio, a typical approach is to divide the audio signals in overlapping frames and compute the features over them. In that way, the evolution or dynamics of the audio features can be analysed over time. Taking the spectrogram is an example of that, it can help analyzing the evolution of the frequency components.
Data preprocessing is really important too: audio can be normalized or filtered as a previous step to feature computation.
In Deep Learning, neural networks try to compute the best features for you. However, a raw audio signal is rarely directly fed into a ANN. Typical procedure here is to compute the spectrogram (which can be interpreted as an image) and use Convolutional Neural Networks or Convolutional LSTM networks, or even Transformers.

what sort of machine learning algorithm would I use?

Depending on the problem, there are state of the art methods for image classification, detection, speech recognition...
The first thing you need is to really well define what do you want to do, knowing the problem... then, once again visit other scientists approaches.
Once you have a clearer idea, if you have hardware and computing time constraints, you need to figure out what kind of solution will adapt to them.
Theses comments may not seem very clarifying, but you dropped two really broad and difficult to answer questions.


(Not a programming question) Is there a common word to indicate predict or forecast

I am not sure if I agree with the first one, since predict is widely used to replace forecast as well. But other related verbs we often hear are:

To estimate
To infer / make an inference


As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported.

# If your main Python version is not 3.5 or 3.6
conda create -n test python=3.6 numpy pyyaml mkl

# for CPU only packages
conda install -c peterjc123 pytorch

# for Windows 10 and Windows Server 2016, CUDA 8
conda install -c peterjc123 pytorch cuda80

# for Windows 10 and Windows Server 2016, CUDA 9
conda install -c peterjc123 pytorch cuda90

# for Windows 7/8/8.1 and Windows Server 2008/2012, CUDA 8
conda install -c peterjc123 pytorch_legacy cuda80

pytorch support for windows Link


The exception list files are used to help the processor find base forms from 'irregular inflections' according to the man page.  They mean that some words, when plural or a different tense, can't be algorithmically processed to find the base/root word.  More details can be found in the morphy man.  I'm not a language processing expert, but this is likely a result of English words that 'break the rules'.  If you think about the code like a human trying to learn English: the student learns rules to use (algorithm) and then has to memorize exceptions for the rules (exception lists).  An over-simplified analogy that does not involve endings/conjugation would a spell checking program.  An algorithm might check for 'i before e, except after c' but would first have to check the word against an exception list to make sure it isn't 'weird' or 'caffeine' - please don't start a linguistics fight about this rule, I am not commenting on the validity of it/that's not the point I'd like to make.


How do I split contents in a text that would include two or more different themes (context) in NLP?

First reading the question I thought this is very easy and I started searching and trying out some libraries (i.e. nltk, and spacy). Here are my attempts and clearly showing that none of them works. I was thinking that these libraries will generate a parsing tree or a context-free grammar to detect sentences, but they are splitting based on a certain characters such as periods or commas.
So the other solution is to learn a classifier that learns the boundaries of sentences. A simple solution can be to use random forest classifier or SVM to serve as a binary classification task for each token. But the missing part is the data to train the model on and what to consider as labels (a BIO tagging scheme is a good idea here). I include two papers here that can be a good starting point for you to first read their methodology and use the datasets they are mentioning to do the task.

Du, Jinhua, Yan Huang, and Karo Moilanen. "AIG Investments. AI at the FinSBD task: Sentence boundary detection through sequence labelling and BERT fine-tuning." Proceedings of the First Workshop on Financial Technology and Natural Language Processing. 2019.
Rudrapal, Dwijen, et al. "Sentence boundary detection for social media text." (2015).


The question in the title is inconsistent with the question body. If you are intent on using a CNN albeit it not being the best idea for this type of data, then for the dataset you are describing a 1D CNN architecture would be most suitable.

To answer the question in the title, yes a 3D CNN is definitely possible and is actually commonly used on 3D data, for example: video (3rd dimension is time), 3D medical images, etc.

The instances in the dataset you described is a vector with 20 instances. This data is a single dimensional structure. You can thus use a 1D CNN as described here.

However, it would be best to use a DNN for such a data structure. As a CNN is essentially cross-correlation of neighboring features. In this case, your data cannot be assumed to have such similarities between adjacent features. A DNN will allow for cross-correlation between distant features to be captured more effectively.


How to set input for proper fit with lstm?

You should be structuring your data as a tuple: Number of samples, timesteps, features. In your case number of samples is 53394, timesteps is 1 and number of features is 3. So the input shape will be (53394, 1, 3).
You can use this snippet of code for the tranformation:

trainX = trainX.reshape((trainX.shape[0], 1, 3))

And set the argument input_shape  as below:

input_shape = (trainX.shape[1], trainX.shape[2])

Hope this helps!


Predicting house price using linear regression


Formula to calculate confidence value in Adaboost

Actually this equation is not quite right.

The actual minima occurs at

In the derivation this BetaM is the solution to the equation

Notice  how in the function we are scaling the weights of wrong predictions up by exp(BetaM) and scaling down the weights of correct predictions by multiplying exp(-BetaM).

But in the code that I have written (also in the sklearn implementation) only the wrong predictions are being scaled up. So to get the relative scaling correct we have to scale the wrong predictions by exp(2BetaM)
i.e.

And as of the hypothesis function has the form

Any scalar multiple of BetaM will work fine as all the predictions will be scaled by the same amount. So for convenience in coding BetaM is written as just

Note: You can perfectly use BetaM = 0.5 * log((1/errM)/errM) . But then you have to scale down the correct predictions too. If you do that then the code will give correct results.


How to add a column for descending row numbers into dataset in R

Use mutate in combination with row_number as follows:
df %>% mutate(row = row_number())


How to tune an weighted voting ensemble method?

The question is a classical optimization problem.
To make it more simple, lets say that you have 2 models (m1 and m2). And you want to do an ensemble.
First idea, is doing the mean of both
(m1.predict(X) + m2.predict(X))/2

Then you want to optimize the best combination of both models.
for w in [0,0.25,0.5,0.75,1]:
     (w * m1.predict(X) + (1-w) * predict(X) ) / 2
     print(evaluate results/MSE/Accuracy...)

Your problem is exactly the same as this one but using the voting classifier.
You just need to optimize the array. In a for loop.
weights=[5, 5, 10, 5, 6, 8, 4, 10]

In this case, your array is not normalized and it has more models. So it will take longer. Also voting classifier fits every time the model, so I will take you a long time. Is easier to code it yourself.


Pearson correlation with data sets that have values on different scales

For quantitative-quantitative comparisons, scales generally do not matter in spearmean, pearson, and kendall correlation.


User identification using machine learning

From your question I'm interpreting that you've extracted data from a log which has TCP session related data, some of the fields you've mentioned are: source/dest IP address, TTL, start time, end time.
You're trying to segregate this network traffic to isolate the traffic belonging to 10 unique client machines.

There are several ways to approach this client traffic segregation problem. But first you need to identify which variables are going to be most meaningful under different circumstances, since the network setup is not described, I'll list the different possible scenarios:

Clients reaching the server without intervening firewall/proxy: In this case each client will have a unique source ip-address which should help segregate the traffic without further analysis.
Clients behind a proxy which filters traffic and changes the ip-address to set a unique ip-address of the gateway for all the traffic coming to the server. In this case, you could use a combination of the following:

a) Segregate clients by TCP/IP packet attributes: Attributes such as TTL, SYN packet size, TCP window size, etc, can identify client OS. For example - Linux and Mac-OSX have default starting TTL values of 64, whereas windows has a default of 128, others like Cisco/Solaris 2.x have 255. So you can roughly segregate the traffic by OS type.

b) Since the TTL is decremented by 1 for each router placed between the client and the server, so even if you have 2 windows machines, but placed in a network segment separated by one more router, then the TTL number will be one lesser than the other windows clients.

c) Further identify unique sessions by the sequence number and acknowledgement number in the TCP headers. Each acknowledgement number of the destination server can be correlated by the seq number of subsequent packets from the client for that session. This will let you segregate users connecting to the same server at the same time e.g. 10 different sessions to stackexchange.com.

d) In case you can get further web-server log details, then attributes such as user-agent strings and HTTP cookies can help you identify a client.

e) In some cases, the random source port no. used by clients for the TCP session might overlap for different clients accessing different dest. ip-addresses at the same time, you can then safely assume these are traffic form two different clients.

This still leaves you with the problem of segregating traffic for different servers at different points of time. For this you could analyze the data in the following manner:

Identify a client using their ISN: There is a certain logic in generating random initial sequence numbers by each OS, and if you have a large sample of ISN nos available, then you could identify the attributes of the random number generator to pinpoint the client. This ISN generation process has been exploited for DoS spoofing attacks, and many OS's have strengthened it since then, but this may still provide some clues to identify machines uniquely.
Analyse SSL handshakes to identify clients via their attributes such as supported cipher suites information exchanged during the SSL setup handshake, etc. See this paper for details about this technique.
The source port nos. are usually assigned in increasing order, and it is unlikely that every machine started at the same time and had the same TCP traffic volume; this helps identify clients uniquely.
TCP/IP timestamps: Most OS's will increment the timestamps in TCP headers and this keeps increasing as long as the system is up. Since systems may be switched on at different times, it helps segregate clients even if they have exactly the same OS and browser versions. See this RFC for mode details.
Some proxies my use extra HTTP headers (Via, X-Forwarded-For) to indicate the exact client hostname or address requesting the data. This provides a  straightforward way to identify the client, but only if you can access the HTTP header information from the packet.

After segregating traffic using all these means, then you may use network traffic time-series analysis to identify distinct users from the periodicity in their usage. For example, the repetitive pattern of a person who logs into work everyday at 8:15 AM and checks on some specific websites for 5 minutes can be identified and that client uniquely segregated from the rest.

More hypothesis and directions to take can only be identified after exploring the dataset in detail.

Once you're satisfied with this preparatory work, you could then calculate and add all these features along with the other direct indicators to train a multinomial classifier for predicting these 10 user ids. You could use a random forest, LogitBoost or Neural Networks as starters. If a variable is not identified as important for the classification, then discard it to reduce the total number of features.

For new traffic, you would need to generate all these features and then use the final trained model to classify the traffic by user.


How can I have an "undefined" category in multi-class classification

Absolutely you can create an approach that forces high-precision class tagging algorithm (at the natural cost of recall). What's more- you can do this with (at least) any method that provides a percentage calue for predictions, which is the vast majority of classifiers. The key is, as you mention, to find the minimum acceptable value of precision and cut the predictions at that value.

If a minimum precision is your only constraint and your solution is not sensitive to the recall (getting all or the highest possible proportion of the websites correctly classified), this is a very simple matter. Some lower percentage of your observations will be classified, but those that are will be more likely to be correctly classified.

For example- if your Precision floor is 70%, your cut could look something like this:

Obeservation predictions falling into the green segment could be classified as positive examples, and predictions falling into the red segment would be unclassified.

This approach would be sufficient for Naive Bayes. Some other approaches (SVM, gradient boosting machines etc) may benefit from a custom loss function definition, in which you define a function that disproportionately punishes false positive predictions.

Something like:

$(y_i = 0) \rightarrow (d = s)\wedge (y_i = 1) \rightarrow (d=1)$

$L(p) = \frac {\sum_i\frac{(p_i-y_i)^2}{d}}{n_i}$

$L(p)$ = loss function

$p_i$ = predicted class likelihood

$y_i$ = actual class (0 for negative example, 1 for positive)

$n_i$ = number of observations

$s$ = Penalty parameter for false positives.

Would heavily punish false positives relative to false negatives. It can also be adjusted to your needs, to more or less heavily punish false positives. For $s = .5$, the function looks something like:

Please note that this function does not necessarily define the approach you should take, but only one possible approach. To create a custom loss function perfectly tailored to your use case, you'd want to know the relative cost of a false positive and a false negative, and customize the function accordingly.


Different number of features in train vs test

You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.

Something like this:

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])


What is the best hardware/GPU for deep learning?

I would recommend you read this article carefully: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/

Basically,

The most important feature for judging deep learning performance is memory bandwidth. GTX 1080 Ti's memory bandwidth is 484 GB/s and TITAN X is 480 GB/s. Read this Quora answer to understand why memory bandwidth is the most important feature better: https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning.
Tim has also given a "theoretical" rough comparison between GPUs based on their tech specs. You can also see that in his comparison 1080 Ti and TITAN X are roughly at the same performance level.
Use 2 1080 Ti instead of a TITAN X since GTX 1080 Ti is definitely cheaper.
I've personally evaluated the performance of 1080 Ti vs. TITAN X and they are roughly the same.


How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?

using the word-topic and topic-word matrix

This would be the same, wouldn't it? The model generates a words x topics and a topics x documents matrix, there is not much left to calculate, the model practically spits it out.

Using the Gibbs method gives you the counts Ntw[1..T][1..W] and Ndt[1..D][1..T], where for example Ndt[1][5] is the amount of words assigned to topic number 5 in document number 1.


Designing a ConvNet to facilitate game playing

I've recently started using OpenCV's python implementation, and I found some good OpenCV tutorials on this website: http://www.pyimagesearch.com/ that I really liked. OpenCV allows you to do Haar Cascades for fast facial recognition (by default it doesn't use a convoluted neural network but an optimized implementation of Ada Boosting that evaluates frames in stages for faster processing). OpenCV converts each frame into a multidimensional numpy tensor/matrix that you can then feed into into your ML algorithm (e.g., in TensorFlow or some other library), although I think most people just use the built-in OpenCV face classifiers. In any case, I believe OpenCV can process up to 70 frames per second, so it should be fast enough for you.

The original paper that invented Haar Cascades:
https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf

The OpenCV documentation that further explains Haar Cascades:
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0


selecting sample from population using genetic algorithm

There's nothing super special about a population size of L^2. Larger populations are often better, but don't worry about trying to hit some threshold needed for the algorithm to work.

There are a couple of potential issues here that I see. I'm making some assumptions that when you say "individuals", you mean people. Which means an average person is going to contribute something on the order of 50-100 kg toward your limit, which puts you at like 50 people in the sample in order to stay under the 3000kg limit.

With your encoding scheme, that's going to mean only very sparse vectors are feasible solutions. You can deal with that, but it likely means a standard GA pulled from a library won't work as well as one that included some prior knowledge. For example, when initializing your population, you might want to bias things pretty heavily in favor of setting bits to 0 rather than 1.

In any case, you have constraints, so you'll want to include some type of constraint handling scheme. A simple thing you can do in this case is to perform a simple local search on every individual you generate. For example, suppose I have an individual i selected from my population. If the total weight of the sample encoded by i is less than 3000kg, iterate over every bit of i (in random order) and see if you can flip a 0 bit to a 1 bit without exceeding your weight capacity. Similarly, if you generate a sample that weighs more than 3000kg, try to find 1 bits you can flip to remove weight until you're under the limit. When you're done, you have a feasible individual and you insert it into the child population.

You can incorporate more sophisticated measures or search schemes as well. You could have your local search operator consider those other variables. You can replace the simple hill-climbing operator with a more sophisticated method like tabu search or simulated annealing. There's a spectrum of algorithms with all sorts of different ways of combining the ideas of the global search of a GA with more targeted local search operators, and there's no general "right answer". You mostly just need to experiment a bit to see what works.


Necessity of balancing positive/negative examples in binary classification machine learning?

Let's start by answering your first question. Is it required to balance the dataset?

Absolutely, the reason is simple in failing to do so you end up with algorithmic bias. This means that if you train your classifier without balancing the classifier has a high chance of favoring one of the classes with the most examples. This is especially the case with boosted trees. Even normal decision trees, in general, have the same effect. So it is always important to balance the dataset

Now let's discuss the three different scenarios placed.

Choice A): This would be what I explained all along. I'm not saying necessarily you will have a bias. It depends on the dataset itself. If the nature of the dataset has a very fine distinction with the boundaries then the chance of misclassification is reduced, you might get a decent result but it's still not recommended. Also if the data does not have good boundaries then the rate of misclassification rises a lot.

Choice B): Since you are placing weights for each sample you are trying to overcome the bias with a penalty. This is also called as an Asymmetric method. Normally these methods increase the accuracy of a model by a slight margin but that mostly depends on the machine learning algorithm you are using. In examples like Adaboost such a model the effectivity of the model increases. This method is also called Asymmetric Adaboost. But this might not necessarily work with all algorithms.

Choice C): Assuming you have weighted the samples accordingly it should do the same as either choice A or choice B. I'll leave this for you to extrapolate based on my previous explanations.


a precision recall AUC score of only .44 seems very low.

I agree, but your model has definitely learned something because you beat your naïve baseline (technical term not criticism!) Depending on what you’re trying to predict, that AUC might either be impressive or disappointing. By a similar token, the model you have trained here might be useful as it is, or might not have reached a minimum viable level of performance.

How do I interpret this?

When your model is able to retrieve half of the positive samples (~50% recall) it happens that about half of the samples it identifies as positive are actually negative (~50% precision). Beyond that, the precision decreases as we attempt to get greater recall, which is what we’d expect.

Is it actually a bad score?

Has anyone tried to predict the thing you’re trying to predict before? Maybe there is a more complex baseline you could compare yourself to. If not, you’ll need to do more work to establish whether you could do better. For an unbalanced dataset, gradient boosted trees are a good baseline beyond the naïve (no learning takes place) baseline. Try something like xgboost if you haven’t already. Take the model you just trained as a new baseline and try some other stuff.

How can I understand this curve in reference to the baseline?

The baseline you are using represents the precision you would see if you just always predicted that samples were positive. This would obviously result in 100% recall. If your model moves above that in terms of precision, anywhere on the graph, it may be learning something. It’s clear this is the case as it is doing consistently better.


Multidimensional K-Means wiith sklearn, centroids problem when plotting

In your plot you used PCA to reduce the dimensionality of your data, but you plotted the first 2 dimensions of your centroids. You should also transform the centroids using the PCA transform you fitted on your data.
This code should work for you
numClusters= 12
kmeans = KMeans(n_clusters=numClusters).fit(X)
centroids = kmeans.cluster_centers_

# Predicting the clusters
labels = kmeans.predict(X)
# Getting the cluster centers
C = kmeans.cluster_centers_

#transform n variiables to 2 principal components to plot
pca = PCA(n_components=2)
pca_fit = pca.fit(X)
principalComponents = pca_fit.transform(X)
principalDf = pd.DataFrame(data = principalComponents
         , columns = ['principal component 1', 'principal component 2'])

colors =['red','green','blue','cyan','yellow', 'lime','orange','coral','brown','peru','khaki','tan']
centroidColor= []
for item in range(numClusters):
  centroidColor.append(colors[item])

dataPointColor=[]
for row in labels:
  dataPointColor.append(colors[row])

fig = plt.figure(figsize = (10,10))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
plt.scatter(principalDf['principal component 1'], principalDf['principal component 2'],
c=dataPointColor, s=50, alpha=0.5)

C_transformed = pca_fit.transform(C)
plt.scatter(C_transformed[:, 0], C_transformed[:, 1], c=centroidColor, s=200, marker=('x'))
plt.show()


Purpose of visualizing high dimensional data?

First of all your explanation about the methods are right. The point is that Embedding algorithms are not to only visualize but basically reducing the dimentionality to cope with two main problems in Statistical Data Analysis, namely Curse of Dimentionaliy and Low-Sample Size Problem so that they are not supposed to depict physically understood features and they are not only meaningful but also necessary for data analysis!

Actually the visualization is almost the last usage of embedding methods. Projecting high-dimensional data into a lower-dimension space helps to preserve the actual pair-wise distances (mainly Euclidean one) which get distorted in the high dimensions or capturing the most information embedded in the variance of different features.


Final layer of neural network responsible for overfitting

Are you balancing your training dataset 50/50? It seems to me that you are making a lot of changes to account for 80/20 balance when it sounds like you have a deep enough dataset to balance your training to be 50/50 and still have a reasonable size for your training set. How about keeping it to a smaller network - as you and some notes above suggest - and with a 50/50 set on a smaller training sample?


This is a classic "graph database" question. I would do the following steps:-

Traverse all the pages using a webcrawler, or if you can interrogate the website server directly that may be qukcker.
Ensure you keep all links between pages, for later use.
Put the web pages into a neo4j database with connections between the pages

This will allow you to be able to do the query of pages with the maximum number of steps, or pages most frequently linked to, etc.


Getting a ValueError from train_test_split

This is the implementation of the train_test_split function

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

Split your train into X and y and try again!


How to find similar meaning log messages in the source code?

A simple idea, non-supervision:
 1.Stemming, reduceing punctuations. Then you can use Bag of word with TF-IDF to represent the log sentences.

Run K-means to cluster your sentences. If your sentences are belong to
5 type. You can set K=5.

Of course, you can use Regular Express to classify those sentences.


A good starting point is to look at the evaluation measures used in the NER shared tasks: https://nlpprogress.com/english/named_entity_recognition.html.
Generally the F1-score can be used for one specific class, but there are different options regarding what is counted as an instance:

every occurrence of the full NE. In this case any difference between the predicted and the gold is counted as false, even if it's only one token difference.
every token in an entity. In this case a partially matched entity counts as "partially correct": if a word is predicted outside instead of inside, it's a false negative and conversely.
Other variants:

count only unique entities, in order to observe the diversity of the entities recognized.
count only entities which didn't appear in the training set, to observe the generalization power.

(writing this from memory, I could miss something)


tdfread displays the File Open dialog box for interactive selection of a data file, then reads data from the file. The file should have variable names separated by tabs in the first row, and data values separated by tabs in the remaining rows. tdfread creates variables in the workspace, one for each column of the file. The variable names are taken from the first row of the file. If a column of the file contains only numeric data in the second and following rows, tdfread creates a double variable. Otherwise, tdfread creates a char variable. After all values are imported, tdfread displays information about the imported values using the format of the tdfread command.

tdfread(filename) allows command line specification of the name of a file in the current folder, or the complete path name of any file, using the string filename.

tdfread(filename,delimiter) indicates that the character specified by delimiter separates columns in the file. Accepted values for delimiter are:

' ' or 'space'
'\t' or 'tab'
',' or 'comma'
';' or 'semi'
'|' or 'bar'
The default delimiter is 'tab'.

s = tdfread(filename,...) returns a scalar structure s whose fields each contain a variable.


Tensorflow classification - maximize the accuracy of certain classes

The best approach for this problem would be to change the value of X. For example, if you are using -%2 and %2 as the classification limit such as  sell(<-%2), buy(>%2) and no action otherwise; you can reduce this to %1, which will in turn reduce the number of samples that fall into this class while increasing number of samples in buy and sell classes.

If you want to keep that value, the way to go is undersampling: when training you take equal number of samples from each class, meaning you ignore some portion of the data from no action class. So you training and validation sets will be balanced and you can use accuracy as a proper metric.

If this also is not a good option for you, another way would be to try changing the classification threshold for each output so that their possible outcomes are roughly equal. But this might be a little complicated to implement as you have 3 classes (for 2 classes this can be done in a relatively simple way by changing classification threshold from 0.5).

If none of these suit you, you can find a more general approach to imbalance problems here. In case you wish to use a different metric, the most popular ones are listed and discussed in detail here. You probably don't need to define a custom metric because there is a one for pretty much every choice.


How to convert vector values to fit k-means algorithm function?

You are 95% there, you just have one hangup...

The vectorization that you are doing is alternatively known as binarization or one-hot encoding.  The only thing you need to do now is break apart all of those vectors and think of them as individual features.

So instead of thinking of the question one vector as $(0,0,1,0)$ and the question two vector as $(0,1,0,0)$, you can now think of them as individual features.

So this:

-      q1,        q2
-      (a,b,c,d), (a,b,c,d)
user1  (0,0,1,0), (0,1,0,0)
user2  (1,0,0,0), (0,0,0,1)

Becomes this:

-      q1a,q1b,q1c,q1d,q2a,q2b,q2c,q2d
user1  0   0   1   0   0   1   0   0
user2  1   0   0   0   0   0   0   1

And you can think of each one of those binary features as an orthogonal  dimension in your data that lies in a 8-dimensional space.

Hope this helps!


Residual plot is tight to zero, but low R2 score

According to the plot there is a huge outlier in your residual. Plus taking into account that your cross-validation sometimes shows quite good results (0.77, 0.74) and sometimes not so great (0.42, 0.36) I would first try to remove that outlier from your data and see what you get.


Where to get models with weights instead of only weights? What's the purpose of .h5 files?

There are several options when saving and loading a keras model, as explained at https://www.tensorflow.org/guide/keras/save_and_serialize:

save the whole configuration, including the architecture, weights and even the last training state
but also the model architecture and the weights can be saved as independent files, and that is what you might have loaded: an .h5 with only the weights, and you also need the .json with the model architecture, based on the snippet below:

-model architecture and model weights to disk as separate files, which is what it might be done by the user who trained the model:
json_config = model.to_json()
with open('model_config.json', 'w') as json_file:
    json_file.write(json_config)
# weights saving to disk
model.save_weights('path_to_my_weights.h5')

So you need to do the following:
-load the model as follows (you need the model json also):
with open('model_config.json') as json_file:
    json_config = json_file.read()
new_model = keras.models.model_from_json(json_config)
new_model.load_weights('path_to_my_weights.h5')


The reduce error pruning strategy works in the following way:

Train a tree on a training data set
Fit a pruning data set (which is different than the training data set) to the tree. What you will have is the tree as was learned at step one, but for each node you will have some instances from the pruning data set.
Prune each node from the tree if the misclassification error computed for the instances from the pruned data set is not larger than the original misclassification error rate computed on the training data. For example, suppose that you do a binary classification and you build a tree from a training data set. Now you fit data from the pruning data set and some instances arrives in a given non-leaf node. Suppose that at that node arrives $2$ positive instances and $8$ negative instances. This gives an error rate of $0.2$. Now from that node the instances are split given to the criteria already learned into two children node, one will have $2$ positive instances and $2$ negative instances, and in the other child node goes the other $6$ negative instances. The cumulative error on children is also $0.2$. This means that going further with the split would make no sense, since the data varies a lot. That means you can actually cut the child nodes and make the original non-leaf node a leaf node.

The point of using fresh data (pruning data set) is to check if the split is useful or the noise from data covers the signal given by the split.

To evaluate the performance of both models, before and after pruning, you need another different data set, named validation data. To see way you have to understand that pruning is actually also part of extended training. The only difference than usual learning is that in training with REM you learn how to extend using some data and you learn how to unlearn using some other data.

If you use the same data from training to do pruning you simply do not progress (you can stop early). To use for pruning the data set for testing will invalidate you model selection criteria, since it will almost always be the case that the pruning tree performs better (original tree learns from training data set, pruned tree learn from both). I dare to say that if you would like to asset the prediction error, you will have to use a different data set, or envelope everything in a cross validation.


Just to point out a minor confusion that there seems to be in the wording: there is mixed use of the the words temporarily and temporally. [OP has since corrected this]

We really only care about the final word, temporally i.e. relating to time, or the timestamps of the points in a cluster. Temporarily means only for a short period of time, which is a slightly different meaning to temporally' being related to time.

The way I understand the definition, a cluster is defined as a stop cluster if the minimum and maximum timestamps included that cluster ($mint$ and $maxt$)

Also, it seems definition 9 implies the usage of definition 7, so I would assume you need to implement it. If would be used a criterion as to whether a candidate cluster $C_i$ is defined as a stop cluster. So the final pseudo code would be something like:

stop_clusters = []    # empty list
for candidate in candidate_cluster:    # iterate over each stop cluster candidate
    TC = cluster_is_tc(cluster)        # check cluster is Temporally Continuous
    TO = cluster_is_not_to(cluster)    # check cluster is Temporally Overlapping

    if (TC & TO):                      # if both criteria are met...
        stop_clusters.append(cluster)  # assign to stop clusters

How you implement those two check functions would be quite straight forward, according to the definition, assuming you have the timestamps of each point, you are trivially checking that all points are contained in the time bounds. Just skimming page 3, I am afraid the paper doesn't make these time bounds completely clear to me, but I hope it is clear to you as you are implementing it.

The same is true for the temporally overlapping criterion: you check that there is no overlap of timestamps in two clusters - if there is overlap (i.e. the intersection is not an empty set), then you cannot call the current candidate cluster a stop cluster.

Edit:

Looking at that implementation you linked, it could be that that are performing the check Temporally Overlapping check at this if line for the case of being non-overlapping and this else line for the other case that they do overlap in time (assuming there index values idx are the timestamps); they are checking that the largest timestamp is less than the minimum timestamp of the previous cluster.

I can't see evidence of the check for temporal continuity of a cluster.


Is there a RandomForest implementation that handles categorical data without encoding in python?

Because sklearn uses the CART algorithm it cannot accept categorical data as-is (as you have pointed out). There is an existing ticket out to change this.
The issue really should be rephrased as supporting categorical splitting on Decision Trees - not Random Forest, as Random Forrest is simply the ensemble method using these decision trees as fitters.
I don't know of a python package that supports this functionality but I do suspect that it would help increase performance because it would avoid a common pitfall of using random forest on hot-encoded data; see One-Hot Encoding is making your Tree-Based Ensembles worse, here’s why?
In the meantime - I use TargetEncoder when I know I need to use Random Forrest. It allows the categorical data to be encoded in a "smart" fashion without needing to use one-hot encoding.


For binary classification, the code for accuracy metric is:
K.mean(K.equal(y_true, K.round(y_pred)))

which suggests that 0.5 is the threshold to distinguish between classes. y_true should of course be 1-hots in this case.
It's a bit different for categorical classification:
K.mean(K.equal(K.argmax(y_true, axis=-1), K.argmax(y_pred, axis=-1)))

which means "how often predictions have maximum in the same spot as true values"
There is also an option for top-k categorical accuracy, which is similar to one above, but calculates how often target class is within the top-k predictions.


How to add numbers to the axes of a graph?

Use xticks. e.g.

x=np.arange(1,16)
y = -60000*(3+np.log(1/x))
plt.plot(x,y,'b')
plt.xticks(x)
plt.show()


Merging information of rows with the same date

As @Emre has pointed out in comments, you need a pandas custom aggregator.

So since you need a string custom  join by /. Create a custom aggregator as

foo = lambda a: "/".join(a)

(or if you need spaces around the join)

foo = lambda a: " / ".join(a)

Then make a pandas groupby as

data_.groupby(by='date').agg({'muscle_time': 'sum',
                              'stretch_time': 'sum',
                              'cardio_time': 'sum',
                              'muscles': foo}).reset_index()

This should get you the aggregated dataframe.


What does a predicted probability really mean, without considering the accuracy of the underlying model?

Accuracy is measured in classification model by comparing the predicted labels to the actual known labels.
The predicted labels are a function of both the predicted probabilities for each class and a predefined threshold(binary classification usually is 0.5)
So if sample A got predict_proba of {0: 0.2, 1: 0.8} it will be labeled as 1(since 0.8 > 0.5).
Accuracy is measure of classification correctness and predict_proba is a direct result of the model underlining function.


Firstly, for understanding the Markov switching models, a nice knowledge of Markov models and the way they work. Most importantly, an idea of time series models and how they work, is very important.

I found this tutorial good enough for getting up to speed with the concept.

This is another tutorial on a similar application of the switching model, which is the regime switching model.

The statsmodels library has a nice support for building the Morkov switching models.

Here is one simple and quick Python tutorial which uses the statsmodels library.


GloVe vector representation homomorphism question

If you're asking if the group homomorphism makes the the process symmetric then no it doesn't directly.  However, they use the fact that they require a group homomorphism to show that $w_{i}^{T} \tilde{w}_k = log(P_{ik})=log(X_{ik}) - log(X_{i})$  This nearly gives us symmetry.  Finally by adding $\tilde{b}_{k}$ into the equation you restore symmetry.

So in short $w_{i}^{T} \tilde{w}_k + b_{i} + \tilde{b}_{k} = log(X_{ik})$ is what ensures symmetry, and the group homomorphism is a tool to get there.

Update:

Some more details
Essentially, what we want is the ability to peform a label switch.  Group homomorphism helps with this process because it perseves a mapping between the $(R, +)$ and $(R, x)$.

$F((w_{i}^{T} - w_{j}^{T})w_{k}^{'})=F(w_{i}^{T}w_{k}^{'}+( - w_{j}^{T}w_{k}^{'})) = F(w_{i}^{T}w_{k}^{'}) \times F(-w_{j}^{T}w_{k}^{'} )= F(w_{i}^{T}) \times F(w_{j}^{T}w_{k}^{'})^{-1} = \frac{F(w_{i}^{T}w_{k}^{'})}{F(w_{j}^{T}w_{k}^{'})}$

The group homomorphism here allows for that to occur.   Therefore we can see that by setting $F(w_{i}^{T}w_{k}^{i}) = \frac{X_{ik}}{X_{i}}$

Now finally we can say that $w_{i}^{T} {w}_k^{'} = log(P_{ik})=log(X_{ik}) - log(X_{i}).$

So as far as your comment, it is the most sensible chocie for their method and of which they buld the core mathematicals to GloVE.  Changing it, I imagine wouldn't be a trivial thing.  I imagine if you did, much of what is derived, including the loss function would change.  But with that said, I imagine there are otherwise to achieve label switching.


predict an array like [1,2,3] increasing

Machine learning is not needed to solve that problem. This deterministic code tests if numbers in an array are increasing:

def strictly_increasing(nums):
    "Deterministically check for only increasing values"
    return all(x2 > x1 for x1, x2 in zip(nums, nums[1:]))

assert strictly_increasing([1, 2, 3])
assert strictly_increasing([2, 4.2, 5.6])
assert not strictly_increasing([1, 1, 1])
assert not strictly_increasing([3, 2, 1])


How to map ground truth to prediction for UNet architecture

It depends on how your UNet architecture is constructed.

In the example image you've shown, the output of the first few layers of convolutions is cropped to 392x392 before it's copied over to the input of the final convolutional layers. Then, it's decreased to 388x388 by convolution without padding. For the example, you're probably best off cropping your image to 392x392, then scaling down to 388x388, as the lost resolution due to the convolution is more analogous to scaling than cropping.

You could also construct the network to output an image of the same dimensionality to avoid this issue. The downside to this approach is that the network will produce some strange/poor results near the boundaries of the image, so you're spending computational power on pixels you will need to disregard anyway.


How to use binary relevance for multi-label text classification?

Just some views/suggestions. After removing the stop words did you stem/lemmatize the text you got?  That would probably reduce the number of unique words in your corpus and brings some forms of words to the same level. But caution while using stemming as it sometimes create noise.
Try postagging and see what can be the important tags you want to keep and eliminate the ones that you feel are giving less relevance to the text.
Did you try to find the some important terms from each document by using tf-idf or chi-square method. They might be helpful to see the relevance of terms for each class/document.
See how far can you reduce the dimensions of the matrix after using the above, if not you have already used them and then apply logistic regression or whatever classifier you wanted to.

On your second question.

I don't think having data from different sources should be a problem since you are trying to create model based on keywords, Hopefully it should be able to see. I am not confident on this part.

I had worked on text classification before with bigger documents and larger corpus with not so fruitful results with various models.

Hope this helps.


What is the best reference for multi-class SVM?

The best resource for multi-classification support vector machines (SVM) is the book "Pattern Recognition and Machine Learning" by Bishop.

The paper "A comparison of methods for multiclass support vector machines"  by Hsu and Lin also covers the topic.


Regime classification pre model training?

Just a couple thoughts:

It looks like these "regimes" could be represented as a latent variable:  you could probably design a bayesian model in which the OLS model depends on the value of this latent variable. This means that the model would still be trained only with the observed features, but would internally predict the value of the regime and this value would determine the parameters of the OLS model.
A more direct approach for this kind of case-by-case setting would be to use decision trees (or random forests), since they can handle independent models in different branches. However I'm not sure how to make decision trees and linear regression work together (or if it's possible at all).


Ordered and unordered categorical features – terminology

The statistical programming language R uses the term "ordered factor," so factor isn't completely safe, though I can't find an example of an ordinal variable being called "factor" without the adjective.

I think ordinal variables are fairly often considered either qualitative or quantitative: they have the quantitative feel from the ordering, but lack mathematical operations.  See the end of this post for a few examples.

But "nominal" seems relatively safe as meaning only unordered.  (I found one contradiction to this, the Bonus Note #2 at https://www.mymarketresearchmethods.com/types-of-data-nominal-ordinal-interval-ratio/, but that's immediately contradicted by having "ordinal" in the next section?)

See also https://en.wikipedia.org/wiki/Level_of_measurement, especially the "Debate" section that lists a few other proposals.  (Chrisman's proposal is nice for including "cyclic" features that are sometimes important in ML but don't fit into most standard libraries without some [unfaithful] encoding.)

A few links to show that the lines get blurred:
https://stats.stackexchange.com/q/159902/232706
https://www.mymarketresearchmethods.com/data-types-in-statistics/
https://stats.stackexchange.com/a/158226/232706
https://web.ma.utexas.edu/users/mks/statmistakes/ordinal.html


This can be done with packages from the dplyr which is part of the tidyverse made by Hadley Wickham. The stringr package (also made by Hadley) is really helpfull in working with vectors of strings. Another package purrr is helpfull for applying functions t lists.

First, lets import the libraries and create data:

library(tidyverse)
library(stringr)

df = tibble(
  medium = c("organic", "(none) > referral > referral > (none) > (none) > referral", "(none)", "organic > referral > referral", "referral > referral > referral > referral", "organic > referral > referral > (none) > referral"),
  conversion = c("0", "0,0,0,0,0,0", "0", "0,1,0", "0,0,1,0", "0,1,0,1,0")
)

You probably have the data as strings while a list represention would be much easier to work with. The following code converts the strings to lists by splitting on ">" or "," (resp. for medium and conversion). Also read upon the %>% which is really handy for working with data frames. After converting the strings to lists the conversion column can be made boolean by mapping the == operator on each row.

df <- df %>%
  mutate(medium = str_split(medium, ">"),
         conversion = str_split(conversion, ","),
         conversion = map(conversion, `==`, "1"))

Removing the items after the first conversion is now a matter of simple indexing. A little magic is done with dplyr::lag and cumsum to get a boolean list that indicates all the steps before and during conversion. Then map is used to get (using the [ operator) all the mediums by boolean indexing with no_conversion_yet.

df <- df %>%
  mutate(no_conversion_yet = map(conversion, function(x) dplyr::lag(cumsum(x) < 1, default=TRUE))) %>%
  mutate(medium = map(no_conversion_yet, medium, `[`))

Filtering the rows that do not have a conversion is now easy. Simple remove all the rows that do not have any TRUE value in the conversion column.

df <- df %>%
  mutate(any_conversion = map_lgl(conversion, any)) %>%
  filter(any_conversion)

Voila, life made easy by the superb packages of Hadley Wickham! (Also check his paper.)


In Mini Batch Gradient Descent what happens to remaining examples

It's an implementation-dependent point but there is no reason that the last few records should be left.
In Keras - It takes the remaining data points as the last step.
Addition of one extra elemtn increases the steps by 1.
Case-I - Data count is divisible by batch_size
epochs = 1
batch_size = 16
history = model.fit(x_train.iloc[:864], y_train[:864], batch_size=batch_size, epochs=epochs)

54/54 [==============================] - 0s 3ms/step

Case-II - Adding an extra data point
epochs = 1
batch_size = 16
history = model.fit(x_train.iloc[:865], y_train[:865], batch_size=batch_size, epochs=epochs)

55/55 [==============================] - 0s 3ms/step -

In your example too, same thing is happening
batch_size = 16
np.arange(0, x_train.shape[0], batch_size)

.....672, 688, 704, 720, 736, 752, 768,832, 848, 864])

When the last slice will happen, it will be a batch of 11 datapoints
len(x_train[864:880])  # Although x_train end at 875

11


Cosine similarity versus dot product as distance metrics

Think geometrically. Cosine similarity only cares about angle difference, while dot product cares about angle and magnitude. If you normalize your data to have the same magnitude, the two are indistinguishable. Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. Note that neither of them is a "distance metric".


Maybe wrong values for precision and recall

If you are calling a higher proportion of cases as Basic with K=5, then this will probably lower your Basic precision, increase your Basic recall, increase your Premium precision, and lower your Premium recall. This is because there is always a tradeoff between precision and recall, and Basic and Premium are opposites in your binary classifier.


Setting input shape for an NLP task in R(Rstudio) using keras 1D convolution layer, when it expects 3 dimensional input (a tensor)

The input layer was producing a 2d tensor, whereas 1d convolutional layer expects a 3d tensor as input. A good explanation of as to why keras expects it and how should you reshape your 2d input to 3d can be found in this answer.

I used Keras R API function k_reshape( 2d_tensor, (list_of_new_dims)) --> k_reshape(input_layer, list(nrow(train_matrix), num_of_feature_vectors, 1).

nrow(train_matrix) - the total number of rows in my matrix (no of samples)

num_of_feature_vectors - total number of columns in matrix (total number of features)

1 - i want a 3d tensor with only 1 element at z axis, therefore z axis is initialised to 1


Geometric and harmonic means in ensembling methods

We can answer this overarching question by exploring a couple sub-questions:

What are the properties of popular averaging formulae?

Geometric:
$ \space $
$ \space $
$ \begin{equation}
\bigg(\displaystyle \prod^N_{i=1} x_{i}\bigg)^{\frac{1}{N}}
\end{equation} $

We can see that any set that contains a zero has a geometric mean of zero.

Also, processing negative numbers is awkward as they can become imaginary.  Even if negative numbers do not result in an imaginary mean (e.g. the set has an odd number of elements), the result is often not what we expect an "average" to be.  For instance, given a set, let's take some averages:

$ \{-1,2,4\} $

Arithmetic mean: $ 1.\overline{6} $

Geometric mean: $ -2 $

The geometric mean is clearly not what we would expect for an "average".  This is because the geometric mean is primarily used for understanding the multiplicative differences in a set, which is useful for averaging data on differing scales.

Let's say we only consider the absolute value of the elements.  Then, given a set below, we still get undesirable results:

$ \{2^0,2^1,...2^{10}\} $

Arithmetic mean: $ 186.\overline{09} $

Geometric mean: $ 32 $

From this example we can see that where the arithmetic mean is linear, the geometric mean is logarithmic.

Harmonic:
$ \space $
$ \space $
$ \begin{equation}
N \cdot \bigg({\displaystyle \sum^N_{i=1} x^{-1}_{i}}\bigg)^{-1}
\end{equation} $

Like the geometric mean, the harmonic mean of a set containing zero is uninformative.  In fact, it is undefined.

A more interesting property of the harmonic mean is that it discriminates against outliers.  For instance, given a set, let's again take some averages:

$ \{ 1e^{-5}, 1e^{-5}, 1e^{10}\}$

Arithmetic mean: $ 3.\overline{3}e^9 $

Harmonic mean: $ 1.5e^{-5} $

An easier way of thinking about the averages is like this:

Where arithmetic is linear, geometric is logarithmic, and harmonic is reciprocal.

Should every learner have an equal representation in the final vote?

If you are using the arithmetic mean, the weak learners are treated equally.  This is generally desired as you do not know which learner is closer to the correct answer.  Using the geometric or harmonic mean would imply that you want to "weight" the set of weak learner outputs according to their respective additive structures.

As a side-note, you can (and probably should) use machine learning algorithms, such as a MLP or SVM, to combine the outputs of your ensemble, which would probably yield superior results to simple averaging.


Split columns by finding "£" and then converting to minvalue, maxvalue

This is more of a general regex question, rather than pandas specific.
I would first create a function that extracts the numbers you need from strings, and then use the pandas.DataFrame.apply function to apply it on the pandas column containing the strings. Here is what I would do:
import re
def parseNumbers(salary_txt):
    return [int(item.replace(',','')) for item in re.findall('£([\d,]+)',salary_txt)]

#testing if this works
testcases = ['£23,000 to £100,000','£34,000','£10000']
for testcase in testcases:
    print testcase,parseNumbers(testcase)

Here, I just used re.findall, which finds all patterns that look like £([\d,]+). This is anything that starts with £ and is followed by an arbitrary sequence of digits and commas. The parenthesis tells python to extract only the bit after the £ sign. The last thing I do is I remove commas, and parse the remaining string into an integer. You could be more elegant about this I guess, but it works.
using this function in pandas
df['salary_list'] = df['salary'].apply(parseNumbers)
df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)
df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)

Checking if this all works:
import pandas
df = pandas.DataFrame(testcases,columns = ['salary'])
df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)
df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)
df

    salary  minsalary   maxsalary
0   £23,000 to £100,000 23000   100000
1   £34,000 34000   34000
2   £10000  10000   10000

The advantages of moving the parsing logic to a separate function is that:

it may be reusable in other code
it is easier to read for others, even if they aren't pandas experts
it's easier to develop and test the parsing functionality in isolation


Keras - understanding ImageDataGenerator dimensions

I think your train_images array should have shape (79, 698, 608, 3). The generator works through each of the first dimensions of those arrays, so is passing a batch of 4d numpy arrays, instead of a batch of 3d numpy arrays.

You can try seeing if that helps, using numpy.squeeze(), like this:

In [1]: import numpy as np

In [2]: a = np.random.randint(0, 10, (2, 1, 10, 10, 3))

In [3]: np.squeeze(a, axis=1).shape
Out[3]: (2, 10, 10, 3)

So be sure to set the axis=1 argument. Then just pass the updated array as you were doing.


Is there an R package which uses neural networks to explicitly model count data?

I skimmed over a paper recently that aims to use Neural Networks as Poisson regression. The method they propose is basically a standard Multi-Layer Perceptron where they use a different loss function, namely:

$$E = -\sum_{n=1}^N[-t_n + y_nlog(t_n)]$$

This is a version without regularization to prevent the overfitting, they use regular weight decay.

They mention that they wrote it in R and Matlab but I don't have a clue if it's available online somewhere, but any neural network package where you can pass your own loss function should suffice.

http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&%20Applic.pdf


Conversion of Interval data to ordinal data

In my opinion etc. is playing key role here. Ordinal data must have the fixed length, which is not in this case as we can go on even further as 1st,2nd,3rd,4th,.. and so on. There is no end to this. You can argue that there must be fixed number of runners(say 50) and hence we have only 1-50 rank. But it is not mentioned in question hence this assumption is not valid.

If no of atheletes participating were mentioned then you're right.


Tools to do Semantic Anotation on 3d CAD Models

I would like to update that I was able to perform annotation on the 3d models by using following tools

Semantic Segmentation Editor
Blender
FreeCAD

Blender and FreeCAD are easy to use, but needs some postprocessing to get the annotations right especially on point cloud data, whereas Semantic Segmentation Editor can be used to directly annotate the point cloud but that stuff is not easy.


How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

For your second question:

  what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0.  In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.


Interested in Mathematical Statistics... where to start from?

When looking for texts to learn advanced topics, I start with a web search for relevant grad courses and textbooks, or background tech/math books like those from Dover.

To wit, Theoretical Statistics by Keener looks relevant:
http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-93838-7

And this:
"Looking for a good Mathematical Statistics self-study book (I'm a physics student and my class & current book are useless to me)"
http://www.reddit.com/r/statistics/comments/1n6o19/looking_for_a_good_mathematical_statistics/


Conventions for dimensions of input and weight matrices in neural networks?

I would not say there is such a convention for it per se (if anyone has anything to comment on this, I would also like to know).
I think to make it clearer how the layer's input x interacts with the weights W, it might better to define the dimensions as the following:

x: (m x n)
W: (n x k)
bias term b: (k)

m remains as the number of examples. n represents number of input features and k represents number of neutrons in the layer.
As we know, we compute the output of the layer y as Wx + b. Therefore, the resulting output matrix will be  (m x k)


To me it looks like your question is more about expert knowledge than ML approach: the main problem is how to build a sensible score based on the available indicators. I have zero knowledge about this domain, but in general this kind of problem requires a lot of back and forth with domain experts in order to correctly specify the task in detail.
However I have one piece of advice which might help (or not, maybe you already know this) for the evaluation of any system you build for this task: assuming that you obtain a predicted score which is meant to represent the future of the company, you could evaluate how well this score matches "the future" as follows:

Split the data at a particular point in time in the past, say 2010
Use only the data before 2010 to train and test a model, i.e. obtain predictions based only on data before 2010
Evaluate how well the predicted scores match the "future" of the companies, i.e. the actual data after 2010. For example a low score should lead to poor indicators, a high score should lead to good indicators (you might need to devise a specific method to correlate the scores with the indicators).


Difference between using RMSE and nDCG to evaluate Recommender Systems

nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1.

RMSE (Root Mean Squared Error) is typically used to evaluate regression problems where the output (a predicted scalar value) is compared with the true scalar value output for a given data point.

So, if you are simply recommending a score (such as recommending a movie rating), then use RMSE. Whereas, if you are recommending a list of items (such as a list of related movies), then use nDCG.


What is the geometrical representation of the y value in a 2-variable perceptron plot?

in the context of binary classification, Y indicate the point at ($x_1,x_2$) belongs to positive(1) or negative(-1)

this piece of code is helpful to understand this

import numpy as np
import random
def generate_data(no_points):
    X = np.zeros(shape=(no_points, 2))
    Y = np.zeros(shape=no_points)
    for ii in range(no_points):
        X[ii][0] = random.randint(1,9)+0.5
        X[ii][1] = random.randint(1,9)+0.5
        Y[ii] = 1 if X[ii][0]+X[ii][1] >= 13 else -1
    return X, Y
X, Y = generate_data(100)
plt.scatter(X[:,0],X[:,1],c=Y)

in this figure, positive(1) class is shown as yellow, and negative(-1) class is shown as purple.


Logistic regression from scratch in Python

For those who care, it turns out that removing the learning rate just makes the gradient happen faster. If alpha not removed, things still converge but incredibly slowly (it takes millions loops to converge)
My theory is that my math was right, but plain gradient descent is computationally expensive.