Combining Neural Network with Reinforcement Learning in a Continuous Space

I might be a bit late for answering but hope it helps!I assume that you are familiar with RL so I will omit lots of details (please if you are still interested comment so I can help you).

Neural Networks and RL: You have two options. The first one is to use a network which you will have as input your feature vectors (states) and output probability of each action. This is called a policy network and you can find a very detailed tutorial with Python Code in order to implement it by A. Karpathy.
Your second option is to use the Q-Network approach. Your input will be again the same but the output will be values of your Q function for each action you have ($Q(a_i)$). You will use the Q-learning  equation $Q(s_t,a_t)=Q(s_t,a_t) +\alpha[r_t+\gamma \max _a'Q(s_{t+1},a')-Q(s_t,a_t)]$. The details of the implementation can be found in the paper of V. Mnih. Also do not worry about the delayed rewards as the discount factor $\gamma will "help" your agent to be affected by the future rewards.

In order to calculate your states, I would suggest you to create a simulation of the environment and a step function. You don't mention what kind of game you are dealing with but the general idea is that the step function will take as input your current state and current action and output the next state and reward (don't mind about the continuous space of your features as you can discretize it - you can use Kalman filters or other models to have a better state estimation as well).

My advice would be to choose your approach (Policy net or Q-net) and read the blog or the paper, create a simulation of the environment and a step function for your game. You can find tons of implementations of the Deep Q-net although I would suggest you to start with a very simple network so you don't get in trouble by tuning the Deep net.


Finding out the scale when using MinMaxScaler()

This is because you were importing a class as name 'scalar' and modifying it by assigning value obtained after transforming. Just change

from sklearn import preprocessing
scaler = preprocessing.MinMaxScaler()
scaled = scaler.fit_transform(finance_features)
print scaler.scale_

This should work.


Loss function for optimising precision & recall / sensitivity & specificity?

The comments raise two important points:

you can't directly optimize for sensitivity or specificity
you should save the model at the best epoch and use that one

Below I explain something that is not directly related, but might be interesting for you:
You can add the weights to the loss of some examples. For example, you can increase the weight of positive examples which means that the model will focus more on correctly classifying positive examples at the potential expense of negative examples. This is especially useful if your data is suffering from class imbalance.


What should you do with attributes that predictive in an interaction?

Should we modify the features in such a way that we would use the difference between the two teams, instead of two values?

Yes, you should definitely design your features "to help" the ML algorithm, this is called feature engineering. Be careful though that there can be many different options for representing the contrast in "average possession": you can think of the difference indeed, but also a ratio or many other arithmetic combinations.

  On the other hand some algorithms might be good at finding this interactions?

Most ML handle take into account interactions between features to some extent, but it depends what kind of interaction and which ML algorithm.


How to Predict the future values of time horizon with Keras?

This answer goes a little bit in a different direction, but I hope it still answers your question. It uses the idea of a rolling forecast/prediction.

Because you use the word horizon, I will assume you mean that you would like to predict 10 days into the future at a given time step. There are a few ways of doing this. With this kind of time-series problem, it is common to make the assumption that only a certain history will influence the next few time steps (neglecting seasonal effects).

Example in words:

So in your case, you might use e.g. the previous 60 days, and predict the next 10.
Taking your 100 rows of data as an example, this means you can actually make (100 - 60 - 9) = 31 predictions, each prediction of 10 time steps ahead (we will need these 31 predictive_blocks later). From 100 rows we lose the first 60 to fit the first model. Of the remaining 40 rows of data, we can predict 10 steps ahead (rows 61-70), then we shift the whole thing one row further and repeat. The last prediction of 10 future points would be for rows 91-100. After that we cannot predict 10 steps anymore, so we stop - and this is why we have to subtract that extra 9. [There are of course ways to continue making prediction, as to use all the data]

Example with a thousand words:

Let me paint the picture; to help explain the idea of a shifting window prediction.

For each train set (e.g. from t=0 to t=5 in red - train set 1), you want to predict the following H time steps (corresponding to t=6 in orange - test set 1). In this, your horizon is simply one i.e. H=1.

From what I understand, you would like to predict the next 10 days, meaning you need H=10.

In order to try this with your example, I think you will need to make two changes.

Change #1

The shape of your train and test sets will need to match the new horizon. Each sample of your model input (the x_train and x_test can stay the same as before. However, each sample in your test set will have to contain the next H=10 values of the label, not just a single value.

Here is a rough example of how you might do this:

# Define our horizon
H = 10

# Create data split, using values from my example above
window_size = 60
num_pred_blocks = 31    # as computed above

# Loop over the train and test samples to create the sliding window sets
x_train = []
y_train = []
for i in range(num_pred_blocks):
    x_train_block = x_train[i:(i + window_size)]    # 31 blocks of 60 * num-columns
    x_train.append(x_train_block)
    y_train_block = y_train[(i + window_size):(i + window_size + H)]    # 31 blocks of 10 * 1
    y_train.append(y_train_block)

Because you are doing out-of-sample testing, your predictions are already interesting to look analyse. Once this runs, you can then create the equivalent test datasets with the new data you mentioned.

Without knowing your data too well, I don't know if your should be predicting the y-values of the same row as the input, or of the following row. Additionally, depending on your data, you could be including the past values of y in each of the x_train blocks. In this case you'd simply swap x for the whole table i.e. data[cols], where new_cols = ['Demand'] + cols.

Change #2

You will need to make the model reflect this horizon, by forcing it to output H values.

Here is an example of how to specify the model:

# Define our horizon
H = 10

# Create the model using the parameterised horizon
fit1 = Sequential ()
fit1.add(LSTM(output_dim = 4, activation='tanh', input_shape =(4, 1)))
fit1.add(Dense(output_dim=30, activation='sigmoid')
fit1.add(Dense(output_dim=H))    # our horizon is produced!

Note:
In your model specification, you don't need to add the final linear Activation, as the preceding Dense layer by default includes a linear activation. See the excellent documentation here.

This is a big topic and there are many things that you could try out. I agree with the comments on your question, that you will need a lot more data to allow an RNN to make a meaning representation of the model.

If you are not just doing this to learn about LSTMs etc., another practical approach might be to look into simpler time-series models such as an ARIMA model (do not be intimidated by the complicated name - it is much simpler than an LSTM). Such models can be constructed quite easily with Python, using the statsmodels package, which has a nice implementation.


avoiding premature convergence with neural networks (EA's)

You can take a look at dropout, it will get you out of the local minima you are stuck in.


Should you use random state or random seed in machine learning models?

Your intuition is correct. You can set the random_state or seed for a few reasons.:

For repeatability, if you want to publish your results, share with
other colleagues
if you are tuning the model, in an experiment you usually want to
keep all variables constant except the one(s) you are tuning.

I usually set the random_state variable, not the random seed while tuning or developing, as this is a more direct approach. When you go to production, you should remove the random_state and or random_seed settings, or set to None, then do some cross validation. This will give you more realistic results from your model.


Your BPE vocabulary is quite small given how the strings you want to segment look like. The biggest problem here is that BPE expects tokenized sentences with tokens separated by spaces. The BPE model is not aware of JS syntax and you feed it with JS code, so there is no wonder that it does not learn anything syntactically plausible.

You can interpret the BPE file a log of what subwords got merged and you can see that the longest string you get is something like:

m').valueOf
H:mm').valueOf

I would recommend some preprocessing, perhaps some JS lexer, so it respects the syntax and perhaps bigger vocabulary.


When to use linear or logistic regression?

Linear Regression is used for predicting continuous variables.

Logistic Regression is used for predicting variables which has only limited values.

Let me quote a nice example which can help you make the difference between the both:

  For instance, if X contains the area in square feet of houses, and Y
  contains the corresponding sale price of those houses, you could use
  linear regression to predict selling price as a function of house
  size. While the possible selling price may not actually be any, there
  are so many possible values that a linear regression model would be
  chosen.

  If, instead, you wanted to predict, based on size, whether a house
  would sell for more than 200K, you would use logistic regression. The
  possible outputs are either Yes, the house will sell for more than
  $200K, or No, the house will not.


Am I using GridSearch correctly or do I need to use all data for cross validation?

Your procedure is, from what I can tell, correct. You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters. Using all of the training data and the hyper parameters found in cross validation, you are then evaluating your final model on the test set.

Indeed, the outliers and the size of your dataset are the most probable causes for the large differences between validation and test. Basically, if the majority of these outlier observations fall in the test set after your initial splitting of your data it is highly likely that your test set scores will be larger than your validation scores. On the other hand, if these outliers are in your training set, you would expect the opposite to be true. This is because, regardless of what data your have in your training set for any random partitioning, these outlier observations are unlikely to be predicted well regardless of what you do, and because the size of your dataset is so small, these outliers will heavily impact your error estimate (especially with RMSE that is incredibly sensitive to outliers due to the squaring).

Either way, report the test set scores only as your final estimates if you want to be honest.

In scenarios in which the variance is large in my test set scores (usually due to a small data set and a large number of outliers), I would highly recommend repeating your entire model building procedure as you described but with a different seed each time (i.e. split your entire dataset into train/test with a different seed each time). Do this until you run out of patience or the variance in your estimate of model performance is small enough. Maybe form bootstrapped confidence intervals from these repetitions to give you information on how much your model's performance varies.


Creating a Object Detection model from scratch using Keras

Before I answer your question, let me tell you this, You can go on and train a model from scratch, but you will definitely end up using one of the object detection architectures, be it Mask R-CNN, Faster R-CNN, Yolo or SSD. Your problem is a simplified version of what these architectures are trying to solve. These are generic object detectors that some of which supports more 1k classes. You have a single class detection problem.

Now back to your question.

  For training an object detection model, should the image be kept as an input and the coordinates as the output of the model? Should there be Convolution layers for feature extraction and then FC layers for learning the features for producing 4 outputs ( coordinates of the bounding box )?
  No, it is not that simple. Training a FCN to output 4 values as Bounding boxes wont work.

All object detectors mentioned earlier are based on assumptions, Faster R-CNN for example generates proposals (regions) its assuption is that these regions are very likely to contain an object, then does an additional step by classifying which class each region contains (you only have one class) and finally refining the output. The most important thing in Faster R-CNN is region proposal network, which iterates through the final convolution layer in a sliding window fashion, generating proposals in the different aspect ratios, for example 1:1, 1:2 and 2:1.

Why am I saying all this? Because I want you to understand, that the problem is not as easy as you think it is.

Further Reading:

https://medium.com/@tanaykarmarkar/region-proposal-network-rpn-backbone-of-faster-r-cnn-4a744a38d7f9
https://medium.com/diaryofawannapreneur/yolo-you-only-look-once-for-object-detection-explained-6f80ea7aaa1e


Learn and sample from a generative model with 2000 boolean features

You may try an Apriori algorithm or more advanced FPGrowth algorithm. It will give you a set of association rules, which, I think, you required:

  e.g. if feature 10 is on, feature 14, 15, and 16 are usually on


An input to a U-Net are normally images with 3 (or 4) channels, as images have a red, green, and blue channel. It is however technically possibly to use other images (or arrays for that matter) as the input which a larger number of channels. If you wanted to use the three different methods that you mention, you would first have to apply all three methods to original image, which would then return an array with 1 or more channels. You then have to stack them in the channels direction, which would result in an array of shape HxWxC, where C is the total number of channels. The total number of channels in the input then depends on the image preprocessing methods you use and the number of channels of the original image.


Welcome to the site, Martin! That's a pretty broad question, so you're probably going to get a variety of answers. Here's my take.

Data Science is an interdisciplinary field generally thought to combine classical statistics, machine learning, and computer science (again, this depends on who you ask, but other might include business intelligence here, and possible information visualization or knowledge discovery as well; for example, the wikipedia article on data science). A good data scientist is also skilled at picking up on the domain-specific characteristics of the domain in which they working, as well. For example, a data scientist working on analytics for hospital records is much more effective if they have a background in Biomedical Informatics.
There are many options here, depending on the type of analytics you're interested in. Andrew Ng's coursera course is the first resource mentioned by most, and rightly so. If you're interested in machine learning, that's a great starting place. If you want an in-depth exploration of the mathematics involved, Tibshirani's The Elements of Statistical Learning is excellent, but fairly advanced text. There are many online courses available on coursera in addition to Ng's, but you should select them with a mind for the type of analytics you want to focus on, and/or the domain in which you plan on working.
Kaggle. Start with kaggle, if you want to dive in on some real-world analytics problems. Depending on your level of expertise, it might be good to start of simpler, though. Project Euler is a great resource for one-off practice problems that I still use as warm-up work.
Again, this probably depends on the domain you wish to work in. However, I know Coursera offers a data science certificate, if you complete a series of data science-related courses. This is probably a good place to start.

Good luck! If you have any other specific questions, feel free to ask me in the comments, and I'll do my best to help!


I believe that the command you are looking for is outreg2. For other output options check out this page.


Multiple Regression, Classification and Boundary Poins

I think you want a clustering algorithm rather than regression. You will have a decision boundary between clusters of data points which will determine whether that particular point e.g. 5,3 belongs to group (cluster) A or group (cluster) B
Fit your clustering model in x1 vs x2 feature space. Take the image below, we have x1 and x2 as the x&y axis. And then the black lines are the decision boundaries, essentially the model.

You can of course just cluster between 2 groups also, as in your case.
You can checkout different clustering algorithms here


How to add noise to supervised (binary-classifier)?

I'm not sure why/where you want to apply the noise, but if you want to add some Gaussian noise to a variable, you can do this:

import numpy as np

target_dims = your_target.shape
noise = np.random.rand(target_dims)
noisy_target = your_target + noise

Now use the noisy_target as input to your model.


Is window based sequencing a good idea to obtain more training data for LSTMs?

Splitting a time-series into analysis windows, usually with overlap, is quite common practice. In anomaly detection, but also in classification or forecasting. It works great as long as your anomalies can be detected by analyzing such windows independently.
In such a setup, the length of the analysis window becomes a critical hyper-parameter - and will be data/task-dependent.
To get an overall anomaly score you can use any standard statistics to merge scores for individual windows. A simple mean would be the first thing to try, but there are more approaches.


Predict next series of numbers based on previous data

You could use LSTMs with a sequence to sequence model perhaps (using e.g. Keras).

For example:
x lines are input, the next line is output.

Just an idea, your question is not that clear.


Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)

Say you have a dataframe of your data in this format:
df = pd.DataFrame({
    'name': ['Bob', 'Bob', 'Bob', 'Tim', 'Tim', 'Tim'],
    'fruit': ['Oranges', 'Apples', 'Bananas', 'Oranges', 'Apples', 'Bananas'],
    'num': [5, 10, 12, 3, 20, 5]
})

You can perform a groupby on fruit and aggregate the sum of the num field. After that you can apply a function on your aggregated dataframe that leverages values from your non-aggregated data like so:
df_agg = df.groupby('fruit').sum()
df_agg.reset_index(inplace=True)

def desc(x):
    d = []
    for idx, row in df.loc[df['fruit']==x].sort_values(['num'], ascending=False).iterrows():
        d.append(f"{row['name']}({row['num']})")
    return ', '.join(d)

df_agg['desc_str'] = df_agg['fruit'].apply(lambda x: desc(x))

This gives you the following df_agg that you are looking for:
    fruit   num desc_str
0   Apples  30  Tim(20), Bob(10)
1   Bananas 17  Bob(12), Tim(5)
2   Oranges 8   Bob(5), Tim(3)


Does it make sense that datetime encodes one-hot-vector like one-hot-encoding or something else like

If you are trying to predict future values then it doesn't make sense to treat them as categorical features. There is nothing you will learn that can predict future data, since you won't see those times again.

The same holds even if you are trying to predict unseen past data but each time appears only once or a small number of times.

Instead the time values should determine the order of your training data. This way you can avoid leaking future data, and models with state can learn features from the sequential nature of the data.

However, in some cases you may want to extract additional features from the time values. Here are some examples:

Time since last interesting event
Number of interesting events in last time window of size n
Time of day (morning / afternoon / etc)
Day of week
Holiday
Season

Let's make up an example. Here is a dataset of times that users visited a website:

time               user
2017-11-01 00:00   Alice
2017-11-01 00:00   Bob
2017-11-02 00:00   Chris
2017-11-03 00:00   Alice
2017-11-04 00:00   Alice
2017-11-04 00:00   Bob
2017-11-07 00:00   Chris
2017-11-10 00:00   Alice

And here is the same dataset with additional features we have added:

time               user    last_visit   weekend?   time_of_day
2017-11-01 16:22   Alice   N/A          No         afternoon
2017-11-01 11:13   Bob     N/A          No         morning
2017-11-02 20:35   Chris   N/A          No         evening
2017-11-03 16:07   Alice   2 days       No         afternoon
2017-11-04 17:20   Alice   1 day        Yes        afternoon
2017-11-04 10:44   Bob     3 days       Yes        morning
2017-11-07 08:06   Chris   5 days       No         morning
2017-11-10 17:11   Alice   6 days       No         afternoon

If we are trying to predict when a certain user might visit next, then these features might help us a lot. For example we might decide that Alice is more likely to visit in the afternoon, or Bob is unlikely to visit two days in a row.


How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

1) It seems that your data are unbalanced, you should look into that. Common techniques include oversampling the minority class, but you might have a bigger problem here.

2) It is unclear that you have enough information for what you are trying to achieve (type of device and location doesn't seems to be enough).

3) Based on the two preceding remarks, you have to acknoledge it is unlikely you will be able to get the date of an event. Would a human be able to guess that is a good question to ask yourself if you try to apply ML on a problem. If the answer is no, then a ML solution is unlikely to work. You can do statistical analysises to try to know if there are devices types or locations that are more susceptible of loss, but you won't be able to get exact time and location of a loss.


LeNet is a family (LeNet-1, LeNet-4, LeNet-5) of convolutional neural network designed by Yann LeCun et al. The name is a play on his name and the French language, where "le" means "the", hence LeNet means "the network". I believe it was originally devised to recognize handwritten numbers on checks (cheques). LeNet is only one early instance of a convolutional neural network; many others exist today.


Counting the transition in a dataframe overtime

For a small dataset try this:
# change type of date column so that it is possible to find min and max correctly
df['date'] = pd.to_datetime(df['date'])

# put value of the oldest resord and newest record side by side
df_min_max_grouped = pd.merge(
    df.loc[df.groupby('Id')['date'].idxmin()],
    df.loc[df.groupby('Id')['date'].idxmax()],
    on='Id',
    suffixes=('_old', '_new')
)

# group by old value and new value and find counts
results = df_min_max_grouped.groupby(['value_old', 'value_new']).size()

results

Output:

value_old  value_new
Active     Notactive      1
           Superactive    1
Notactive  Superactive    1
dtype: int64

The approach I chose for this answer:

Convert date values to be of type datetime, Correct ordering will not work with string dates.
First find the oldest and the newest records for any Id,
This happens in : df.loc[df.groupby("Id")['date'].idxmin()] for oldest and df.loc[df.groupby("Id")['date'].idxmax()] and newest records.
Join the oldest and newest records on the basis of Id, and then
Group by old value and new value combination to get a count in each group.

I found that resetting index of the resulting Series produces nice output:
results.reset_index()

value_old
value_new
0

0
Active
Notactive
1

1
Active
Superactive
1

2
Notactive
Superactive
1

The combinations not listed here have 0 records against them. I hope you find this helpful!


How to expand abbrevations in text during preprocessing?

Do it in two steps: replace the ones with 's' first, then do the rest of them.


How do I plot linear regression results if input and target have different sizes?

You can plot your residuals. I am sorry I can't write it in Python but in R you would do it like this:

plot(my.model$residuals)

This will give you a scatterplot that you WANT to look like a null plot. Any curve or pattern in the points means that your model is not a good fit.

If you are interested in assessing each variable's prediction power I recommend (because you only have six) doing 1 regression model on each variable by itself and then plotting the residuals of those models, again looking for a null plot. You can also look at the correlation of each predictor variable vs your response variable. Those with higher correlation will be better predictors.


I'm not sure whether I understand the question, but if you want to include var1(t) along with the other lagged features var1(t-1), var2(t-2), etc. you can simply add var1(t) as an additional feature.
This means that your matrix with previous shape

(number of samples, number of timesteps, old number of features)

now becomes

(number of samples, number of timesteps, new number of features = old number of features + 1)

Judging from your code this can be simply done by adding var1(t) as a new column to your train_X matrix, like in the first three columns of your first code snippet.

However I'm not sure why you want to do this. Forecasting at time t usually assumes that, once your model is ready to work on real data, you won't have any data at t, only from the past. If you build your model based on data at time t without being able to retrieve these information once in production, then you've got a problem! Also, you write that var1 and var2 are correlated, hence the LSTM is already picking up this correlation as you are feeding past values of these variables together.


I assume that you are referring to the fact that training a decision tree (e.g. with the C4.5 algorithm or a variant) involves the selection of the features used as conditions in the nodes. Typically at each iteration the algorithm selects the most discriminative feature by ranking all the features using for instance their information gain.

This is indeed very close to a feature selection process:

ranking features by their informative power w.r.t the label is a very common way to do feature selection
the decision tree can end up selecting only a subset of the available features, exactly as if feature selection had been applied.

SVM works very differently and (in general) doesn't have a similar process of selecting particular features. But it's true that in a loose sense any classifier does some kind of feature selection: it measures how relevant each feature is in order to predict the correct class.

However it's important to understand that a real feature selection process consists in removing features before training, i.e. the learning algorithm doesn't get the option to use the features which have been removed. This differs a lot from the above cases because:

if the feature selection removes too many features, the model might not be able to perform as well as with all the original features
if the feature selection correctly removes redundant and/or useless information, the model is likely to perform better than if all all the features were provided. This is because the algorithm can get confused by the abundance of information, either because it's harder to find what is relevant among many options or because it doesn't have enough instances to properly assess the relevance of all the features.


Generally, this is done using spark-rowsimilarity algorithm - it is a class of content based recommendation. However, the actual process of doing this is quite simple. Here are the steps:

For each movie, convert your categorical variables into columns. For lets say that actor_1 has Brad Pitt, Daniel Craig, and Vin Diesel for different movies. This will become three columns with a 1 denoting which movies have each actor. Your movie matrix will look something like:

Movie Name, Has_Brad_Pitt, Has_Daniel_Craig, Has_Vin_Diesel, ...
MI-6      ,     1        ,       0         ,     0         , ...
Fast&Furios,    0        ,       0         ,     1         , ...
Casino Royale,  0        ,       1         ,     0         , ...

Now, to find similarity score of movies, you can just compute the cross product of the two vectors. Higher the value, more they are similar.

This can be done by the spark-rowsimilarity algorithm in one shot. You may have to do some work in encoding categorical variables.


What are some methods for clustering individuals into distinct groups based on Features?

Sounds like you are on the right track.  There are many ways to attack this problem.  If you want to visualize the clustering, it would help to reduce the data down to two components.  This could be done via PCA or manifold learning if you want to go non-linear (http://scikit-learn.org/stable/modules/manifold.html).

In terms of clustering, there are many different methods to do this.  Here are some comparisons of different methods: http://scikit-learn.org/stable/modules/clustering.html


How to handle image classification network where output classes can be subsets of one another?

It depends what you're trying to achieve.

In general, there are things that are vehicles but aren't vans, trucks, or sedans, so your two proposed schemes are not equivalent even if you were somehow able to obtain a 100% accurate classifier.

In general, the way to answer questions like this is to try it both ways and see which works better.


out of memory error when consrtucting 2d list from 2 numpy arrays

I would recommend breaking the problem down a little bit to reduce the memory usage at any one time.

The first part of your main function gets the IDs using getIDList. That seems fine, so leave it there.

I would then break that list down into smaller chunks, calling get3Dmatrix on each chunk in turn. Altering your code, it might look something like this:

# Get number of entries in ID list
N = len(ID_list)

# break it down into a number of chunks e.g. 4, based on your progress bar
import numpy as np    # should already be imported

N = len(ID_list)
num_chunks = 4           # you can play with this number, making it larger until you don't get emmory errors
chunks = np.linspace(0, N, num_chunks)

for i in range(len(chunks) - 1):
    this_sublist = ID_list[chunks[i] : chunks[i + 1]]
    sub_data_set = get3Dmatrix(this_sublist)

    # At this point, either save this sub_data_set, or try appending it to another list toi make one final numpy matrix at the end before saving

...

print("Begin saving in numpy file")
np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
print("%s time takes in seconds" % (time.time() - start_time))

Even from the traceback you added, it is hard to say where exactly in your code that is happening.

Roughly looking at the dimensions you mention, it also doesn't seem plausible the a 16Gb machine is running out of memory - so I must not completely understand just how many images/patches are being saved.


Association rule mining and Orange3 package Python

In Orange 3, AssociationRulesSparseInducer is removed.

The documentation of Orange-Associate may help you on your task.

References

https://github.com/biolab/orange3-associate/issues/4

https://stackoverflow.com/questions/39667272/orange-data-mining-version-3-3-python-association-rules


Correct order of operations involved into Dropout

The usual processing for your suggested layers:

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

would be (reading left to right)

dense output -> relu -> apply dropout mask -> apply "inverse dropout" divide by p

The precise combination may vary depending upon optimisations, and can in theory be changed a little without affecting the result (it doesn't matter to the end result numerically if we scale then mask or mask then scale for instance). However when dealing with vectorised optimisations (like those found in TensorFlow and Theano), it is normal to accept a percentage of "wasted" processing, and just have that naive left-to-right processing happen. It is often harder to parallelise decision branches than to simply process all items, even repeated multiplying by and adding zeroes for a significant fraction of each array.

There is no "normalization" implied by activation functions, so this is not a concern. From your comments, it seems you are worried that dividing by p could mean that the output of a neuron that would be between 0 and 1 (because you were using sigmoid activation for example) would now be between 0 and 1/p - i.e. larger. That is true. Is this a problem? No it is not, and in fact it is required. The impact of the larger values is fully compensated for by the weights learned in the connections between layers. If you used "vanilla" dropout then the weights would be correspondingly larger, but you would need to scale the outputs down (new range would be 0 to p) during testing/prediction.


Why do we need activation function (like ReLU) after an affine layer?

This affine operation $y = W^{\top} x + b$ has already add nonlinearity to the system given that $b \neq 0$.

This is not considered a non-linearity in the context of data science. Different disciplines define linearity sometimes in subtly different ways. Critically, the $+b$ performs identically in terms of fitting to data, as extending $x$ with a new dimension, always $1$, and moving the values of $b$ into the weights $W$. This simpler multiplication is clearly linear.

Also importantly, the affine transformations form a group such that any two affine transformations combined are just another affine transformation with different parameters. Without a non-linearity in a hidden layer, a 2-layer neural network would be the same as a single layer one, and not able to learn whole classes on non-linear relations.

No matter how many affine transformations you apply to inputs for instance, you will not be able to approximate the XOR function, or any significant portion of $y=\text{sin}(x)$


dealing with HuggingFace's model's tokens

For your first question, you can check if the tokenizer covers a certain string with the following:
text = 'today is a good day '
ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))
ids2string(tokenizer(text)['input_ids'])
> <s>today is a good day </s>

If emoji is not included in the tokenizer creation, the tokenizer will replace it with the unknown special token. You can access that with tokenizer.special_tokens_map['unk_token']. You can drop or keep them, shouldn't make much of a difference.
Alternatively, if you're going to fine-tune, you can add your own tokens to the existing tokenizer with tokenizer.add_special_tokens. However, in this case the embeddings for that token will be random. And you need to train them.


I think the comment is true for any kind of network where the neuron has a linear transformation function and there is no activation. Convolution is just a special case of linear transformation.

Basically, if your first layer outputs linear combinations of your features, and the second layer outputs linear combinations of the first layer outputs, then the second layers outputs is a linear combination of the initial features.

In the case of convolution, the interesting thing is that the convolution product is associative. So that if you apply two kernels consecutively $K_2 * (K_1*X) $ it is equivalent to $(K_2 * K_1)*X $ so that you can calculate the combined operation $ K_1 * K_2 $ as a unique kernel easily.


How to compute time-lagged correlation between two variables with many examples at each time t?

An application of a specific correlation formula depends on the data-type (continuous or rank data etc). Given that your data is continuous, you can apply  Karl Pearson formula. However, if you're interested  interested in cause and effect relationship, you may prefer to use simple regression  model.


What's the difference between setting negative label as 0 or -1 in binary classification?

This will depend on the algorithm/code that you are using.
You are right that this is often not an important difference but that doesn't mean it can't create errors if you are using code which expects one or the other without being aware of it.
You should equally keep track of data types such as integers vs floats which will similarly produce different outcomes in lots of data science tools such as in Python or R.
Since lots of classifiers implemented in packages like sklearn will specify whether they are for continuous/discrete outcome spaces (classification vs regression), you shouldn't have to worry about this.
To illustrate this for sklearn:

source: https://scikit-learn.org/stable/_static/ml_map.png


How to visualise GIST features of an image

Given that code is trivial for both (GIST + Network and Raw Pixel + Network), you can try three approaches for a given project.

GIST + Dense layers (GIST is not space-distributed)
Raw Pixels + CNN + Dense Layers
Raw pixels + CNN + Dense + input layer 2 (GIST) + Dense

For some projects, GIST can help since it is an abstract feature that CNN might or might not learn.
EDIT: This paper compares GIST and CNN
Regarding:

Is there a way to visualize the extracted features on top of the image?

This can be done with an attention layer in approach 3 (CNN + GIST).
CNN provides spacial distribution (Required for visualization) and dense layer that merges CNN's output with GIST can be used with an attention layer.
Paper for visualization


It depends on couple of things but one of the important ones is how big is your set.

Note that the difference is 0.01 in R squared, so if dataset is small taking 20 samples do determine class might be costly, on the other hand if you can afford it 10 samples might be to little do ensure class membership.

Depending on your next usage I would weigh in data size.


A/B testing: How to calculate p-value on post test segments?

I recently wrote about this in a blog post.
Given this is a rate evaluation metric, you will want to use the z-test. The basic steps are (more details in the blog post)

calculate the pooled standard error of your pairwise comparison
calculate z-statistic by normalizing the delta or lift by the standard error
look up the cdf value of the normalized delta
p-value = 1-cdf(z)
given that this is also an A/B/n test, you should also want to apply multiple testing correction using the bonferroni procedure or the benjamini hochberg procedure, when evaluating significance


This was the idea behind paranoid linux which began as fiction and became a real project that did not reach fruition:

"Paranoid Linux is an operating system that assumes that its operator is under assault from the government (it was intended for use by Chinese and Syrian dissidents), and it does everything it can to keep your communications and documents a secret. It even throws up a bunch of "chaff" communications that are supposed to disguise the fact that you're doing anything covert. So while you're receiving a political message one character at a time, ParanoidLinux is pretending to surf the Web and fill in questionnaires and flirt in chat-rooms. Meanwhile, one in every five hundred characters you receive is your real message, a needle buried in a huge haystack.
~Cory Doctorow (Little Brother, 2008)

When those words were written, ParanoidLinux was just a fiction. It is our goal to make this a reality. The project officially started on May 14th, and has been growing ever since. We welcome your ideas, contributions, designs, or code. You can find us on freenode's irc server in the #paranoidlinux channel. Hope to see you there!"

I think the most modern equivalent is Pirate Linux along with the TOR Project.

BTW, Cory Doctorow's book "Little Brother" is kind of a fun read if you are a geek.  If you are reading this then you probably ARE a geek :-)

The reality is that it is very difficult to hide one's true intent when high quality clustering, classification and anomaly/novelty detection is used.  I work with guys who have done this type of behavior detection for some very high profile three letter agencies and it is very very easy to detect nefarious behavior if you have enough data.  So much so that undercover good guys sometimes clearly show up in the data sets.


Build neural network to calculate points for a board game

Theoretically, you could take many pictures and map these pictures to the score of each player. However, I would advise against it. First, you would need plenty of pictures and it might be infeasible to cover all possible game scenarios. Second, game scoring is discrete whereas a traditional neural net would approach it as a regression. This means that your outputs will likely be close but rarely perfectly on the money (and you will need to round up). You could however try to predict who is currently winning and turn this into a classification problem.
Now back to predicting points. Since there already exists a complex point-scoring system based on the board, it would be rather silly to try to let a neural network figure it out on its own. Instead, I would suggest that you implement the point-scoring system and have the neural network translate a picture of the board into a data representation of that board. Your system would do something like this:
def scoring_system(board):
    // Insert logic here
    return points

def your_neural_network(image):
    // Your code
    return board

points = scoring_system(your_neural_network(image))

So, instead of having this be a regression problem, you have multiple problems to solve:

object detection (where are the pieces)
object classification (what type of piece is it?)
image segmentation (where is the board, what territories are there)
Etc.

Now some of these bits could be solved with traditional computer vision, but neural networks can be well-equipped as well.


Morphological Analyzer for Hindi and English in Python

One option is the ploygot package which can perform morphological analysis in English and Hindi.
from polyglot.text import Word

word = Word("Independently", language="en")
print(word, w.morphemes)


The R function is “round”. The default is to round to the nearest integer, which is what you want. The function takes vector inputs, so you can apply it to your column in one line instead of using a “for” loop.

Then you’ll tack in on to your original data in any number of ways, such as “expanded_data <- data.frame(original_data, round(column))”.


Look at this blog. Mainly you have saved operations as a part of your computational graph. So after you load your model, you can restore the session and call the predict operation that you created for training and validating your data, and run it on the new data hy feeding into the feed_dict. Make sure it is in the same format and same shape as your training data. I am adding a small snippet of code, maybe that will help.

checkpoint_file=tf.train.latest_checkpoint(checkpoint_directory)
graph=tf.Graph()

 with graph.as_default():
     session_conf = tf.ConfigProto(allow_safe_placement=True, log_device_placement =False)
     sess = tf.Session(config = session_conf)
     with sess.as_default():
          saver =     tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
          saver.restore(sess,checkpoint_file)
          input = graph.get_operation_by_name("input").outputs[0]
          prediction=graph.get_operation_by_name("prediction").outputs[0]
          #newdata=put your data here
          print sess.run(prediction,feed_dict={input:newdata})

To use operation_by_name, you will have to name the operations and variables in the original model. Secondly, in the feed dict you will have to give all the parameters needed to perform the operation(prediction here), so if you are using dropout you will have to give that too. There might be more simpler ways of doing this. This is the way I did it.


Efficient database model for storing data indexed by n-grams

See Lucene NGramTokenizer

Are you sure you can't just use lucene or similar indexing techniques?

Inverted indexes will store the n-gram only once, then just the document ids that contain the ngram; they don't store this as highly redundant raw text.

As for finding ngrams that contain your query sub-n-gram, I would build an index on the observed ngrams, e.g. using a second lucene index, or any other substring index such as a trie or suffix tree. If your data is dynamic, probably lucene is a reasonable choice, using phrase queries to find your n-grams.


Ingestion of periodic REST API Calls into Hadoop

You can use Kafka to ingest data into HDFS or any other cloud storage like S3 or Google storage.
 And you can use Gobblin to schedule your kafka consumer to write into HDFS.

Kafka Producers => Kafka Consumer => Gobblin (Monthly/weekly/Daily/hourly/minutes) => HDFS

Hope this will work for you


Are there any actual, commercial uses of GANs already?

I have seen a showcase of a commercial company that used a version of GAN-based super resolution to reconstruct a more detailed image of the seabed out of low resolution depth gauge profiles.

I could imagine that you could use that to visualize a seabed for VR presentations (I'm not sure whether that is done). However I very much doubt the added value over more traditional interpolation, when applied for deciding whether a fairway should be dredged.


I'm surprised that there is no other tool, especially since usually when there are word tokenizers available there are also sentence tokenizers (the two tasks are often performed at the same time).
In case you need to train your own tokenizer, you could use the Universal Dependencies Hindi corpus.


How do you search a high dimensional for the global maxima using as few samples as possible?

I think you probably should use some randomized optimization algorithm such as randomized hill climbing or genetic algorithm. These are probably more suited for such a problem


I would not suggest GMM at this point as the distribution of points in the space is not well-shaped enough. Even if you want to use it it's better to look at your data in PC space (i.e. using PCA). My suggestion would be:

1) Think of your features. What are they? Are you going to use these gr-chromacity as features? If yes you should know that kernel methods work better on this as the features are highly nonlinear. The image show that you need a feature mapping anyway.

2) It seems you have already thought of kernel methods as you put SVM as a tag. you can use it for classification. Might work better than GMM here. Also think of probabilistic graphical models as they have been used intensively for image segmentation and your images are structured enough (a football field has its fixed position in the image anyway).

3) If you have raw labeled dataset, I'd recommend to think of smarter features for segmentation. in gr-chromacity you already loose some information about colors which is the most important thing for you here. I would recommend taking the position of pixels into account as well. Then a PCA on the new data may reveal some more linearly separated classes.


How to methodologically show that a given 'time-series/sequential' data is not really sequential?

TL;DR:
There is no off-the-shelf formula to determine if you should go for time-series or not. Hypothesis testing is the most formal approach, AFAIK.

Just to find a common ground, let's first distinguish between using timestamp vs. dealing with time-series.

Timestamp data itself may contain lots of information that could be quite useful in many domains. For instance, hour, day of week and month are widely used in
demand forecasting applications. However, using the timestamp in this fashion does not make it a time-series approach. Here, time plays only a role of just another feature.

When dealing with time-series, one would expect to benefit from temporal dependencies present in the data (as well as from spatial dependencies). BTW, most of the methods are limited to evenly spaced time-series. If you have timestamps of some arbitrary events, you would have to do pre-precessing/aggregation before being able to treat is as a time-series.

Should you treat your data as a time-series?

I would suggest considering two basic hypotheses (i.e., try proving the opposite if you believe your non-time-series model is just right):

(1) there is a time-domain relationship between observations;
(2) it is indeed relevant to your problem.

First, you may try to investigate (1) by checking if there is a time-series model that explains well your observations. As suggested in comments, autocorrelation test is a way to get a clue if there is a linear relationship between observations through time.

You may wish to go beyond linearity, and try to build a non-linear predictor by using a time delay (TDNN) or a recurrent (RNN) neural model.

Before deep diving in pure statistics for a formal answer, I would suggest a simple test: train your best predictor model on a real time-series and also do it on randomly shuffled (broken) time-series. If there is no significant difference in performance - there probably nothing in time-domain but noise or some sort of complex chaos.

For instance, a time-series of the chaotic process like x(t+1) = 4 * (1-x(t))*x(t) looks quite meaningless, but a very simple non-linear model will be able to predict it perfectly. On the other hand, good luck trying to predict an randn() generator, which is also a chaotic time-series :-).

If you cannot reject (1), i.e. there are strong evidence of time-domain correlations, then you should verify if (2) is true. Here it could be as easy/hard as it gets, depending on your domain. Speaking more formal, I'd go for trying to find out if the estimates of

P[Y(t)| X(t), X(t-1)..., X(t-n)] # predictions of the target variable given past observations

are as good as

P[Y(t)| X(t)].  # predictions of the target variable given only current observation

You could try to answer it empirically by comparing the performance of a time-series vs. non-time-series models in a sufficient number of trials, or by using some specific knowledge of your domain.

E.g., P[cat | pixels of current picture] = P[cat | pixels of the current picture, pixels of the previous picture], which may be true or not, depending whether the pictures were taken in the same or different households.


What is the name of this similarity distance metric?

This is the weighted Jaccard Index.

https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance

This is different from the regular Jaccard Index (Similarity).


What are some easy to learn machine-learning applications?

I would recommend to start with some MOOC on machine learning. For example Andrew Ng's course at coursera.

You should also take a look at Orange application. It has a graphical interface and probably it is easier to understand some ML techniques using it.


How to implement conditional probability distribution on set-valued Random variables

Keeping it to discrete space, the joint probability of an event can be interpreted as the fraction of the size of this subset compared to the whole state space.
In case of events, being sets themselves, the definition remains the same.
$$P(X_i,Y_j) = \frac{|\{(x,y) \in  (X_i,Y_j)|\}} {|\Omega|} $$
$|\Omega|$ is just the set of all combinations of single events (x,y).
Let us now check your formula with the following example:
$$X_0 = Y_1 = \{0\}, ~~ X_1 = Y_0 = \{1\}$$
Then the total space space $X \times Y$,
$$ \Omega  = \{(0,0),(0,1),(1,0),(1,1)\} $$
And for example:
$$ P(X_0,Y_0) = P( (0,0) ) = \frac{1}{4} $$
Your formula however implies,
$$ | (X_i \cap Y_i) \cap  (X_j \cap Y_j) | = 0 $$
as $X_i \cap Y_i$ is empty per contruction in the example.


Use liblinear on big data for semantic analysis

Note that there is an early version of LIBLINEAR ported to Apache Spark. See mailing list comments for some early details, and the project site.


Neo4j and Spark GraphX are meant for solving problem at different level and they are complimentary to each other.

They can be connected by Neo4j's Mazerunner extension:

  Mazerunner is a Neo4j unmanaged extension and distributed graph
  processing platform that extends Neo4j to do big data graph processing
  jobs while persisting the results back to Neo4j.

  Mazerunner uses a message broker to distribute graph processing jobs
  to Apache Spark's GraphX module. When an agent job is dispatched, a
  subgraph is exported from Neo4j and written to Apache Hadoop HDFS.

  After Neo4j exports a subgraph to HDFS, a separate Mazerunner service
  for Spark is notified to begin processing that data. The Mazerunner
  service will then start a distributed graph processing algorithm using
  Scala and Spark's GraphX module. The GraphX algorithm is serialized
  and dispatched to Apache Spark for processing.

  Once the Apache Spark job completes, the results are written back to
  HDFS as a Key-Value list of property updates to be applied back to
  Neo4j.

  Neo4j is then notified that a property update list is available from
  Apache Spark on HDFS. Neo4j batch imports the results and applies the
  updates back to the original graph.

Check out this tutorial to get an idea on how to combine the two:
http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html


The stem function doesn't allow several digits as leaves, as far as I can tell. So you're left with implementing it from scratch. It's not too difficult but OMG it took me such a long time to get it right... I always get confused with R weird types conversions. Anyway, here it is:

d<-strtoi(unlist(strsplit('1717 1719 1645 3739 3024 3664 3830 2991 2430 2730 3469 5086 2119 3021 3292 2844 3426 2067 3215 2767 3124 2573 2840 2449 2584 1505 1390 1645 2497 3466 3228 3192',split=' ')))
dd<-ldply(d,function(x) {data.frame(stem=x %/% 1000,leaf=sprintf("%03d",x %% 1000),stringsAsFactors=FALSE)})
for (i in 0:9) { cat(paste(i,"|","")); l<-dd[dd$stem==i,]$leaf; if (length(l)>0) {cat(sort(l))}; cat("\n") }

Result:

0 |
1 | 390 505 645 645 717 719
2 | 067 119 430 449 497 573 584 730 767 840 844 991
3 | 021 024 124 192 215 228 292 426 466 469 664 739 830
4 |
5 | 086
6 |
7 |
8 |
9 |

Remark: there's no way to do this with the stem function simply because the purpose of a stem and leaves plot is not to display every exact value: the plot is only intended to give a "big picture" view of the distribution of the values. What matters for that is how many leaves there are for every stem, and it's enough to give a single digit since precision doesn't matter. From this point of view the stem function does the job:

stem(d)

Or variant:

stem(d,scale=0.5)


Why has Hadoop failed to become popular?

I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system.

Python or R are handy at the beginning when you just need to try out things but when it comes to putting things in production, Hadoop is the way to go. It does not directly provide any data scientist tools what it does provide is the base where data would be stored, processed and can be used to apply machine learning algorithms using Spark.

So to summarize, I see Hadoop being used in the following contexts

For Data Storage in Production Systems
Hadoop YARN as a cluster manager for tools like Spark, Flink
If someone still wants to code in R/Python/SAS - Hadoop being used as backend.

Hope this helps.


Is it normal when BLEU score on filtered data by length is greater than BLEU score on whole data

The original BLEU scores 25.9 and 25.7 are very close, there might not even be any significant difference. It's totally possible that model B performs better than model A on the filtered data only by chance. It's also possible that model B actually performs better than model A on shorter sentences. And finally it's worth noting that BLEU score is based on the number of n-grams in common, so it's likely to be affected by the length of sentences independently from the model being tested.

Conclusion: based on the information provided, this difference seems perfectly reasonable.


what happens when a decision tree can't be split into further unit values?

Well, sometimes the features simply do not provide enough information to get 100% accuracy (like in your case), even with a model a flexible as the Decision Tree.
The Decision Tree works by trying to split the data using condition statements (e.g. A < 1), but how does it choose which condition statement is best? Well, it does this by measuring the "purity" of the split (conditional statements split the data in two, so we call it a "split"). The purity is simply how well the statement split up the target variable (so, a good split would be [2, 0], which basically translates to "If I did this split, I would get two $D = 0$ instances, and zero $D = 1$ instances.", and a bad split would be [1, 1], saying "With this split I would get one $D = 0$ instance and one $D = 1$ instance"). If the model finds that no further splits can reduce the purity, it stops.
If you want to look into it further, there are a couple of measures for measuring purity (or rather, impurity), the main ones being gini and entropy.


If you set epsilon decay to 0.999 you will need
$$
\epsilon_{max} \cdot \epsilon_{decay}^x = \epsilon_{min} \\
1 \cdot 0.999^x = 0.01 \\
x \approx  4603
$$
4603 episodes to reach minimum epsilon. After 91 episodes you will reach
$$
\epsilon_{current} = \epsilon_{max} \cdot \epsilon_{decay}^{episodes} = 1 \cdot 0.999^{91} \approx 0.913
$$
which is exactly what you can see in your plot. It's not a problem but remember that this model still makes over 91% moves randomly.
Average reward should not decrease over time. It can mean a few things for example error in dqn algorithm or too high learning rate in your model. The best way to debug is to start with as simple environment as possible and let your model learn to play it and only then increase the difficulty.


Should I oversample my validation data to get better F1 score and PRC?

what you encounter are real-world problems rarely taught in classes.

For training, I would test SKLearn's class_weight = "balanced" or class_weight={0:0.995, 1:0.005}. It's a very robust technique.
For testing, you can't fiddle with the class_weight. It's meant to simulate the real-world data.
Make sure you don't overfit. E.G. Decision Tree max_leaf_node 5-10 OR (not both) max_depth 3-5.

Your results aren't that bad. A precision 1 of 0.03 is "low", but you only had 36 labels. Roughly speaking, the model labeled 1,000 as 1 (100/0.03 * 36). i.e. 97% false alarm. But with a good recall, you got most of the labels.
Putting it differently: in a dataset of around 8,000 you got 1,000 labeled as 1. That's around 12%. Out of those 1,000 there are about 30 real 1's. You get most of them (around 75%).
Imagine this was Churn Prediction. You don't want to lose customers. Without ML, you might send all your 8,000 customers a discount of 10%. That's expensive. With ML, you would only send 1,000 customers a discount and get most of them wanting to leave. That's a strong improvement over no model.
The same argument applies to many other cases such as Predictive Maintenance.


First of all, feature maps are the output of the convolution after an activation function (e.g. ReLU or sigmoid) is applied, not the matrix that the image is convolved with. Usually, this is called a filter.

The magical thing about CNNs is that we don't know what the filters should look like for any given problem. The CNN works out what each filter should look like automatically. This is done through the backpropagation procedure. Without getting heavy into the math of it all, essentially every time a training example (or a batch of examples) goes through the network, the values inside each filter get updated by some small amount. This small amount is determined using the derivatives of a loss function. As each step of the training procedure is completed, the values inside each filter (if all goes well!) slowly converge towards a value that minimises the loss function, thereby producing the best quality predictions.

For more information on the math, I encourage you to read the chapter on backprop in the free online book Neural Networks and Deep Learning, available here.

A more simple explanation is also provided here


How to evaluate clusters base on an attribute of the dataset?

Use external cluster evaluation.

In particular, use ARI and NMI.


How to visualize multivariate regression results

I personally like dotcharts of standardized regression coefficients, possibly with standard error bars to denote uncertainty. Make sure to standardize coefficients (and SEs!) appropriately so they "mean" something to your non-quantitative audience: "As you see, an increase of 1 unit in Z is associated with an increase of 0.3 units in X."

In R (without standardization):

set.seed(1)
foo <- data.frame(X=rnorm(30),Y=rnorm(30),Z=rnorm(30))
model <- lm(X~Y+Z,foo)

coefs <- coefficients(model)
std.errs <- summary(model)$coefficients[,2]

dotchart(coefs,pch=19,xlim=range(c(coefs+std.errs,coefs-std.errs)))
lines(rbind(coefs+std.errs,coefs-std.errs,NA),rbind(1:3,1:3,NA))
abline(v=0,lty=2)


Why does my model produce too good to be true output?

Assuming that these results are obtained on a valid test set with no data leakage, these results don't show overfitting because overfitting would cause great performance on the training set but significantly lower perfomance on the test set.
Make sure that your instances between the training and test set are truly distinct: there might be some data leakage, for example if there are duplicate records for the same patient.
Another potential flaw is the gold standard labeling: if the patient has this T2DM  in their medical record, it means that they are already diagnosed right? And since the features are also based on the medical record, it's likely that this medical record contains direct indications about this diagnosis (for example in the drugs prescribed to the patient). There are two interpretations about this:

either the task is purposefully defined by this T2DM label, and in this case you can just enjoy the great performance but it's not technically about detecting diabetes in general.
or the goal is to detect patients with diabetes including the ones who are not diagnosed yet, but then it's likely that your gold standard is incorrect for this task.

[edit following updated question]
Your update clarifies which exact task you're targeting, and it corresponds exactly to my first interpretation above: given that your goal is to predict which instances satisfy this T2DM criterion and that the features contain direct indications about it:

I think you're right to keep these features, if a very useful information is available it would be absurd not to exploit it (assuming it's also available in the same form in any future dataset you plan to use, of course)
The very high performance you obtain makes perfect sense for this task, it's not a bug. It simply happens that the task is easy, so the system is able to predict the label very well.

However this also means that you could do even better without any ML: currently the ML method gives you around 99% F-score because it doesn't perfectly represent the criterion used for the gold. But since the gold standard label is based entirely on the features, the most direct way to "predict" the label is to apply the criterion. There's no point using a complex ML method to predict an information that you can obtain from the same data more accurately with a deterministic method.


This is the output when I run your code. It seems to be doing fine.

[  1.58876117   3.17458055  11.11863105]

Try to run it again? Perhaps you changed some parameters accidentally before posting it?


how can i detect medicine name and info(use and contents) by using medicine wrappers

For data, you can search on IEEE Dataport , Kaggle.

For detecting medicine names and other info trained deep learning models like CNN or you can also perform fine-tuning from the existing model.


What you presented is the  typical proof of convergence of perceptron proof indeed is independent of $\mu$. Hence the conclusion is right.

Typically $\theta^*x$ represents a hyperplane that perfectly separate the two classes.

The formula $k \le \frac{\mu^2 R^2 \|\theta^*\|^2}{\gamma^2}$ doesn't make sense as it implies that if you set $\mu$ to be small, then $k$ is arbitarily close to $0$. It is saying that with small learning rate, it converges immediately. You might want to look at the termination condition for your perceptron algorithm carefully.


Is it possible to make a CS:GO Machine Learning AI?

The answer is yes:

Example of a neural network outplaying human players in DOTA.  I haven't been able to find much regarding what kind of neural network but here is what is on the OpenAI website.  If you're a beginner you can learn some basic architecture and design principles in neural networks using Python and Keras (a neural network library).


Multi-output regression problem with Keras

Accuracy is a metric for classification, not regression.

$$Accuracy = \frac{\text{Correct classification}}{\text{Number of classifications}}$$

So when you use accuracy for regression only the values where actual_label == predicted_label are evaluated as true are counted as correct classifications. That will happen quite rarely when you are doing regression, resulting in an accuracy that is close to zero.

Instead you should use something like mean absolute error or mean squared error as validation metrics for regression.


How to learn from time series with multiple values for each time points

I had such a problem earlier and that what I did is to
separate the date to year, month and day.
I used logistic regression.
For my side, it was helpful and the test percentage was 77%.


How to test the quality of a word embedding?

One way to test your embedding is see how often your model agrees with the common consensus of how other embeddings complete word analogies. A collection of established word embedding analogies are here.


It looks like you're using Keras, so this answer is going to refer to the Keras embedding layer.

The error message is pretty clear - Keras embedding layers don't accept strings as inputs. The embedding layer requires an array of positive integers and can only be used as the first layer of a model. The embedding layer requires that you encode each word as an integer so you might have to use the Keras tokenizer.

Source: Embedding Layers


Denoising Autoenoders with Variable Length Input

Recurrent Neural Networks can deal with variable length data. You might want to have a look at:

Andrej Karpathy: The Unreasonable Effectiveness of Recurrent Neural Networks.
Christopher Olah: Understanding LSTM Networks.
Hochreiter, Schmidthuber: Long short-term memory.

Another idea (which I have not tested so far and  just came to my mind) is using a histogram approach: You could probably make fixed-size windows, get the data from those windows and make it discrete (e.g. vector quantization, k-means). After that, you can make a histogram of how often those vectors appeared.

You could also use HMMs for recognition of variable length data.

Transformations (e.g. Fourier transform from the time domain in the frequency domain) might also come in handy.


Model parameters & Hyper parameters of Neural Network & their tuning in training & validation stage

The parameters of a neural network are typically the weights of the connections. In this case, these parameters are learned during the training stage. So, the algorithm itself (and the input data) tunes these parameters.

The hyper parameters are typically the learning rate, the batch size or the number of epochs. The are so called "hyper" because they influence how your parameters will be learned. You optimize these hyper parameters as you want (depends on your possibilities): grid search, random search, by hand, using visualisations... The validation stage help you to both know if your parameters have been learned enough and know if your hyper parameters are good.

If you want to know more about hyper parameters and parameters in general in machine learning, look for "deep learning versus shallow learning".


Hierarchical clustering with the consensus matrix as similarity matrix

To address your two questions:

Agglomerative clustering requires a distance metric, but you can compute this from your consensus-similarity matrix. The most basic way, is to do this:
distance_matrix = 1 / similarity matrix
Although, they may explicitly state in the paper what function they use for this transformation.

I think this is just to say that the matrix is symmetric. The x-axis of the heatmap will be n=0,1,2,3,4 and the y-axis of the heatmap will be n=0,1,2,3,4. This is the same procedure as for a correlation matrix. Just keep your matrix as is, and it will keep that order.


Algorithm or function or model that encourages clustered classfication?

Why not create another set of classes which corresponds to the groups you want? This works if you care more about groups. Later on you can subsequently classify within groups. Another idea would be to devise a loss function which penalize less within group misclassification and more outside group classification but this involves going into implementation details of your NN. As an idea it would involve writing something like softmax multiplied with misclassification cost matrix


Optimum way to train a keras nn by a dataset

I would recommend running a script before even using Keras just to clean up your data into a more standard format such as .h5. You can then bin this data into batches with a given index in the key id. Then writing the DataGenerator in Keras becomes extremely easy.

The way I do it is to have the following type of keys:

dataset_identifier_ix

where dataset is either train, validation or test, depending on the set you wish to use. Identifier in your case will be either image and label. Lastly, ix is the batch index. Inside each of these batches you have $n$ instances with which to train your model.

If this is the structure of your .h5, then the DataGenerator code can be like

class DataGenerator(keras.utils.Sequence):

    'Generates data for Keras'
    def __init__(self, dataset, filename, X_identifier, Y_identifier,
                 batch_size, percent_data_use = 1, shuffle=True):
        super(DataGenerator,self).__init__()

        with h5py.File(filename+'.h5', 'r') as hf:
            keys = list(hf.keys())
            # Get dimensions of the input space
            temp = [i for i in keys if dataset+'_'+X_identifier in i]
            x_dims = hf[temp[0]].shape[1::]
            num_files = len(temp)
            # Get dimensions of the output space
            temp = [i for i in keys if dataset+'_'+Y_identifier in i]
            y_dims = hf[temp[0]].shape[1::]

        # The batch size
        self.batch_size = batch_size
        self.num_files = num_files
        # Assumes the file ids are always from 0 to num_files
        self.file_ids = list(range(int(percent_data_use*num_files)))
        # Calculate the number of batches
        self.num_batches = int(percent_data_use*num_files)

        self.filename = filename
        self.dataset = dataset
        self.X_identifier = X_identifier
        self.Y_identifier = Y_identifier

        # Dimensions of the input and the output
        self.x_dims = x_dims
        self.y_dims = y_dims
        self.input_shape = (int(x_dims[0]*sampling_percent),) + x_dims[1:]

        self.sampling_percent = sampling_percent
        self.shuffle = shuffle
        if shuffle is True: self.on_epoch_end()

    def __len__(self):
        """Number of batch in the Sequence.

        Returns
            The number of batches in the Sequence.
        """
        return self.num_batches

    def __getitem__(self, index):
        """Gets batch at position `index`.
        Arguments
            index: position of the batch in the Sequence.
        Returns
            A batch
        """
        X = np.zeros((self.batch_size,) + self.x_dims)
        Y = np.zeros((self.batch_size,) + self.y_dims)

        with h5py.File(self.filename+'.h5', 'r') as hf:

            # Input and output identifiers
            x_id = self.dataset + '_' + \
                    self.X_identifier + '_' + str(self.file_ids[index])
            y_id = self.dataset + '_' + \
                    self.Y_identifier + '_' + str(self.file_ids[index])

            X = np.asarray(hf[x_id])
            Y = np.asarray(hf[y_id])

        if len(X) == 0:
            return None

        ## Restructure X, and Y for use in the Keras network
        return X, Y

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        if self.shuffle == True:
            np.random.shuffle(self.file_ids)


Right order for Data preparation in Machine Learning

Data processing is a state of the art. You need your gut or intuition to be able make a good model. You have to perform data cleaning first. Garbage IN = Garbage OUT. If you don't clean data your model will be a garbage itself. Data Scientist spend 20%-30% of the time in cleaning data. There is no rule for cleaning data. But you could follow few guidelines for Data cleaning.

Remove Duplicate values:- You need to be an artist to recognise the duplicate data. You could drop the duplicate records and just keep one record out of all duplicate records. pandas.DataFrame.drop_duplicates will perform data cleaning for you.
Data integrity:- you need to ensure whether the data is accurate. If few fields of Age column contains Male or Female that actually don't make any sense. Or say Age contains 200 years for a person who in actual is 20 years old. You could say its outlier detection as well. But it doesn't necessarily be an outlier.
Null Values (Data Imputation):-
You could use sklearn library or pandas library for this. Both comes with its own features.
Tip: Use latest KNNImputer. Imputation for completing missing values using k-Nearest Neighbors. It gives far better results. Reference

PERFORM SPLIT NOW :-

To avoid Data Leaks this has to be done. Standardising data before the split means that your training data contains information about your test data.

Column Standardisation: It is required to scale the data. By doing this, you transform your data such that each column have a mean equal to zero and Variance of value 1. This is implemented by default in many algorithms. This will help your model to execute faster and eliminate problems related to scaling. (Every column is of same scale and model does't give preference to one column which has high values.)
Encode categorical features.

NOTE:- Splitting is performed before Feature Scaling (Standardisation). StandardScaler is fit_transformed on training data and only transformed on test data.


Cost functions penalizing certain types of misclassification more heavily in tensorflow

You are talking about having different missclassification costs.

Classification consists of outputing a continuous score which is then thresholded to decide the discrete classification. The threshold (ex. probability>0.5 leads to C) is chosen to satisfy your apetite on sensitivty/specificity.

The training algorithm only cares about your continuous score. This means that you can make it easy for yourself by doing simple binary classification (typically with softmax_cross_entropy_with_logits-loss) and create your vague class B after training is done by thresholding say [0,0.4) is A, [0.4,0.6] is B and (0.6,1] is C.

To achieve exactly what you're saying, penalizing certain errors more, then you want to use the weighted_cross_entropy_with_logits. This throws out the interpretation of your outputted score as a probability, and you will need to work more to get a reasonable threshold afterwards so this is not recommended as a first step as it will probably add more complexity than results.

I also recommend reading the wikipedia article on roc analysis


ARIMA, Exponential Smoothing and others indeed require evenly spaced sample points. As you write, you could bucketize your data (say into days), but as you also write, you would lose information. In addition, you may end up with missing values, so you would need to impute, since ARIMA is not very good at handling missing values.

One alternative, as you again write, is to feed time dummies into a regression framework. I personally do not really like categorical dummies, because this implies a sharp cutoff between neighboring categories. This is usually not very natural. So I would rather look at periodic splines with different periodicities. This approach has the advantage of dealing with your uneven sampling and also with missing values.

Be very careful about interpreting $R^2$. In-sample fit is notoriously misleading as a measure of out-of-sample forecast accuracy (see here). I would argue that this disconnect between in-sample fit and out-of-sample forecast accuracy also means that there is no connection between in-sample fit and how well a model "understood" the data, even if your interest lies not in forecasting, but only in modeling per se. My philosophy is that if you can't forecast a time series well, you haven't understood it in any meaningful sense.

Finally, don't overdo the modeling. Just from eyeballing your data, it is obvious that something happened in June, on one day in August and in September/October. I suggest you first find out what this something was and include this in your model, e.g., as explanatory variables (which you can include in ARIMAX if you want to). What happened there is obviously not seasonality.


LIME visualization outputs padded regions as important - Mel-spectrogram (audio analysis)

Feeding the neural network 75% irrelevant data (zero padding) seems less-than-ideal, both in terms of computational inefficiency and because it makes your model potentially vulnerable to this data.

I would consider a smaller model with 128x128 input input format (ex: MobileNet). You can either upscale the mel-spectrograms you have now from 128x64 -> 128x128 or extract windows that are 128 long (potentially adjusting FFT/hop size to make the window cover same area in time).


How to give a higher importance to certain features in a (k-means) clustering model?

You cannot really use k-means clustering if your data contains categorical variables since k-means uses Euclidian distance which will not make a lot of sense with categorical variables. Check out the answers to this similar question.

You can use the following rules for performing clustering with k-means or one of its derivates:

If your data contains only metric variables:

Scale the data and use k-means (R) (Python).

If your data contains only categorical variables:

Use k-modes (R) (Python).

If your data contains categorical and metric variables:

Scale the metric variables and use k-prototypes (R) (Python).


I want to build a deep learning model to predict the next job title when
  a current title is given. Are there any ways that I could achieve this
  using some deep learning model?

I think that approaching this problem in the classification way (input:- current job embedding, output:- getting next job title as a class) can somewhat be time-consuming (not necessarily hard) to train.

The reason behind this:-

Assuming that each of the job-title pair is unique in your dataset,
given the volume of your dataset, there can be many classes, so much that even deep learning models may take time(for processing, optimization, etc.) to train.
Also as there can be many to many relationships in the dataset, hence using the classic approach might not be a good option.

  what kind of model?

As per your given examples, the kind of model that we need here is one, that can learn the hierarchy of the job titles from the text itself, except for some cases where the sample denotes that the individual of that job title has changed their domain of work(this can be attributed as noise).

  Can we use any of the text generation models for this scenario?

A deep learning model such as seq2seq(https://www.geeksforgeeks.org/seq2seq-model-in-machine-learning/) or noisy autoencoders(https://towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763). I know that there is not much content of 'noisy autoencoders for nlp' on the internet, but we can always try. These models can learn the complex relationships between the words of the job title and the job title itself.

I would recommend noisy-autoencoders as I assume (a strong one) that only some if not all corresponding words in the current and next job title are different, and this is the purpose of noisy autoencoders:-

They can accommodate noise very well(many to one and one to many
relationships between current and next job title)
They keep most of the content in input and output is the same, which I think you are gonna need because of the strong assumption I mentioned above.

I hope it helps, thanks!!


What are differentiable modules used in deep learning

"Differentiable" means that you can compute the derivative of the operations in the module, and therefore you can compute the gradients of the loss function with respect to the module parameters (i.e. use backpropagation).

This is normally a requirement for operations involved in neural network computations.

Note: you can use non-differentiable operations as part of a computational graph, but you won't be able to backpropagate gradients through them, and therefore any learnable parameters involved in operations prior to the non-differentiable one would not be able to learn.


Mean across every several rows in pandas

You can try like below with shifting the frame,

df['shift1'] = df.var1.shift(1)
df['shift2'] = df.var1.shift(2)
df['avg'] = df[['var1','shift1','shift2']].mean(axis=1)
df.iloc[range(2,n,3)]


What is the best Big-Data framework for stream processing?

You might also want to take a look at Apache Flink. It is a new contender to Apache Spark as it can do both, realtime and batch analysis. It also claims to be faster than Apache Spark, but I think mostly in depends on your usecase.

Good thing is: Apache Flink project has some Storm compatibility layer in its development version. In this way you can take your existing Storm project and simply run it inside Flink. For me it was at least 2 times faster than Storm in a very simple topology. So give it a try! :)


Determine what data, does certain column have in common

A good place to start would be to look at the Spearman's Rank Correlation between your dependent variable (academic level) and your individual independent variables (other columns). This should give you a basic indication of whether any of your columns are correlated with academic level.  This should be straightforward to implement in both Excel and SPSS.

If you wanted to go a step further you could set the problem up as one of Multinomial Logistic Regression. This would allow you to build a model to directly attempt to predict academic level from your other variables. I'm certain there's probably some way to do this in Excel, but SPSS can definitely handle it.


How would you - on-the-fly - prevent a neural network from overfitting using a Keras callback?

Sounds like you're just looking for EarlyStopping, which will stop training when validation loss does not improve for N epochs. It's the same as Iter in catboost.


Spectral Networks and Deep Locally Connected Networks on Graphs

k is a fixed parameter indicating the input layer and i,j refer to the index of the matrix F , used to traverse throught the matrix.


Let's start by creating some fake dataset.

software = sample(c("Windows","Linux","Mac"), n=100, replace=T)
salary = runif(n=100,min=1,max=100)
test = data.frame(software, salary)

This should create a dataframe test that will look like somewhat like:

    software    salary
1    Windows 96.697217
2      Linux 29.770905
3    Windows 94.249612
4        Mac 71.188701
5      Linux 94.028326
6      Linux  7.482632
7        Mac 98.841689
8        Mac 81.152623
9    Windows 54.073761
10   Windows  1.707829

EDIT based on comment Note that if the data does not already exist in the above format, it can be changed to this format. Let's take a data frame provided in the original question and lets assume the dataframe is called raw_test.

    windows sql excel salary
1     yes  no   yes    100
2      no  yes  yes    200
3     yes  no   yes    300
4     yes  no    no    400
5      no  no   yes    500

Now, using the melt function/ method from the reshape package in R, first create the dataframe test (that will be used for final plotting) as follows:

# use melt to convert from wide to long format
test = melt(raw_test,id.vars=c("salary"))
# subset to only select where value is "yes"
test = subset(test, value == 'yes')
# replace column name from "variable" to "software"
names(test)[2] = "software"

Now, you will get a datframe test that looks like:

  salary software value
1     100  windows   yes
3     300  windows   yes
4     400  windows   yes
7     200      sql   yes
11    100    excel   yes
12    200    excel   yes
13    300    excel   yes
15    500    excel   yes

Having created the dataset. We will now generate the plot.

First, create the bar plot on the left based on the counts of software that represents usage rate.

p1 <- ggplot(test, aes(factor(software))) + geom_bar() + coord_flip()

Next, create the boxplot on the right.

p2 <- ggplot(test, aes(factor(software), salary)) + geom_boxplot() + coord_flip()

Finally, place both these plots next to each other.

require('gridExtra')
grid.arrange(p1,p2,nrow=1)

This should create a plot like:


Firstly, i want to notice that adaboost is a ensemble of stumps, and each stump is added to the ensemble sequentially, trying to compensate the errors of the existing ensemble.

Saying that:

In every step you have a new weighted dataset, and the Adaboost algorithm tries to fit a stump that splits the new dataset best. So yes, it is possible, and it does happen, that a single feature is used more than once (but the splitting value calculated by the stump should be different in order to reduce the ensemble's error.)
Stumps' creation is a continuous process, they are calculated in
every step for the new weighted dataset.


The short answer is both.

Filter pruning

One approach is to remove filters that are "less contributing" to the overall learning every epoch or so. Defining "less contributing", that's the interesting bit about this research. For example it could be based on the running mean average of each filter, and remove filters of CNNs that have smaller mean. Some other research out there,

Using L1 norm to prune filters paper
This article list several other approaches (https://jacobgil.github.io/deeplearning/pruning-deep-learning)

Dense layer pruning

Obviously the parameter bottleneck of pretty much all CNNs happens at the first dense layer connected to the output of the last convolution layer. Therefore, doing proper pruning on this layer will be quite effective.

I couldn't find any good papers that introduces parameter compression for dense layers. But I'll update my solution if I come across any.

Knowledge distillation

Another really cool approach is known as knowledge distillation. Knowledge distillation is when you train a smaller network to mimic a larger network. One popular paper is found here.


How to merge columns, value count them and then plot the results?

It can be done like:

import pandas as pd
melted = pd.melt(df1, id_vars=["index"], var_name="Var", value_name="Score").dropna()
table=pd.crosstab(index=melted['index'], columns=melted['Score'])
%matplotlib inline
table.plot.bar() #for simple axes subplots


Estimating Predictive Uncertainty for unlabeled data

See Authors comment in the thread here

So by default the dropout is only used in training time. Then in test time the dropout probability is set to 0. But there is a easy way to enable it so that it stays 'on' even during the test time.
    import keras

inputs = keras.Input(shape=(10,))
x = keras.layers.Dense(3)(inputs)
outputs = keras.layers.Dropout(0.5)(x, training=True)

model = keras.Model(inputs, outputs)

By setting training=True it remains active in during test time. This can be called Monte Carlo dropout or more popularly dropout at inference time. I hope it helps.


Can't get Keras functional model to work: "could not broadcast input array..."

Fixed it, sorry. I just needed to say lambda x: x[:, 0, :], so it included the sample index.


First of all, visualize the missing values :

I am using Python 2.7

import pandas
import numpy
from pandas import DataFrame
import seaborn as sns

df = DataFrame({'A' : [0,1,numpy.nan, 5 ,6],'B':[30,numpy.nan,numpy.nan,8,10]})
df

     A  B
0   0.0 30.0
1   1.0 NaN
2   NaN NaN
3   5.0 8.0
4   6.0 10.0
sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap='BuPu')

If u wanna drop rows containing missing values,

df.dropna(axis=0)

        A   B
    0   0.0 30.0
    3   5.0 8.0
    4   6.0 10.0

If u wanna drop columns containing missing values,

df.dropna(axis=1)

If you wanna fill NaN with a value:

df.fillna(0)

  A B
0   0.0 30.0
1   1.0 0.0
2   0.0 0.0
3   5.0 8.0
4   6.0 10.0

You can do a ffill/pad, bfill/backfill. Hit shit tab and expand the documentation if you are using jupyter.

For more info refer : https://pandas.pydata.org/pandas-docs/stable/missing_data.html

I hope this answers your question.


It is bounded by unity, but your first vector does not encode a probability mass function, since 0.5 + 0.7 > 1.0. If the 0.7 had been 0.5 or the 0.5 a 0.3, the distance would have been 1.0 since the distributions are maximally separated, having no overlap.


How to use an existing model as in input into a new model

This depends a  bit on the used case but you could also create the 2nd model independent and then use a third model to combine them. This has the advantage that there should be no leakage between the two models (if variables are independent) and you can merge them in a very controlled way.

What is unclear for me is what advantage you would have from taken input from a model that can not be used in the same context. (offline vs real time)

If you want to use them combined and Model A as input for B you would need to guess what the output for Model A is.


Latent loss in variational autoencoder drowns generative loss

I don't like the reduce_sum version of the kl-loss because it depends on the size of your latent vector. My advise is to use the mean instead.
Moreover it is a notorious fact that training a VAE with the kl loss is difficult. You may need to progressively increase the contribution of the kl loss in your total loss. Add a weight w_kl that will control the contribution :
Loss = recons_loss + w_kl * kl_loss
You start with w_kl=0 and progressively increase it every epoch (or batch) to 1. This is a classic trick.
Your learning rate seems good, maybe you can try a little higher (4e-4).
If you don't like the tricks, the Wasserstein auto-encoder may be your friend.


Can cosine similarity be applied to multidimensional matrices?

Don't hack. Do the math instead.

You could reshape your matrix into a vector, then use cosine.

But whether that is sensible to do: ask yourself.

You could also ignore the matrix and always return 0. That is a proper similarity, too. Just usually not useful.

Don't just use some function because you heard the name. You need to understand it's properties, and you need to check whether these properties are helpful for your problem, or not. Choose one because math says this is solving your problem; don't replace the math with random guessing.

Cosine similarity link 1
Cosine similarity link 2
Cosine similarity link 3


Building a prediction model for dynamic coordinates and later categorize as binary classification

I was able to find a solution.

Thanks to this article which uses LSTM with binary classification modeling: https://www.analyticsvidhya.com/blog/2019/01/introduction-time-series-classification/


What is a good method for detecting local minims and maxims?

One way to do it is to calculate first derivative (difference in discrete domain) and find where there is a change in the sign. That indicates the existence of a local minimum or maximum. For example:

from numpy import diff, sign, cos, pi, arange
import matplotlib.pyplot as plt
t = arange(0,2,0.01)
x = cos(2*pi*t)
# Find derivative of x
first_derivative = diff(x)

# Calc. sign difference
sign_diff = sign(first_derivative[1:]) - sign(first_derivative[:-1])

# Find local min and max
local_max_index = [i for i,k in enumerate(sign_diff) if k == -2]
local_min_index = [i for i,k in enumerate(sign_diff) if k == 2]

# plot results
plt.figure()
plt.plot(t,x)
plt.plot(t[local_max_index],x[local_max_index], 'ro')
plt.plot(t[local_min_index],x[local_min_index], 'ro')

Hope it helps!


How can I fit categorical data types for random forest classification?

You need to convert the categorical features into numeric attributes. A common approach is to use one-hot encoding, but that's definitely not the only option. If you have a variable with a high number of categorical levels, you should consider combining levels or using the hashing trick. Sklearn comes equipped with several approaches (check the "see also" section): One Hot Encoder and Hashing Trick

If you're not committed to sklearn, the h2o random forest implementation handles categorical features directly.


How to create and format an image dataset from scratch for machine learning?

Looking at existing challenges around and their data format (for example http://www.kaggle.com/c/datasciencebowl/data) I would say put the images in a folder per class. You can use the file names for the index.


Meaning of Instance Space and Concept Class, (PAC Learnable)

This kind of language is typically associated within a field of math called computational learning theory (clt).  CLT is inherently abstract since it's trying to derive general observation about unknown target functions.  That's why you get generic terms like instance space and concepts.  Broadly speaking an instance space is typically just the domain of X (your sample population) and an instance of that space is a particular x with various features.  For example, X may be the set of all people and an instance of x will be an individual person who is 30 female and medium height etc.

The concept space on the other hand is subset of the domain X such that X -> {0, 1} or in other words a subset that maps X via Boolean function as either 0 or 1.  For example, Our concept space can be people who walk to work.  Every person X can either say yes they walk to work (and thus x -> 1) or no (x -> 0).

If this is still too abstract for you, I would recommend stepping back and building a more foundation  footing on basic machine learning before you start trying to study them as broad generic objects.


How to measure a relationship between interests of topics between two users?

You can create a sparse user-interest matrix, impute missing values with some reasonable value, and apply a similarity function. E.g., center all non-missing values on 0 and mutate each value to adjust for the user's and interest's mean. Then impute 0s for all missing data, and use a traditional similarity function like cosine.

You might also use Bayesian methods. You could model the relationship between users and interests. In this way you wouldn't have to impute missing data and it would account for confidence in inference and prediction.