Why don't we gradually update the activation parameters in RNN from one activation to the next as the network is learning more?

The motivation to use RNN is that the length of the sequence or position info is random in the data.

For example we could use the sample trained RNN model to translate the following setences:

I am a man
I am a woman and you are a man

In RNN, we do not consider the position of the words, only consider the relation between words. Thus the different activation parameters trained for different positions are useless.

Moreover to make RNN better informed (using previous words / next words) we could use Gated recurrent unit (or LSTM) and Bidirectional RNN.


how to visualize InceptionV3 hidden layers

Examples

Here are two adapted functions from the first of the links below that should get some weights out for a given input sample for you and plot the activations for some filters of that layer:

def getActivations(layer,sample):
    units = sess.run(layer,feed_dict={x:np.reshape(sample, [1,784], order='F'), keep_prob:1.0})
    plotNNFilter(units)

def plotNNFilter(units):
    filters = units.shape[3]
    plt.figure(1, figsize=(20,20))
    n_columns = 6
    n_rows = math.ceil(filters / n_columns) + 1
    for i in range(filters):
        plt.subplot(n_rows, n_columns, i+1)
        plt.title('Filter ' + str(i))
        plt.imshow(units[0,:,:,i], interpolation="nearest", cmap="gray")

This is the result, showing the second layer of activations from their model used for MNIST classification - the sample used was clearly a 7:

Rather than paste the entire code here, I suggest you read their instructions/explanations in context. The first link contains the function above.

Inspect activations at given layer using Tensorflow - (blog version)
Visualising layer activations using Keras
Using the tf.summary.image operator on a Variable, which will log the given Variable, allowing you to view it in Tensorboard. Examples here.

Some background

You can do this a couple of ways:

extract the activations for a given sample and plot them - you will get plots of varying sizes as you move through the network, correspoding to the dimensions of the weight matrix.
select your target layer, freeze all layers before that layer, then perform backbrop all the way to the beginning. This essentially extrapolates the weights back to the input, allowing you to display them over an input sample - giving insight as to what a particular layer is focusing on.

Here is a lecture presented by Matt Zeiler, who co-authored the original paper that showed the second method above. Their approach gives some results like this:

They use a "deconvolutional model" - which is also named the transpose convolution, which describes it more accurately. A deconvolution is actually a mathematical operator, which has a different meaning, so many people choose not to use the term.

Another approach to this, which goes a bit beyond what you asked, was proposed a few months later, by Simonyan et al., which looks at the saliency maps of weights/images - this is "the spatial support of a given class in a given image". IT can help localise objects - read the paper if that sounds good. In Section 4 of their paper, they claim their method to be a generalisation of the approach from Zeiler and Fergus.


Difference between machine learning and artificial intelligence

The subject areas Artifical Intelligence and Machine Learning (plus Data Science) are loosely defined, such that it is hard to make strict statements about how they relate. In the general case, it seems that there are parts that overlap, but that they are quite far from being "the same subject with two different names" as suggested in the question.

The term Artifical Intelligence has many possible meanings and interpretations - which version to refer varies by time and by the source using it. Textbooks on artificial intelligence will often cover topics such as search algorithms, logical deduction and other things which are clearly not machine learning as it is practised today.

For instance, we could take it to refer to Artificial General Intelligence (or "hard AI"), and it should be clear in this case that at least some form of learning algorithm(s) would be required to meet the goals of AGI. However, it is far less clear how much of AGI can be solved by combining machine learning into complex structures.

The term Machine Learning has a few different working definitions, but this is a popular one:

  A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

This is far more tightly defined than Artificial Intelligence, but still has a lot of scope.

The trend to conflate AI and ML appears to be a media and marketing issue, not a technical one. I suspect this is in part due to advances in the last 5-10 years in neural networks. Neural network models have made strong progress, especially in signal processing of images, video, audio. There is also an analogy with biological brains which can be compelling - especially when the subject matter is simplified for consumption by mainstream media.

It is worth mentioning Data Science too. Like Artificial Intelligence, the term is somewhat fuzzily defined. Also like AI, Data Science has more to it than just Machine Learning. To Data Science practitioners, ML is part of a toolkit to achieve goals - for some people it is a large part of what they do, for others it is just one part of a wider scope (actually training and refining a ML model might take only a small fraction of a professional data scientist, analyst or statistician's time). I think it is reasonable to state that Artificial Intelligence and Data Science relate to Machine Learning in a similar way.


How to Implement a Custom Loss Function with Keras for a Sparse Dataset

From the mentioned problems you are facing, this seems like a problem of exploding gradients. The exploding gradients problem can be identified by:

The model is unable to get traction on your training data (e.g. poor loss).
The model is unstable, resulting in large changes in loss from update to update.
The model loss goes to NaN during training.

More about Exploding gradient problem can be found at this article

I would suggest you to use some gradient clipping technique in you code and this will remove the NaN generation during model training.


Why are bigger embedding vectors not necessarily better?

You can think about phenomenons close to the curse of dimensionality.

Embedding words in a high dimension space requires more data to enforce density and significance of the representation.
A good embedding space (when aiming unsupervised semantic learning) is characterized by orthogonal projections of unrelated words and near directions of related ones. For neural models like word2vec, the optimization problem (maximizing the log-likelihood of conditional probabilities of words) might become hard to compute and converge in high dimensional spaces.

You’ll often have to find the right balance between data amount/variety and representation space size.


Warning when plotting confusion matrix with all sample of one class

If you insist on giving no positive examples it will have the same problem. As the error message states, the tool wants to compute the recall. The recall metric is given by:
$$recall = \frac{tp}{tp+fn}$$
$tp$ (true positive) is equal with the number of positive cases predicted as positive, which is $0$ in your case.
$fn$ (false negatives) is the number of positive cases predicted as negative, which again is $0$.
What you ask is to compute $\frac{0}{0}$ which is undefined. You can add positive cases also or you should study the tool (I'm not too familiar with Python stack)and perhaps you find an option to not compute recall metrics for you.


How to normalise estimate weighted on correlations (some will obviously be negative)

The appropriate method, if you have the data to use it, is linear regression.  Simply using correlations fails to take into account that the other variables are also correlated with each other; for example, any averaging method like the one you describe will overweight any duplicated variables.


Machine learning system which can learn from data and human rules

The area of combining machine learning and human knowledge is called expert-augmented machine learning.
One common method is called RuleFit. RuleFit has two steps:

Fit a tree-based model to make rules.
Fits a linear model with these rules as input.

The result is highly interpretable which can be filtered or applied by humans.


Dropping Missing Observations under MAR Assumption

One option is to run the regression analysis both ways (imputation vs dropping) and see which results are more useful. For your problem, find the best empirical estimate of bias and see if that estimate changes because on the feature engineering choice.


Sliding window approach using SVR & LightGBM

Tensorflow is designed to really make your life easier (especially with the fancy new additions with the 2.0 version). I think you should use pandas to generate sliding windows: I would think that for a N sized time series with K lookback window you will have N-K+1 examples by sliding lookback.

And while you're at it have you treated your lookback window as a hyperparameter? i.e optimise your model performance on a validation test by searching the best lookback?


Your link is to paid course :) In ray-tracing too few samples will generate something like at the top  In fact the link with the picture answers your question https://chunky.llbit.se/path_tracing.html

2) Ray-tracing is hard... but not impossible, google for "python ray tracing module"... But something looking close - easily https://stackoverflow.com/questions/22937589/how-to-add-noise-gaussian-salt-and-pepper-etc-to-image-in-python-with-opencv Although actually on the ray-traced images the noise can change because of slope and environment.

If you still want ray-traced noisy images, better to find tutorials for 3D modelling programs, like "ray tracing in 3D Studio MAX tutorial"


Differences between applying KMeans over PCA and applying PCA over KMeans

There is a difference. In your first strategy, the projection to the 3-dimensional space does not ensure that the clusters are not overlapping (whereas it does if you perform the projection first).

This is because some clusters are separate, but their separation surface is somehow orthogonal (or close to be) to the PCA.

If you increase the number of PCA, or decrease the number of clusters, the differences between both approaches should probably become negligible.


Are ontologies and the Semantic Web dead?

(While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.)

Is the Semantic Web dead? Are ontologies dead?

Many people moved away from the Semantic Web. On the other hand, there are still many using it. There's always been a lot of confusion about where its value lies. There are use cases that genuinely benefit from the semantic web technologies but they are quite few and far between.

Linked Open Data

That's partially why the subfield of Linked Open Data was created. You can see Linked Data as a more pragmatic stripped down version of the Semantic Web (or as a necessary enabler of the grander semantic web vision).

Linked Data, while more pragmatic, still uses ontologies. It is just not so uptight about using OWL and designing your ontologies first and in a very formal way.

A knowledge base such as Freebase (now being replaced by Wikidata) doesn't even build on Semantic Web technologies (it's always been available also in the form of RDF dumps though). On the other hand, even Freebase builds on concepts similar to those in the Semantic Web and it does have a form of ontologies.

Software to generate ontologies

There is no software that would create high-quality ontologies automatically for you. At least not in the sense of OWL/first-order logic ontologies. On the other hand, many kinds of software, including Cratilo mentioned by you, can help you to build a lower step in the semantic spectrum such as a glossary or maybe even a folksonomy which can later be transformed into more of a taxonomy or ontology by other algorithms. There might be even software to create fullblown OWL ontologies but I think that's still rather an active area of research.

Ontologies and methodologies

As always, it depends on what exactly your needs and goals are. In fields such as biomedicine and life sciences in general, they create complex formal ontologies because they have data with highly varied structure and ontologies help them keep things organized and usable. In contrast, an eshop might be better off sticking with normal relational database modelling and adding only global identifiers (URIs) in the spirit of Linked Data if they want to build a proper knowledge graph later.

Even if you need ontologies you might want to skip the methodologies first. Especially if you are really new to the field and have little idea what your needs are.

Jena, Sesame, ...

Jena, Sesame, Virtuoso, etc. are triplestores - they are used to store and query RDF. Most ontologies can be represented in RDF. Even those written in OWL. RDF has the semantic part, RDF/S, which allows you to formulate some basic ontologies. That might well be enough for start. Even RDF/S can get confusing and convoluted when you start to think about blank nodes, named graphs (are they fixed? are they dynamic?), etc.

While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.


MemoryError is exactly what it means, you have run out of memory in your RAM for your code to execute.

When this error occurs it is likely because you have loaded the entire data into memory. For large datasets you will want to use batch processing. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access it in batches. If you are using Keras there is a helper class with a very efficient implementation of batch processing. Take a look at this blog post. This is a good starting point for avoiding MemoryError.

As a short term fix you can train your model using a subset of the data available to you and discard the rest. Doing this really is a shame however.


RMSE is higher for bigger values of target variable - how to decrease

It looks like that your prediction is clamping at 750.
Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on.
So, first of all, please assure that your data doesn't have a trend.


Recommender system that matches similar customers with similar highly rated products?

You want to do three things:

Cluster similar customers based on their profiles and their preferences for certain devices
Cluster similar devices based on ratings but also device attributes
Create a recommender system based on clusterings 1 and 2

Here is how I would do it:

Cluster customers via usual means (PCA, K-Means, etc.) based on their demographics
For each cluster develop a model that predicts the "rating" of a device based on the devices attributes (e.g. via ensemble learning, association rules, etc.). Then use the model to predict the rating of a new device for each cluster!
Implement a simple recommender based on the cluster membership and a relevant device set based on ratings, this works because of step 2 each device now has a ranking. If you want to "manipulate the customer" further you can add simple rules to add devices below the relevant rating threshold if they satisfy certain criteria (price, "is on sale", etc.).

Voila!


Here I am answering my own question:

To obtain an IDW surface interpolation restricted to the area containing the original set of xyz data points:

Create a rectangular grid based on the xy point ranges using expand.grid {base}
Create a convex hull using the function convexhull.xy{spatstat}
Cut an irregular grid using the function inout{splancs} on the
rectangular grid masked by the convex hull
Perform the interpolation using the function idw{gstat} with the new grid.
Be sure to detach spatstat, because it has a different function for
IDW with the same name

Code:

###
# IDW in R
#
# Load gstat:
library(gstat)
#
# An example XYZ data set:
set.seed(1234)
x <- rnorm(20)*1000+3000
y <- rnorm(20)*1000+3000
z <- rnorm(20)+10
xyz <- data.frame(x,y,z)
#
# Calculate ranges and spacing:
xr <- round(range(x),1);xr
yr <- round(range(y),1);yr
b <- round((xr[2]-xr[1])/60,2);b
#
# Create a grid:
grd1 <- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),
    y=seq(from=yr[1],to=yr[2],by=b))
plot(grd1)
#
# Load spatstat and splancs
library(spatstat)
library(splancs)
#
# Extract convex hull
w2 <- convexhull.xy(x,y)
#
# detach spatstat because it has a different "idw" function:
detach("package:spatstat", unload=TRUE)
#
# Create a new cropped grid:
grd2 <- grd1[inout(grd1,w2$bdry[[1]]),]
plot(grd2)
#
# Convert grids and dataset to sp objects:
coordinates(grd1) <- ~x+y
gridded(grd1)   <- TRUE
coordinates(grd2) <- ~x+y
gridded(grd2)   <- TRUE
coordinates(xyz) <- ~x+y
#
# Run the interpolations:
idw1<-idw(formula=z~1, locations=xyz, newdata=grd1)
idw2<-idw(formula=z~1, locations=xyz, newdata=grd2)
#
# Plot the results:
image(idw1, col=terrain.colors(16))
contour(idw1, add=T)
points(x,y, col=4, pch=19)
image(idw2, col=terrain.colors(16))
contour(idw2, add=T)
points(x,y, col=4, pch=19)
#


Multiple regression (using machine learning - how plot data)

There might possibly be a better way but one way of doing this is to map the variable to different aesthetics of the graph. I used python and used library plotninne which is as I understand tries to mimic ggplot2 from R.

from plotnine import *
   # df is the data frame that contains all the vairables as columns
   (ggplot(df, aes('actual_value', 'predicted_value',
   color='(dependent_var2)',
   size='dependent_var3',alpha='dependent_var4',shape='factor(dependent_var5)'))
    + geom_point()
    +theme(legend_title=element_text(size=8),
    legend_text=element_text(size=4)
    )

This will give you following graph:


The reason the other bars are missing is because of your method of summing the values and the missing values in your dataset. The way you are adding the values together means that if even one value is missing (NA) the total for that column will be missing as well, and as a result, will not be in your final plot. It is better to use pandas built-in methods (for conciseness, speed, but also to avoid errors such as the one you encountered) to add up the values as follows (using random values):
import pandas as pd
import matplotlib.pyplot as plt

datainput = pd.read_csv("MD_COVID-19.csv")
fig = plt.figure()
ax = fig.add_axes([0,0,1,1])
regions = ['Allegany', 'Anne_Arundel', 'Baltimore', 'Baltimore_City', 'Calvert', 'Caroline', 'Washington']
# select all region columns and calculate the sum column-wise
total_cases = datainput[regions].sum()
ax.bar(regions, total_cases)
plt.ylabel("Number of Cases")
plt.show()

Which gives the following result:


What is the difference between symmetric bipartite graphs and a complete bipartite graph?

Your first link's usage of 'symmetrical bipartite graph' is indeed puzzling. From my knowledge of RBMs it would have been better for them to say 'complete bipartite graph' as you suggest.

The only formal reference I can find online for 'symmetry' in bipartite graphs is this link. But that does little to explain the usage in your case.


Courses: Andrew Ng Machine Learning Course from Coursera is what introduced me to machine learning, and I cannot think of a better way to do it. The course focuses on the theory rather than practice, but it offers solid theoretical foundations. fast.ai Machine Learning course is more programming oriented, I haven't taken it but it looks good.

Books: Hands on machine learning with Scikit-learn and Tensorflow by Aurélien Géron has a decent amount of theory explained in a very simple way, and it complements it with lots of code examples. If you want to dig deeper in statistics and theory, The Elements of Statistical Learning might be a good resource.

Practice: Kaggle is for sure the best place to practice and learn. After reading something or doing some course I advise to download the Titanic dataset and try to submit predictions.


Try general linear model to see how the features interact.  Decide if the problem is classification or trend or anomaly then post back your discovery then I can help .
I applied K-Means, TSNE dimension reduction, and PCA and saw maybe two clusters.
it is not conclusive clustering.  There were a few observations discovered by severity, age, and gender of the heart events.
from sklearn.decomposition import PCA
from scipy.cluster.vq import vq, whiten
from scipy.cluster.hierarchy import linkage, fcluster,dendrogram
from sklearn.preprocessing import LabelEncoder
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn.cluster import KMeans
from sklearn.manifold import TSNE

calltype=['Emergency','Emergency','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Hosp to Hosp','Hosp to Hosp','Emergency','Hosp to Hosp','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency','Emergency']
district=['District 1','District 2','District 3', 'District 4','District 5','District 1','District 2','District 3','District 4','District 5','District 1','District 2','District 3','District 4','District 5', 'District 1','District 2','District 3','District 4']
gender=['Male','Male','Male','Male','Female','Male','Male','Female','Female','Male','Male','Male','Male','Male','Male','Female','Male','Male','Female']
complaint=['Chest Pain','Heart Problem','Cardiac Arrest','Heart Problem','Cardiac Arrest','Chest Pain','Cardiac Arrest','Heart Problem','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem','Chest Pain','Heart Problem','Heart Problem','Heart Problem','Chest Pain','Heart Problem']
 age=[28,50,76,45,52,34,44,55,86,65,60,36,45,65,60,35,80,46,70]
 area=['Rural','Rural','Rural','Urban','Urban','Urban','Urban','Rural','Rural','Rural','Urban','Urban','Urban','Rural','Rural','Urban','Rural','Urban','Urban']

 df=pd.DataFrame({'calltype': calltype,'district':district,'gender':gender,'complaint':complaint,'age':age,'area':area})
 df['severity']=df['complaint'].apply(lambda row: 3 if row=='Chest Pain' else 2 if row=='Heart Problem' else 1 if row=='Cardiac Arrest' else 0)

 encoder=LabelEncoder()
 df['district']=encoder.fit(df['district']).transform(df['district'])
 df['gender']=encoder.fit(df['gender']).transform(df['gender'])
 df['area']=encoder.fit(df['area']).transform(df['area'])
 df['calltype']=encoder.fit(df['calltype']).transform(df['calltype'])
 print(df)

 sns.countplot(x='gender',hue='severity',data=df)
 plt.legend(['Cardiac Arrest','Heart Problem','Chest Pain'])
 plt.show()

 print('Gender 0=Female 1=Male')
 fp=df.pivot_table(index=['severity'],columns=['gender'],values=['age'],aggfunc='mean')
 print(fp)

 print("\n More males had severe heart problems")

 fig,ax=plt.subplots(figsize=(10,8))
 x=np.linspace(0,len(df),len(df))
 ax.bar(x,df['severity'],alpha=0.3,color='green')
 twin_ax=ax.twinx()
 df[['age']].plot(ax=twin_ax, c='red')
 plt.show()

 print('\nAges 30-40 had high frequencies of cardiac arrest')

 print("\nK-Means\n")
 labels=df['gender']
 xs=df['age']
 ys=df['severity']
 points=np.column_stack((xs,ys))
 model = KMeans(n_clusters=3)
 model.fit(points)
 print(model.inertia_)
 new_points=np.random.permutation(points)
 new_labels = model.predict(new_points)
 print(new_labels)
 xs = new_points[:,0]
 ys = new_points[:,1]
 plt.clf()
 # Make a scatter plot of xs and ys, using labels to define the colors
 _=plt.scatter(xs,ys,c=labels, alpha=0.5)
 # Assign the cluster centers: centroids
 centroids = model.cluster_centers_
 # Assign the columns of centroids: centroids_x, centroids_y
 centroids_x = centroids[:,0]
 centroids_y = centroids[:,1]
 # Make a scatter plot of centroids_x and centroids_y
 _=plt.scatter(centroids_x,centroids_y, marker='D',s=50)
 plt.show()

 print("\nTSNE\n")

 model=TSNE(learning_rate=100)
 X=df[['calltype','district','gender','age','area','severity']]
 transformed=model.fit_transform(X)

 xs=transformed[:,0]
 ys=transformed[:,1]

 plt.scatter(xs,ys,c=labels)
 plt.show()

 print("\nPCA\n")
 pca=PCA(n_components=3)
 pca.fit(X)

 transformed=pca.transform(X)
 print(transformed.shape)

 xs=transformed[:,0]
 ys=transformed[:,1]
 plt.scatter(xs,ys,c=labels)
 plt.show()


High train and val results. Bad test and predict results

I figured it out.

First: I was making the labels the wrong way.

Instead of this:

train_labels = to_categorical(np.array(
[0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

do this:

train_labels = to_categorical(training_generator.classes, 4)

The first snippet wrongly generated a 4-ary array, where only the first and second element got a 1 assigned.

Second:

I found out my method of evaluation was wrong. I tried to use model.evaluate_generator which yielded the wrong results. I think it was because I tried to feed the model raw images, instead of creating bottleneck features from the raw test images and then feeding them to the model.

so instead of this:

from keras.models import load_model
conv_base = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(conv_base)
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator = test_datagen.flow_from_directory(
        '/content/output/test',
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical')

test_loss, test_acc = model.evaluate_generator(test_generator, 216 // batch_size)
print('test acc:', test_acc)
print('test loss:', test_loss)

I did this (Which worked):

from keras.models import load_model
model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

bottleneck_features_test = model.predict_generator(
    test_generator, 261 // batch_size)

loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    '/content/output/test',
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    classes = folders,
    shuffle=False)

bottleneck_features_test = conv_base.predict_generator(
    test_generator, 261 // batch_size)

test_labels = to_categorical(test_generator.classes, 4)

(test_eval_loss, test_eval_accuracy) = model.evaluate(
   bottleneck_features_test[:256], test_labels[:256], batch_size=batch_size, verbose=1)

print("[INFO] test accuracy: {:.2f}%".format(test_eval_accuracy * 100))
print("[INFO] test Loss: {}".format(test_eval_loss))

I hope this helps someone who's struggling with the same!


I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function?

In the case of neural networks, the loss function depends on the weights and the biases (which are often not mentioned extra; they are also weights).

The loss function itself is parametrized heavily. Its parameters is the data (input and labels). Its variables (by which the derivatives are calculated) are weights.

  And how would zero mean and equal variance help in optimization?

Have a look at the derivative of the sigmoid function. It is biggest at 0. This means the gradient there can be big. This helps by learning, because the basic learning rule is

$$w \gets w_i + \Delta w_i\;\;\; \text{ with } \Delta w_i = - \eta \frac{\partial E}{\partial w_i}$$

So you can get bigger adjustments if you normalize it to have mean of 0.

The part about the variance ... hm. That's harder to explain. I'm not entirely sure about it. One thought is that you want the data to be in a very restricted, similar domain (independent of the application) so that you can treat results independently from your application. Also, it might help with mini-batches not varying too much.

  in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma?

I'm not sure what exactly the question is.

Practically: Just use a function. For example, numpy.random.normal if you're using Python.
Mathematically: This is called sampling. You have a process which has a random distribution and you take examples from that.
Technically: If you really want to implement this yourself, you will most likely end up using a random number generator which generates samples from a uniform distribution in [0, 1].

Once you have those sample numbers, you simply assign those numbers to the weights.

  And where optimizer will move the point (initialized weight), up or down to find the local minima?

The optimizer calculates the gradient of the error function. I recommend having a look at the early chapters of the Udacity course. I'm pretty sure this was covered there, too. Another resource would be neuralnetworksanddeeplearning.com

  So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?

Tom Mitchells book "Machine Learning" covers similar topics as the course.


MNIST 2-layer NN failing to recognize certain digits

Both sigmoid activation and MSE loss often don't work so well.  I would start by using standard methods, i.e., softmax and cross-entropy loss.


Why my classification accuracy is high for both training and testing data?

Your data contains 7621 records and 3873 of them are unique records.
When there is duplicates in different folds, cross-validation fails. And these duplicates causes to high train&test set performance metrics. Removing the duplicates decreases the test set's accuracy and f-1 scores:
import pandas as pd
from sklearn.model_selection import KFold,cross_validate, train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSvUmbtHUh2e0iYj7nMDaP8Tf_pCnCa-HrWwAmaxrERxxvd2y_5qxuSP10t6db4RUSjTdOi9WshZhoR/pub?output=csv', index_col=0)
df.drop_duplicates(inplace=True)

input_selected_features = df.drop(labels = ['Target'], axis = 1)
target_selected_feature = df['Target']

# k fold cross validation scores
kf = KFold(n_splits = 10, shuffle = True, random_state=42)
cv_results = cross_validate(estimator = DecisionTreeClassifier(), X = input_selected_features, y = target_selected_feature, cv = kf, scoring = ['accuracy', 'f1'], return_train_score = True)

# print(cv_results)
print('training accuracy - ',cv_results['train_accuracy'].mean())
print('testing accuracy - ',cv_results['test_accuracy'].mean())
print('training f1 score - ',cv_results['train_f1'].mean())
print('testing f1 score - ',cv_results['test_f1'].mean())

> training accuracy -  0.9990828974566813
> testing accuracy -  0.7454174325368284
> training f1 score -  0.9990071171229987
> testing f1 score -  0.727174422877005

Note that, also the first column in the original csv file is an index column (1,2,..). It is better to read that column with index_col argument.


adding a supervising process during knn process

KNN Regressor, per se, does not have a dedicated learning process as it just takes weighted average of the dependent variable of the k-nearest neighbors from your training set of the test datapoint whose dependent variable you have to predict.

You can try using Support Vector Regressor or MLP (Neural Network) Regressor offered by scikit-learn and see if the performance has increased with respect to number of test cases where predictions resulted in efficiency > 1.

If you are adamant on incorporating this condition in your learning process, you can use Keras and create a custom loss function which returns a high value if your efficiency results in a value greater than 1. Although the training will be sensitive to this value and higher values can shutdown the neurons, resulting in all zeros output. But if you are careful you will be able to get sensible predictions with efficiency < 1.

def custom_loss(voltage, current, torque, speed):
    def loss(y_true, y_pred):
        #your logic here
    return loss

model = Sequential()
model.add(Dense(16, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
model.add(Dense(1))
model.compile(loss=custom_loss(input_1, input_2, input_3, input_4), optimizer='adam')


Random forests should be able to handle categorical values natively so look for a different implementation so you don't have to encode all of those features and use up all your memory..
XGBoost
Dimensionality Reduction
CatBoost
Hierarchical Clustering Using Scipy
Target Encoding


Object detection/recognition, pre-processing error

You have to create labels for each of the images and then split it into train and test. I believe you have 20,000 images - so you have to also have 1 label for each image not jus the 4 categories alone in an array. One of the most important steps in training a DNN model to do image classification (as is your case) or any image related task is creating labels (in your case) and otherwise annotations for other problem statements. I think you have just created a array of what your labels are and havent associated them with the actual image itself.


Why does the naive bayes algorithm make the naive assumption that features are independent to each other?

By doing so, the joint distribution can be found easily by just multiplying the probability of each feature whilst in the real world they may not be independent and you have to find the correct joint distribution. It is naive due to this simplification.


Do you apply confidence threshold before calculating mAP for YOLOv2?

In the end the solution is not to apply the threshold prior to the mAP calculation. After researching further it turns out by definition that mAP is (roughly speaking) a measure of precision recall trade off calculated using all possible thresholds. By applying the threshold first you are not accounting for the higher recall the model would obtain by choosing a threshold less than or equal to the one applied.


In a scenario where consequences of prediction errors are not equivalent, you are usually still interested in training a model to predict accurately from the data set, and would not change the objective function in supervised learning.

Typically when consequences of FP and FN differ, you would:

Use the confidence of the prediction given by the model. In XGBoost, that is objective: "binary:logistic"
Use area under ROC as a base metric for deciding best tuned model (this is eval_metric: "auc" for XGBoost). This metric is a measure of how well sorted your classes are - the higher the value, the easier and more effective it is to tune the confidence level later.
Decide a weighting for your costs FP and FN - this is entirely up to you, you say that FP is worse than FN, but in order to make an optimal decision you have to turn that into a numerical statement. If you can assign a relative financial cost to a business for each kind of mistake, then that would be a good start.
Using the best tuned model, use the weightings to calculate costs at different cutoffs of class confidence before assigning positive class that you wish to act upon in some way. Predict against a test set to get class probabilities. Then use different cutoff points for positive class, count your FP and FN given that cutoff, and multiply the totals by the costs you chose later. The lowest-scoring cutoff point should be the one to use in production. If FP has a higher cost than FN, then you will likely find that a confidence level > 0.5 is required.

You can also look into machine learning approaches that consider the consequences of actions - e.g. reinforcement learning - in order to achieve a similar result within a single learning framework. However, that is not something you can do solely within XGBoost, and as long as your problem remains balancing between FP and FN in a single step prediction, then the above approach should be OK.


This is the problem: model.add(keras.layers.Dense(1, activation='softmax'))

For predicting real-valued data such as age, it is customary to set the activation as linear or in this case, you can probably use relu.

To illustrate, softmax will create a distribution over the output, in this case the output of the model will be always 1. Since the age is positive definite continuous number, you need an activation that has a range of real numbers >= 0 which the relu activation satisfies.

  As another question , are my layers good to predict a number for given picture ?

The architecture seems reasonable so try changing the model's final activation first.


Reducing (Variance) | the gap between my weights

Try the softmax function:
weights = [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]

temperature = 1
weights = np.array(weights) / temperature
new_weights = np.exp(weights) / np.exp(weights).sum() # softmax function

You can tweak the temperature hyperparameter.
Higher the value - the "new_weights" come closer - yet order is preserved


If you are exploring the best hyper-parameter combination that maximizes precision you could try GridsearchCV along with KerasClassifier. You create a wrapper of your model and use it as the estimator of the GridsearchCV method. Also, since you want to maximize precision, you would have to set the scorer parameter to precision. After that you are going to have to set the hyper-parameter space that you would like to explore and finally run the experiment.

Please bear in mind that this approach is computationally expensive. If your dataset is very large, a space of n > 2 (i.e 2-d), would need a cluster so as to produce results in a timely manner.


I interpret it as the input takes binary value $0$ (false) and $1$ (true) while the output take $-1$ (false) and $1$ (true).

The map is $$sign\left(\sum_{i=1}^nw_ix_i+w_0\right).$$

If we let $w_i=0.5, i \ge 1$, then we have the mapping rule to be

$$sign\left(\frac12\sum_{i=1}^nx_i+w_0\right).$$

For the AND operator, if we set $w_0=-0.8$, for it to take positive value both $x_i$ has to be $1$.

For the OR operator, if we set $w_0=-0.3$, we just need one of them to take positive value for it to be true.


How can we model the class which maximizes the event probability?

In your method, you would build a model which maps the input features (ad type, customer descriptors) to the output (purchase made = 0/1). You can use a model such as a logistic regression, or a decision tree (which can model the interaction effects between the ad type and the customer descriptors.) This seems to be reasonable and is one of the approaches commonly used for ad selection.

A few other options based on customer similarity:

Near-neighbour based approach: Based on the customer descriptors, identify similar neighbours of the current customer. Compute the probability of conversion for each ad type for all the neighbours put together, and pick the ad type which has the highest probability of conversion.
Cluster based approach: Form clusters of customers based on the customer descriptors. Map the current customer into one of the clusters, and pick the ad type which maximizes conversion probability for that cluster.

With respect to designing a system of ad selection using such a model, it is also important for the system to explore new regions of the space, so some small randomization of predictions will help here.


How to preprocess data for image classification from a .txt file?

I am assuming from the description inside main directory there are more directories of  Apples, Bananas, Oranges, etc. and inside them you have .txt files containing information about the images.

import shutil

with open(file_path, 'r') as f:
    img_names = f.readlines()
    img_names = [img.strip() for img in img_names]

for i in range(len(img_names)): shutil.move(img_names[i], folder_path)

Once images are moved into there respective folder you can use ImageDataGenerator approach to train model.


how to do incremental deep learning on data stream that can adapt to constantly generated data points?

What you are describing does not require a change in the deep learning models you would use but a change in how to estimate them: you need an online estimation procedure rather than an off-line one. You can have a look at the seminal work of Botou [1998], or Chapter 8 of the book by Goodfellow et al. (2016).


How to tune bandwidth in machine learning kernel model?

As shown in wikipedia for KDE, a rule-of-thumb bandwidth estimator can be given if the underlying density for your data is Gaussian. This estimator is given by: $h = (\frac{4\hat{\sigma}}{3n})^{1/5}$, where $h$ is the bandwidth of your KDE estimation, $n$ the number of data and $\hat{\sigma}$ the estimation of the standard deviation of your sample.

If the underlying distribution for your data is not gaussian, you can still try with this bandwidth, but it might smooth everything too much. In that case, you should go for cross-validation with smaller bandwidths.


An exhaustive, representative test database in phrase search algorithm

Based on the example I assume that the target is persons names.

Let's be clear, there's no such thing as an exhaustive dataset containing all possible persons names in the world. Also a crucial part of the question is: in What kind of names? In which language? Persons names in English are pretty different from names in Chinese for instance. And there is also the difficult question of transliteration of proper names.

That being said, a few resources exist. They would usually be found by searching resources for "personal names", "record linkage", "named entities matching/coresolution". The following ones probably don't cover all the requirements but it's a start:

Febrl (see also here)
Found This paper which presents a large resource and this corresponding resource description but couldn't find the data.
The EMM news explorer has an interesting database of named entities including persons names with all the spelling variants/transliterations.


You should try contacting cities about getting the sensor data from their Traffic Controller Systems. These usually have several detectors per lane at intersections (they also control the traffic lights so you get data on those as well). Some of these data are property of the companies that supplies these system, in which case you'll need to contact the company. These companies are generally not keen on sharing the data with commercial parties, but generally are interested in sharing it with educational parties as outreach.

I know one public dataset of this kind of data, the city of Utrecht in the Netherlands provides a sample of traffic density data. Unfortunately the metadata is mostly in Dutch.


Why is the Naive Bayes classifier of sklearn faster than sklearns SVM?

The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation.

Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates can be calculated independently and thus possibly very quickly.

In comparison, with SVM, every single record in the data base will require a computation of the distance function to determine the optimal decision boundary.

We can observe that for SVM, we have a complexity of $O(\text{#features} \times \text{#observations}^2)$ or more depending on the choice of SVM and kernel. On the other hand, for naive bayes we would have a complexity of $O(\text{#features} \times \text{#observations})$.

Hence in general, we can say that SVM will take longer than Naive Bayes.


Figure out relative importance of entity attributes

Approaches 1 & 2
1. Noise
If we adopt a frequentist view, we can consider the attributes of a movie to have an underlying probability distribution (or, if you'd rather be Bayesian, you can choose to believe they come from a certain distribution). For a single attribute, its values are a sequence of random variables drawn from the same distribution. We can model our data for one attribute by considering a data model of the general form $Y = X + \epsilon$, where $Y$ is a random variable representing one measurement, $X$ is a random variable coming from the "true" underlying distribution and $\epsilon$ is a noise term.
For some variables, the noise term is negligible (e.g. the runtime of a movie is actually not noisy at all, it's a real, objective, fixed value). For others, noise may follow a normal distribution, or another distribution with mean 0. In that case, over many measurements, the law of large numbers dictates that the mean of your $Y$s will converge to the expectation of $X$. Here comes the first warning: if you have noisy data and few of them, your data might not mean much.
Finally, you might be so unlucky that you have noise $\epsilon$ coming from a distribution with a mean which is not $0$, in which case you are likely to have systematically biased data.
In both approaches, you partition your observations into groups/classes (in 1 based on the values of one attribute, in 2 on the values of all other attributes). This can lead to class imbalance, in which case you might end up with some classes with many observations (and presumably little noise) and some classes with few members (maybe only one!) and a possibly a lot of noise.
Furthermore, if you look at variation only, and not correlation, then you might end up with high variance (even comparatively, if one attribute is very noisy perhaps due to previously mentioned class imbalance) and low correlation, which would erroneously lead you to deduce that an attribute has good explanatory power.
2. Multicollinearity
Approach 1 opens you up to the problem of collinearity: two predictor variables might be strongly linked, in which case it's not easy to say which one is really a valuable predictor and which one can be considered just an epiphenomenon. For instance, assume fantasy as a genre is very popular, but people dislike overly long movies. Fantasy movies would then be very long and have a large revenue, but it would be wrong to conclude that the long runtime is responsible for their large revenue. Even worse: while this example used two attributes which you probably have access to, a hidden, unobserved (i.e. latent) attribute might actually be causally responsible for the phenomenon and for the correlation between the two observed attributes.
In approach 2, you're doing something akin to statistical matching, so you at least stand a better chance of avoiding multicollinearity and confounding variables.
Personal recommendation
I would personally start with some exploratory data analysis (EDA). Plot your data! Use heatmaps, small multiples, violin plots, the works, anything that might give you insight into its structure.
Next, start small, go big if needed. Try measures of "linear" correlation, first Pearson correlation. For categorical variables, you can use Spearman rank correlation or Kendall rank correlation. You can also use measures of rank correlation (e.g. those mentioned previously) to estimate non-linear correlation. You can also use SHAP values, Q-Q plots, partial plots.
You can then move on to maybe a simple decision tree (or random forest), or a linear regression if you encode your categorical variables. If that gives you good accuracy, then you can examine what your model has learned. You can use stepwise linear regression, where you either add or subtract attributes one by one and examine the change in predictive power.
You can also try dimensionality reduction, for instance do PCA or ICA then look at which attributes carry a lot of weight in the first few PCA/ICA components. You can also try other types of matrix factorization.
If accuracy is poor with simple models, you can try more complex models, such as neural networks, then use interpretability models designed for those, such as LIME, which builds a locally accurate linear surrogate of a deep network.


how to build word2vec content based recommendation?

You can find a number of blogs teaching the use of Word2Vec,

Introduction to Word Embedding and Word2Vec
A Beginner's Guide to Word2Vec and Neural Word Embeddings

If you are building a hotel recommendation system, I think the name of the hotel is not a good feature. Users are less concerned with the name of the hotel than its location or description provided.

You can take a small description from the user and the location in which they would like to search for the hotel. Using Doc2Vec, convert the description ( text ) into a fixed-sized vector ( embedding ). Find a hotel whose description is the most similar to the one provided by the user ( maybe using Cosine Similarity or any other measures ).
Before this find a hotel that is close to the location provided by the user ( say a hotel which lies in a radius of 2 km from the given location ).

Using these two factors, you can prepare a list of hotels whose location and description match the user's needs.


Classification of non-stationary acoustic signals

As you try to classify sequential data, you can try simple recurrent neural network or their advance version LSTMs - but as you have a short signal, RNN should work fine.

You can read this paper on sound classification with LSTM and this  medium article


Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated)?

Theoretically, normalization does not influence the performance of the model. In order to understand this let us have a look at a standard linear regression.
$$y_i = \boldsymbol{w}^T\boldsymbol{x}_i + b+\varepsilon_i$$

If we Introduce the scaled independent variable $\boldsymbol{z}_i=\dfrac{1}{\sigma}\left[ \boldsymbol{x}_i - \bar{\boldsymbol{x}}\right] \implies \boldsymbol{x}_i=\bar{\boldsymbol{x}}+\sigma\boldsymbol{z}_i$. This will result in
$$y_i = \boldsymbol{w}^T\bar{\boldsymbol{x}}+\sigma\boldsymbol{w}^T\boldsymbol{z}_i+b+\varepsilon_i.$$

If we introduce $\tilde{b}=b+\boldsymbol{w}^T\bar{\boldsymbol{x}}$ and $\tilde{\boldsymbol{w}}^T=\sigma\boldsymbol{w}^T$ we can rewrite the equation into

$$y_i=\tilde{\boldsymbol{w}}^T\boldsymbol{z}_i+\tilde{b}+\varepsilon_i.$$

Hence, we see that the transformed independent variables just change the bias (if a translation is included) and the weights are scaled by $\sigma$. The significance of the parameters will not change only their specified values.

But if normalization does not really enhance the model, why do we still do it? The reason is more computationally inspired. If we had very large values the weights would need to be very small such that we can scale the output of the regression into a reasonable range. A large range of numerical values forces us to reserve more memory for our variables that we use. Hence, it is better to normalize our inputs, such that the parameters don't need to scale the inputs down to a reasonable amount that it fits the output.

  When we normalize the training set we ought to normalize the target set too. Won't it affect the performance? I mean won't the data set
  change completely because the model had different ranges of features
  as compared to the ranges of features in the target set.

Normalizing the output is not necessary, but it can also improve the numerical efficiency. You can just use the previous linear transformation on your dependent variable (output) and you will see that you can rewrite it to a standard linear regression in the new output. Just remember to transform your inputs and retransform your outputs if you want to use the original variables.

Does the significance of the parameters change?

In order to show that scaling the inputs by a constant factor $\sigma$ does not influence the significance of the parameters, we will calculate the $t$-value for a given coefficient. If the $t$-value stays invariant the $p$-value will also stay invariant.

For the linear regression $y_i = \boldsymbol{w}\boldsymbol{x}_i+\varepsilon_i$ (bias absorbed into the weight vector). The regression coefficients are given by

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\boldsymbol{X}^T\boldsymbol{y}\quad \text{ , in which } \quad
\boldsymbol{X}=\begin{bmatrix}\boldsymbol{x}_1^T \\ \vdots \\ \boldsymbol{x}_N^T\end{bmatrix}$$

is the data matrix with an added $1$-column for the bias.

Additionally, we need the matrix
$$\boldsymbol{C}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\quad \text{ and } \quad s_e = \sqrt{\dfrac{(\boldsymbol{y}-\hat{\boldsymbol{y}})^T(\boldsymbol{y}-\hat{\boldsymbol{y}})}{N-p-1}},$$

in which $N$ is the number of observations, $p$ is the number of predictors, $\boldsymbol{y}$ is the vector of outputs and $\hat{\boldsymbol{y}}$ is the vector of predicted outputs. We saw that the predicted values will stay invariant under scaling. Hence $s_e$ is invariant under scaling.

The $t$-value for a regression weight given by

$$t_i= \dfrac{\hat{w}_i-\mathbb{E}\left[\hat{w}_i\right]}{s_e\sqrt{c_{ii}}}.$$

The $c_{ii}$-values are the corresponding diagonal values of the $\boldsymbol{C}$ matrix.

If we scale our inputs by $\sigma$ (we ignore the 1 column of the data set which is only important for the bias) then we observe

$$\boldsymbol{X}'=\sigma\boldsymbol{X}
\text{ , }\hat{\boldsymbol{w}}' = \dfrac{1}{\sigma}\hat{\boldsymbol{w}} \text{ , } \mathbb{E}\left[ \hat{\boldsymbol{w}}'\right]=\dfrac{1}{\sigma}\mathbb{E}\left[ \hat{\boldsymbol{w}}\right] \quad \text{ and } \quad \boldsymbol{C}' = \dfrac{1}{\sigma^2}\boldsymbol{C}.$$
The last condition implies
$$\implies c_{ii}' = \dfrac{1}{\sigma^2}c_{ii} \implies \sqrt{c_{ii}'} = \dfrac{1}{\sigma}\sqrt{c_{ii}}.$$

By these observations, we see that the $t$-value stays invariant.

$$t' = \dfrac{\hat{w}_{i}'-\mathbb{E}\left[\hat{w}_{i}' \right]}{s_e\sqrt{c_{ii}'}}=\dfrac{\hat{w}_{i}-\mathbb{E}\left[\hat{w}_{i} \right]}{s_e\sqrt{c_{ii}}}=t$$

Hence, the significance of the regression coefficients didn't change either. In theory, the same should apply individually scaling the variables but the algebra gets more complicated.


I don't think there's anything close to doing this:

It would be very hard to even define the task objectively, as different humans wouldn't agree about what is credible or not.
It would require a complex system to represent reliable background knowledge... and again people wouldn't agree what should be considered "reliable" or not.
Generally the state of the art in NLP is still far from solving tasks related to Natural Language Understanding in a satisfying way. Juding the credibility of a text requires not only a real understanding of the text but also an ability to reason at a higher level. It's not clear whether this level of AI can ever be reached.

If you find a package which pretends to achieve this task, try to apply it to its own documentation because it's not credible ;)


Removing Categorial Features in Linear Regression

I think using Linear Regression is not a good option as,

This performs very well on numeric variables(categorical -> binary).
Cannot handle Missing Data(suggestible to ignore those records).
When you are trying to predict, By chance a category is not trained then it cannot predict.
For example: A variable has 4 categories and in the random sample it
could pick only 3 categories, if the test has 4th category then it
will throw an error. So, we should be careful when the data is
divided between Test and Train

Now, what are the other algorithms which are available:

Random Forest(you cannot use this when any category variable has more than 52 categories, in your case it shouldn't be an issue)
XGBoost

Do let me know if you need any additional explanations.


Model biased towards low frequency data?

With structured data, you have in general 4 challenges:

(1) Missing data

(2) Outliers

(3) Cardinality

(4) Rare values (as a rule of thumb <5%)

Rare values in categorical variables tend to cause over-fitting, particularly in tree based methods. Ph.D. Data Scientist Soledad Galli has an amazing course on the subject (Udemy: "Feature Engineering". Below a screenshot from her course, but to be fair to her, I'm not going to post the solution.


Encoding features like month and hour as categorial or numeric?

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through.

(More Info)


How can I increase the number of iterations per epoch in MATLAB?

I think you misunderstand the difference between the terms:

Epoch - A single pass over the entire training dataset.

Mini-batch - Number of samples to pass thru the model before performing a single update of its weights.

Iterations - Number of required mini-batches to complete a full epoch.

Example:
you have a training dataset with 256 samples, a mini-batch size of 32 and epoch number 100.
Each training sample will be viewed by the model 100 times (as per the epoch number).
Each epoch will be done in 8 iterations (i.e. the model will be updated 8 times in an epoch).
Each iteration will include 32 training samples passed thru the model. The back propagation algorithm will calculate the required change in the model weights for every single sample, but the actual update will take place on the mean (or sum) of all 32 samples in the mini-batch.

As you said at the end, "iterations per epoch are set automatically according to how many training examples there are present".
There is no reason to alter this parameter, if you want to change the number of passes over the data, use the epoch number.


What are possible approaches to deal with unseparable data even after PCA?

Is your problem unsupervised? If not, the data does not need to be "visually" separable in two dimensions, you can use classifiers in multiple dimensions to create separations for you (linear or nonlinear, just depends on the classifier).

If your problem is unsupervised try other methods for clustering data such a K-Means or tree-based clustering (i.e. hierarchical).


Running examples from scikit-learn tutorials

Seems like you've got to install the module, for example with the pip package manager. Perhaps you could install anaconda, which is highly recommended


Q table creation and update for dynamic action space

It is a finite MDP with states represented as 6 dimensional vectors of integers. The number of discrete values in each index of the state vector varies from 24 to 90.
The action space varies from state to state and goes up to 300 possible actions in some states, and below 15 possible actions in some states.

In combination, this could be over a billion state/action combinations. It is likely to be too much for running a tabular method over. The dynamic actions, changing depending on what the state is are not an issue, but the total size of such a table probably will be.

If I could make some assumptions (just for the purpose of testing the model), I could reduce the states to about 400 and actions to less than 200.

This is a much more tractable number for creating a table. Your total number of records will be at most 80,000.
There are a couple of things worth looking at in more detail:
Number of training examples needed
With a tabular method, as well as constructing the table, you need to collect data for each possible state/action combination. If there is some randomness in the environment, then you need to collect data from each possible state/action combination multiple times. For a table with size 80,000 this might be very fast and easy if you have a fast simulation of the environment. However, it might still not be a feasible size if you can only collect information from a single real environment and the time step represents a day . . .
You will need to check your own numbers to figure out whether this is an issue.
Perhaps even the billion plus items for your full description is not an issue if you have a fast computer and lots of memory to store the table.
Dealing with ragged data in a table
You don't have to do this. If you have enough memory spare and want to take the easy route, you could just use a tensor description (i.e. allow all values in each dimension, and enumerate them from 0 to whatever size). See next section for how you should limit action choice.
However, it may be more efficient to do something about the ragged data.
The simplest way to do this is to generate a unique ID for each allowed state vector, and use that to reference a value in a nested hash structure, e.g. a Python dict of dicts.
A very simple ID generator would be to concatenate your state vector and action labels into a list, then convert that into a string using some kind of join function. Example in Python:
# Empty Q table
q = dict()

# Define an example state, action and value
state = [20,12,56,9,76,30]
action = 176
value = -2.4

# Store an example state, action and value

# This state to state_id conversion should be a re-usable function
state_id = '/'.join([str(x) for x in state])
# Example state_id is '20/12/56/9/76/30'

if state_id in q:
  q[state_id][action] = value
else
  q[state_id] = {action: value}

The easiest way to manage the Q table with this approach is to add entries as and when you need them. Don't pre-populate with zeros, only write action values when they are needed by your agent. That will mean that you only create table entries for state, action pairs that exist.
Filtering action values
You will need code, that you consider to be part of the environment, that knows which actions are valid depending on the current state. This code could return the list of all possible actions. For instance you might define a function allowed_actions(state). You then use that list when working with the Q table:

When looking for a maximising action to drive a policy, look up the whole dictionary of action to value (in q[state_id] and pick the max one)

When you reach a new state that you have not seen before, it may be convenient to populate all possible actions from that state with zeros. Prototype Python code for doing that might look like this:

state_id = '/'.join([str(x) for x in state])
if not state_id in q:
  q[state_id] = { action: 0 for action in allowed_actions(state) }


How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)?

If we avoid a Neural Network,

You should reduce the dimensionality of images e.g. PCA [ Assuming no translational variance among images]
Can try the same for the other two datasets with 50 features
Then try the ML models

If that doesn't work, we may try CNN for the image data.
In that scenario, we will need a mixed architecture of simple Neural Network and CNN.

Check this Paper
Blog on the paper


What is the difference between fasttext and DANs in document classification?

The first most important difference consists in the fact that when using fasttext you are training a language model, i.e. your own embedding vecotrs, while DAN is an architecture (not a language model) that require either a random initializion of embedding layers (which are then trained along with the other layers) or to use pre-trained embeddgings like GloVe (or even fasttext vectors!).
DAN is something that has become popular in some sense (even though I never saw this paper before now). Aveaging embedding vectors of single words before feeding them to a dense layer is a common practice if you need to perform some task at a paragraph or document level.
Just to add some peculiarities of fasttext embeddings, their are trained not for single words, but for n-grams. So during the preprocesing of the corpus from which the embedding are learnt, words are splitted into several chunks of character. For example:
'matter' would become [ma, mat, att, tte, ter, er]
and a unique embedding is then learnt for each chunk like 'ma' or 'mat'. The training follow the same logic of word2vec vectors, which means that from every chunk the model tries to predict context chunks. The advantage of learning embeddings for each chunk relies on the ability of these vectors to learn specific morphological features that classic token-level embeddings usually miss.
If it might help, for a good survey on word embeddings I suggest to take a look at this.


Bagging or Random Subspace Method for Random Forest?

Most modern implementations do both, at least optionally.

sklearn has max_features and bootstrap.
ranger has mtry and replace/sample.fraction.
xgboost's random forest has colsample_bynode and subsample.
h2o has mtries/col_sample_rate_per_tree and sample_rate (and a couple modifiers).


Take a look at the polyglot library. It has polarity lexicons for 136 languages, including Russian.

  The scale of the words’ polarity consisted of three degrees: +1 for
  positive words, and -1 for negatives words. Neutral words will have a
  score of 0.

You can use it like this:

>>> from polyglot.text import Text as T
>>> text = T("это очень плохо. А это намного лучше, даже хорошо!")
>>> text.polarity
0.33333333333333331

>>> for s in text.sentences:
...  s, s.polarity
...
(Sentence("это очень плохо."), -1.0)
(Sentence("А это намного лучше, даже хорошо!"), 1.0)

>>> for w in text.words:
...   print("{:<16}{:>2}".format(w, w.polarity))
...
это              0
очень            0
плохо           -1
.                0
А                0
это              0
намного          0
лучше            1
,                0
даже             0
хорошо           1
!                0


Featuring scaling whole data set before spliting it.

You must use the same scaling factors (μ and σ) for training and test data. The reason that you must use the same scaling factors is that you should compare the data with the same pre-processing. Note that scaling factors (μ and σ) must be computed using only the training data. That is because in practice you are not provided with test data and you just have to evaluate your model on test data.


Size of Output vector from AvgW2V Vectorizer is less than Size of Input data

I think the issue can be one of the two :

A . You have missing value in x_train

B . One of the values in x_train has no word that is there in modelwords.

In both the cases ,

if everyword in modelwords:
     w2v = training_model.wv[everyword]
     count += 1
     sentence += w2v

condition doesn't get satisfied and you end up not addding any new value to sentence


How to interpret Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals?

"How can we determine the strength of association based on the Pearson correlation ?
Briefly speaking, Pearson  (linear) correlation (r ) is not to be interpreted the way you interpret . It simply tells the linear (constant) relationship  between two variables. It does not tell anything  about the causal relation between two variables. It is association measure based on Covariance. Another measure- is called Pearson (sample) correlation coefficient measure which is based on distance. This measure reflects a cause and effect relationship.


There are some suggestions which I think can improve CNN performance:

Input size is (6, 3) so it is not preferable to use MaxPooling layers.
Use padding = 'same' in all the convolution layers which output convolved image of the same dimension as input.
Use Relu or LeakyRelu as activation function.
Use Adam optimizer and tune the learning rate.
Convert some of the Dense layers to 1x1 Convolution with no. of filters = no. of units in dense layer as suggested by sh37211

e.g.

no_of_filters = [32, 64, 64, 32]
kernel_size = [3, 3, 3, 1]

input = Input(shape = (6,3,3), name = "input")
layer_output = [input]
for i in range(4):
    convolution = Convolution2D(no_of_filters[i], kernel_size[i], kernel_size[i], padding = 'same')(layer_output[-1])
    activation = LeakyReLU()(convolution)
    layer_output.append(activation)

flatten = Flatten()(layer_output[-1])
flatten_dropout = Dropout(0.5)(flatten)
fc = Dense(output_dim = 13)
activation = LeakyReLU()(fc)
layer_output.append(activation)
fc = Dense(output_dim = 1, activation = 'tanh')

model.compile(loss = 'mean_squared_error',optimizer = 'adam')


How to calculate a weighted Hierarchical clustering in Orange

Weighted linkage probably does not mean you get to specify weights of features (build the distance matrix yourself!)

Instead this most likely refers to the well-known weighted group average strategy you will find in most textbooks often called WPGMA. There are two different definitions of "average", so this is likely simply the "other" average linkage.


How to divide a dataset for training and testing when the features and targets are in two different files?

You can load the features in a variable (X) and the 2nd column of targets in another variable (y). Then use train_test_split available in sklearn library.

X_train,X_test,y_train,y_test = train_test_split(X,y, test_size=__)

BTW you need to preprocess the data first. There are a lot of NAN values, id and date probably has no value in predicting the target.


Determining whether a Machine Learning model is overfitted with regard to the stability of the features

You need to look for the differences in the training loss and the cross-validation & test losses. If those are low, it means the model performs fairly well. Ideally, train loss should be roughly equal to the cross-validation and test losses. If not, the model may be overfitted.
This difference also hints at an insignificant overlap between the train data points and the cross-validation and test data points. Such features are said to be unstable. In such a case, the model only gets to see the data points in the train data and not those in the cross-validation and test data, and is simply overfitted thereby. It, hence, performs poorly. You can verify the same by computing the percentage of data points present in the cross-validation and test data from those in the train data for different features in your dataset.


How to use embedding to reduce features for a regression problem

Doing it is easy. Simply create a mapping between your 11 values and embeddings of any size. Choosing the values for the embeddings is typically done through training a neural network that embeddings are part of. You could use dimensionality reduction techniques like PCA for instance as an alternative.
Now, embeddings only make sense if they represent a feature of dimensionality $D$ with embeddings of dimensionality between 1 and $D-1$. So in your case, I don't know how much you will benefit from the dimensionality reduction itself. You may experience benefits of replacing sparse with dense features, but this needs to be empirically tested.


xgboost classifier predicted negative probabilities

You have to set the option objective = binary:logistic to get probabilities between 0 and 1, otherwise you only get relative scores.


Independence/correlation test between features (not feautre-label)

This package is very useful, this does the feature engg. for you, gives you the important variables. sample code would be something like this.

    set.seed(123)
    boruta.input <- Boruta(target variable~., data = training_data, doTrace = 2)
    print(boruta.input)

    #plot a graph for better understanding
    plot(boruta.input, xlab = "", xaxt = "n")
    lz<-lapply(1:ncol(boruta.input$ImpHistory),function(i)
    boruta.input$ImpHistory[is.finite(boruta.input$ImpHistory[,i]),i])
    names(lz) <- colnames(boruta.input$ImpHistory)
    Labels <- sort(sapply(lz,median))
    axis(side = 1,las=2,labels = names(Labels),
    at = 1:ncol(boruta.input$ImpHistory), cex.axis = 0.7)

By using this you can get all the important features but the downfall is it takes time if you have more data
In my case my data consists 40 features and 200,000 records it took almost 2 hours but the results were good.

For better understanding you can go through this link

--

I think you can use this test too, in the description it tells that it can be applied on numerical data too.

If you have data which is Nominal data, you can use G-test

Attaching one more link along with this, which consists of tests for different kinds of features. Might be helpful in the future Link

I have a question, why aren't you applying PCA(or any DR Techniques) on the data-set? (assuming that you din't try any DR Techniques)

If you apply DR Techniques, I think you might get better Components which would aid you in getting better results/better understanding of data.

As you mentioned that you were using NB Classifier, why did you choose that, is there any specific reason?
As you know that there are many other Classifiers which might outrun NB Classifier.
In my case I used NB Classifier for twitter sentiment mining, as those are short sentences and NB Classifier best suited for my analysis.

Hope my answer is helpful!

NB: Naive Bayes

DR: Dimensionality Reduction


A convolutional autoencoder will only make sense if you work with images (2D signals) or time series (1D signals). Convolutions identify local patterns in data, if this is not the case in your data it will most likely not solve your problem.
Using pre-trained AE will only help, if it was trained on similar data. Similar data in this case does not refer to the data type, but rather to what the data represents. If you have an AE which was trained to compress images of cats it will not work well on images of chairs, because cats and chairs do not share the same features. Although if you like to compress images of dogs you could use the weights of the AE for cats as a starting point (Transfer Learning).
What kind of loss are you using? MSE or Cross-entropy? Speaking from my experience, using cross-entropy yields better results (although this is problem dependent). Another issue could be vanishing gradients which can happen in very deep networks and with activation functions like sigmoid. What you can do is reduce the depth of your network, replace the sigmoid with ReLU and maybe try a different optimizer.
In any case PCA is a safe bet. It's linear, deterministic, well studied and quicker to use than to train a NN. Whatever method you use, you can use PCA as benchmark to see if your method beats it. Although with the size of your data you may run into memory issues.


Text classification based on n-grams and similarity

First of all, both in your question and your tags, you used Clustering and Classification interchangeably. Be careful as they are totally different problems. I give you a comprehensive tour on how to encode your texts and find similarities so you will be fine doing any of them.

Classic Solution

As you mentioned, n-gram modeling and using Bag of Words variants (for example Tf-Idf) is the classic solution. There are points about this approach:

It does not model the order of words in sentences. So each document/sentence is modeled as a set of vocabularies regardless of the sequence of their positions. With n-gram model where n is larger than 1, you capture some concurrences of the words but still far from understanding the text.
TF-IDF is sparse by nature (there are huge amount of dimensions and a huge fraction of zero values in the matrix). For short documents (like your example sentences), it will be end up in vectors with a few values and a huge amount of zeros in other dimensions. You will need to reduce that dimensionality as I will explain.
It can not handle out of vocabulary words in case you will use it in a classification or text search problem.
The good thing about it is that it is pretty simple and works as a basic algorithm for such a task. The information theoretic concepts behind TF-IDF, beautifully weights words in the document e.g. stop-words will be automatically ignored.

How To

Generate your TF-IDF matrix
Apply Singular Value Decomposition to your matrix to reduce the dimensionality to k dimensions where k is the number of different topics in your corpus. You get back a $m\times k$ matrix back which is the reduced version of your TF-IDF. $n$ is the number of your documents and $k$ the number of different topics in it
You need to simply run your clustering algorithm on this matrix and get your results.

What you did above is called Latent Semantic Analysis. The best topic modeling out of BoW model.

How To Choose $k$
Just visualize! Use different values for $k$ and using t-SNE or UMAP (actually use UMAP! I just mentioned t-SNE for sake of completeness of my answer) for visualizing your data in 2 dimensions and try to guess what is the best $k$. Please note that Clustering problem has no evaluation. At the end of the day it is a smart search.

Classic Solution with Modern Tools

Increase the usage of UMAP in the above pipeline. It is a pretty strong nonlinear dimensionality reduction and visualization algorithm. You can drectly feed your TF-IDF to UMAP and see the results or after applying SVD, try to use UMAP for visualization and dimensinality reduction.

Modern Approaches to NLP/NLU

Neural-based language modeling has a long history which is out of scope here, but in 2018, Google released an algorithm which was a breakthrough in NLP and is widely used nowadays. It is called BERT. A startup called HuggingFace implemented an easy-to-use software package for BERT in Pytorch and Tensorflow called Transformers. For your use-case, I recommend another package from Technical University of Darmstadt in Germany called sentence-transformers which is based on SBERT (Sentence-BERT). You can easily find sample codes for document similarity search in their repo, for example this is an exact solution to your problem.

In case of further questions please just drop a comment.

Good Luck!


White circles extraction from a little darker background

Follow the following steps:

Step 1: use an averaging filter to minimise the noise present in the image

Step 2: threshold the image such that pixels with RGB values within [0, 0, 0] and [1, 1, 1] are set to 0 and rest as 1. This step will lead you to generate a binary image.

Step 3: use Hough transform in OpenCV to detect circles. Tune the parameters of the Hough transform to get your desired output.

Check this out for more details of Hough transform for circles: https://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html


feature importance between two data distributions

My two cents:
I can think of several ways calculating specific terms repressiveness:

1) Calculating rate of documents(tweets/posts) where these terms appear more than
1(or X > 1) times. You can decide on X by looking at the distribution of frequency per document.
2) Calculating terms TF-IDF score per each document and aggregating across all documents. Aggregating across all document  depends on your need and what you mean by representative.
3) Calculating terms appearance rate - how many times these terms appear divided by total number of words in the corpus(possibly after subtracting stop words).


This is actually slightly similar to the problem that insurance companies face except that it seems like your loss costs are known.  Insurers have some probability of loss and then, given loss, the magnitude of the loss follows some distribution.  The cost to the insurer is dependent on both and they tend to be inversely related (lower losses are more likely than higher ones.)

In your case, the value is known so you don't need to predict it the way insurers need to predict losses so you could simply:

Model the probability (phat)
Multiply the predicted probability by the known value (score = phat * value)
Recommend based on the resulting score

Insurance companies typically do the same thing in calculating premiums except that they also need to model value.  They sometimes model the two components jointly but typically they have separate models for frequency and severity and then just multiply them together to determine how much premium they should charge somebody.


Hierarchical Clustering customized Linkage function

Fork sklearn and implement it yourself! The linkage function is referenced in cluster/hierarchical.py as

  join_func = linkage_choices[linkage]

and

  coord_col = join_func(A[i], A[j], used_node, n_i, n_j)

If you have time, polish your code and submit a pull request when you're done.


Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a concise word similarity.


Training 128x128 autoencoders on 512x512 images, produces strange gridline after recombining

The problem is that the model has no idea what the image should look like outside of the 128x128 crop you feed it with, so it fails to extrapolate appropriately. One simple fix is to train the model on bigger (and overlapping!) crops, like 144x144, and then crop out 8 pixels from each side.

However, even this approach does not guarantee absence of any visible edges, as it's hard for the model to enforce this "boundary consistency" without direct access to its output from the previous patch. This could be approached with more powerful decoders, like PixelRNN / PixelCNN


The most straightforward approach to understand their proposal is to look at their source code, which is linked from the article: https://github.com/yangji9181/ClusChurn/blob/master/model.py#L35

It reveals that the "embedding" is obtained by means of a mere linear layer.


You can convert series to the list and do the comparison, the order must have remained same any way.

or you if you want to keep the reference you can convert list to series using

pd.Series(list, index= y_test.index)


With tensorflow, currently the most straightforward and easy way to get persistence for your model is to use a tf.train.MonitoredTrainingSession. You just need to use it instead the normal tf.Session() that is frequently used. This an illustrative Python snippet:

with tf.train.MonitoredTrainingSession(checkpoint_dir='/tmp/mymodel',
                                       save_summaries_secs=600) as sess:
   _ = sess.run(train_op, feed_dict={...})

With this, your model is automagically saved every 600 secs in /tmp/mymodel and restored the next time you restart the program.


Intuitive explanation of Lovasz Softmax loss for Image Segmentation problems

The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

The loss can be optimized on its own, but the optimal optimization hyperparameters (learning rates, momentum) might be different from the best ones for cross-entropy. As discussed in the paper, optimizing the dataset-mIoU (Pascal VOC measure) is dependent on the batch size and number of classes. Therefore you might have the best results by optimizing with cross-entropy first and finetuning with our loss, or by combining the two losses.

Here is the Implementation of Lovasz Softmax Loss in Pytorch & Tensorflow.

A step by step explanation of the important steps of the code:

Step 1: compute the errors of the predictions:

signs = 2. * labels.float() - 1.
errors = (1. - logits * Variable(signs))
errors_sorted, perm = torch.sort(errors, dim=0, descending=True)

The "errors" vector is positive when the predictions are false and negative when the predictions are correct. Nothing special here. Note that a margin of (1) is used here, which means those correct predictions that have a margin lower than 1 is considered as an error.

Step 2: Lovasz extension w.r.t sorted errors

gts = gt_sorted.sum()
ntersection = gts - gt_sorted.float().cumsum(0)
union = gts + (1 - gt_sorted).float().cumsum(0)
jaccard = 1. - intersection / union

At this point, the vector "jaccard" contains the evolution of the Jaccard index with respect to the sorted errors. It varies between 0 and the actual Jaccard index of the prediction.

Step 3: gradient of the Lovasz extension w.r.t sorted errors

The next step of the code consists of computing the gradient of this vector. It is computed by a backward difference:

jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]

This vector says what is the effect of an error on the evolution of the Jaccard index. The idea is to minimize the errors that penalize the Jaccard index the most.

Step 4: Compute the loss

In the last step of the code, the actual loss is computed as the dot product between the error vector and the gradient of the Lovasz extension. Relu is used because only a positive part of the error vector contains prediction errors.

loss = torch.dot(F.relu(errors_sorted), Variable(grad))

More Info


Decision Tree : how to determine target in a model with no labels?

Use Clustering under unsupervised learning. That will categorize the customer based on similar parameters. You can define the number of cluster you need to form, in you case it is two(trustworthy and not). If there are more features it will be more helpful for the algorithm.
This might help.
https://towardsdatascience.com/an-introduction-to-clustering-algorithms-in-python-123438574097


What exactly is convergence rate referring to in machine learning?

A rate is always a gain per some time/step. A rate can exist even if the maximum is never reached. In supervised learning a loss function is defined, which is expected to have a global maximum, that we try to reach by gradient descent. How much closer we get with each timestep/iteration/epoch/batch is the rate of convergence.


How to resolve country and nationality entities?

I believe that Lemmatisation is the right thing. Anyways another way to do it is to use WordNet. For a word which its POS is Noun you can query if it has a member holonym, this feature will show the country which it belongs. For instance, Canadian -> Canada. Then you have to be careful, cause if you take the member holonym of Canada that will be the British Commonwealth. I guess you could have a Levenshtein distance threshold to ignore these.

You can have a look on the online web app.


How does Gradient Descent and Backpropagation work together?

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases.

Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the derivatives and gradient descent is the process of descending through the gradient, i.e. adjusting the parameters of the model to go down through the loss function.

Back-propagation is called like this because to calculate the derivative you use the chain rule from the last layer (which is the one directly connected to the loss function, as it is the one that provides the prediction) to the first layer, which is the one that takes the input data. You are "moving from back to front".

In gradient descent one is trying to reach the minimum of the loss function with respect to the parameters using the derivatives calculated in the back-propagation. The easiest way would be to adjust the parameters by substracting its corresponding derivative multiplied by a learning rate, which regulates how much you want to move in the gradient direction. But there are some more advanced algorithms like for example ADAM.


What is a simple image classification ANN architecture and a simple data set?

I'd advice going through cs231n course, which covers this.

A simple but good architecture is using a feed forward net with one hidden layer and ReLU.

As for dataset - CIFAR-10 is good, if you don't like MNIST.

Tensorflow is quite easy to use framework.


A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected?

This is not answerable in general. You don't even know the skewness or standard deviation around the median without missing data, so you can't compare those quantities. Without more information about the distribution there is no way to figure out how much data is missing, or where in the distribution the missing data is located.


best approach for CNN training with multiple subcategories and one category

Okay, so I used the custom loss function to optimize both ways simultaneously:
 - binary rejected vs approved
 - multiple categories
And using the coefficient argument to balance between two losses.
Thats it


How to build a supervised artificial neural network?

Note that I am cross-posting this answer because the question is cross-posted, and one of the questions will likely be deleted. If this is a breach of etiquette, please let me know. Cross-post is here.

This question has two parts, at least implicitly.  The second part is, generally speaking, how to train and predict from neural networks (in Python).  The first part is (implicitly) how do we build a model that predicts (correctly) which person won a presidency based on demographics.

The answer to the second question is quite easy to state, but harder to do.  It boils down to the rather unhelpful statement:  learn about neural networks, and use that knowledge.  Google for tutorials on neural networks. I would make recommendations, but I don't know your mathematical, statistical, and computer science background, and thus cannot really make suggestions.

The first question, however, I can answer, although likely not in the way you'd like.  Just using the data you've listed, it's highly unlikely you'll get any sort of accuracy out of your neural networks, or any other model you fit.  Why?  For several reasons.

The first is that unless you can always classify the president into one of two (or more) distinct groups, as you can in the US (Democrat, Republican), your data are changing all the time.  You can't classify anything if it doesn't have a class, and that's difficult in some countries.

Secondly, even if there are distinct classes, demographic data is likely not a set of good predictors.  Demographic data changes depending on how good the local governance is, as well as the overall governance.  You might, might be able to code that in somehow.  But demographic data also changes due to a lot of other things. When it comes down to voting, people choose to vote based on a bunch of things, and their demographic status is only one factor.

The third reason is more a caution than anything.  Neural networks and all kinds of machine learning/statistical modelling methods can give good predictions with the right kind of data.  Something like predicting a president is relatively difficult, especially in smaller countries.  (Nate Silver did it well in the US, but there was a lot more data there.)  And even if you've got great data and a great model, there's invariably something you miss.  George Box said it right:  "All models are wrong, but some are useful."  When you mention predicting correctly, all sorts of alarm bells go off in my head.  The problem with trying for "correct" prediction is that there are no guarantees your prediction will be correct, often even in simple cases.  No model is perfect.


How does GlobalMaxPooling work on the output of Conv1D?

Apparently I forgot how the convolution works. The input is multiplied element wise with the filter weights and the products are then summed. That's how a single value is obtained on the output.


Visualizing a graph with a million vertices

I also suggest Gephi software (https://gephi.github.io), which seems to be quite powerful. Some additional information on using Gephi with large networks can be found here and, more generally, here. Cytoscape (http://www.cytoscape.org) is an alternative to Gephi, being an another popular platform for complex network analysis and visualization.

If you'd like to work with networks programmatically (including visualization) in R, Python or C/C++, you can check igraph collection of libraries. Speaking of R, you may find interesting the following blog posts: on using R with Cytoscape (http://www.vesnam.com/Rblog/viznets1) and on using R with Gephi (http://www.vesnam.com/Rblog/viznets2).

For extensive lists of network analysis and visualization software, including some comparison and reviews, you might want to check the following pages: 1) http://wiki.cytoscape.org/Network_analysis_links; 2) http://www.kdnuggets.com/software/social-network-analysis.html; 3) http://www.activatenetworks.net/social-network-analysis-sna-software-review.


What is the point of using MissingIndicator in Scikit-learn?

Good question. Here is my take on it:

The following piece code creates the union of both the imputed features and the indication of which ones were imputed.

transformer = FeatureUnion(
     transformer_list=[
         ('features', SimpleImputer(strategy='mean')),
         ('indicators', MissingIndicator())])

I can see it becoming useful for a learner that may learn to not rely as much on features that are often missing. Or perhaps, the fact that some features are missing at all can be helpful for the learner. As, missing values can mean that something happened differently in the data collection, which can be useful to make a prediction.

Let's say I have the following problem:

I want to classify users based on their watching habits on my fancy streaming platform. Users have to rate a movie thumbs up or thumbs down. Of course, not all my users will watch all movies, so I'll have blank values. By using let's say the SimpleImputer, I'll basically make an educated guess on how they would have liked the movies they didn't watch. That can be useful as then I can compare users across all movies. However, knowing which movies the user has watched altogether is probably as informative. This is what the MissingIndicator provides you.


Why is the dimensionality reduction and/or feature selection techniques are considered to be not always effective?

This is helpful when the data is not understandable and you don't have any data dictionary or you have too many columns which doesn't make any sense even after through investigation, then it is wise to go for Dimensionality Reduction.

There is very good chance that you loose information at granular level, for example you gave 100 columns and you got 10 PC's(Applied PCA) which can explains most of the Data. As you can only get that much out of that technique.

Feature Selection: If you miss even a single feature which is significant WRT target variable but you overlooked that feature then your model might not be able explain the most, you are even satisfied with the result but still there is window of improvement. You should be very careful and you need check as many times(many iterations) as possible to build a good/decent model and get the best results out of it.

I hope this may help you.


Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

To replace by column means, an easy approach would be to use the base R function colMeans. Let's say you have a data frame df.

1) If you want to replace the NAs per column one by one, you could try this:

df <- sapply(df, function(x)ifelse(is.na(x), mean(x, na.rm=TRUE), x))

2) If you want to replace all NAs in one go, you could try this:

df <- ifelse(is.na(df), rep(colMeans(df, na.rm=TRUE), rep(nrow(df), ncol(df))), unlist(df))


How to implement clustering on categorical feature vectors in R?

You should use dummy variables and then you can toss it directly into K-means. If you have a lot of categories the efficient way to do this is through a one-hot-encoding (sparse-encoding).

Here's a little demo using clustering and then using the clusters in a regression model. In general, you should avoid doing that but it's illuminating in this case.

library(glmnet)
library(Matrix)
n <- 1e5
nclusters <- 5
set.seed(420)
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.,data=ls)
print(head(xs))
# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- rep(1, dim(xs)[2])
# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))
# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.,data=clusterpred)
model <- glmnet(y=ys, x=xs, alpha=0)
cmodel <- glmnet(y=ys, x=cxs, alpha=0)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
# Looking at the difference RMSEs
print(sqrt( sum( (ys-yhat)**2 )))
print(sqrt( sum( (ys-yhatc)**2 )))


Classification procedure using Minimum Redundancy Maximum Relevance (MRMR)

If you are not using the quantized values, what was the point of using the MRMR? from what I understand, it helps you "select a feature subset set that best characterizes the statistical property of a target classification variable" (sounds like PCA). So after you have done that, it helps your classifier to focus on the more important features.


how to get the Polysemes of a word in wordnet or any other api?

There are several third-party Java APIs for WordNet listed here: http://wordnet.princeton.edu/wordnet/related-projects/#Java

In the past, I've used JWNL the most: http://sourceforge.net/projects/jwordnet/

The documentation for JWNL isn't great, but it should provide the functionality you need.