<p>Are there any papers that deal with something like the following scenario?</p>

<p>Given a large corpus of text, find all n-tuples of documents that are semantically identical.</p>

<p>This is not a categorization problem as if it were to be treated as one, there would be an indeterminate (until its finished) number of categories, with the vast majority of categories only having one document in them.</p>

<p>In fact, I don't think it is a part of any of <a href="http://en.wikipedia.org/wiki/Natural_language_processing#Major_tasks_in_NLP" rel="nofollow">these</a> major areas on NLP.</p>

<p>Additionally, I believe that all purely statistical approaches are doomed to fail here as the training set cannot cover every post.</p>

<p>So, are there any papers on this topic?</p>

CC BY-SA 3.0
2012-08-10T11:15:51.767Z
The most obvious approach would be to learn some topic model ([LSA](http://en.wikipedia.org/wiki/Latent_semantic_analysis), [LDA](http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation), [Semantic Hashing](http://www.utstat.toronto.edu/~rsalakhu/papers/semantic_final.pdf)) on your documents and then use knn on the latent document representations to find similar documents.
alto
CC BY-SA 3.0
2012-08-11T23:53:37.200Z
I believe your question needs clarification: what is "semantically identical"? What do you mean by "post" and why should a training set cover every post. But more than that, your question is not theoretical computer science, it's more practical data mining, so I'd consider asking at stats@SE or CS@SE (once you clarify the vague points). Finally, google topic modeling.
Sasho Nikolov
CC BY-SA 3.0
2012-08-12T01:02:00.677Z
@SashoNikolov, in order: Has the same intent, for example something like a body of text, and a paraphrased version. Post just means some text. A training set should cover every post because how else will it know anything about what will match with that post? It is not really about data mining, I am looking for papers that relate to this (not how to implement the papers for example).
soandos
CC BY-SA 3.0
2012-08-12T08:35:49.010Z
Related questions on [cs.SE]: [1](http://cs.stackexchange.com/q/2955/98), [2](http://cs.stackexchange.com/q/2922/98).
Raphael
<reference-request>
Semantic similarities in text
soandos
# Semantic similarities in text

(Asked by: username_0 on 2012-08-10)

Are there any papers that deal with something like the following scenario?

Given a large corpus of text, find all n-tuples of documents that are semantically identical.

This is not a categorization problem as if it were to be treated as one, there would be an indeterminate (until its finished) number of categories, with the vast majority of categories only having one document in them.

In fact, I don't think it is a part of any of [these](http://en.wikipedia.org/wiki/Natural_language_processing#Major_tasks_in_NLP) major areas on NLP.

Additionally, I believe that all purely statistical approaches are doomed to fail here as the training set cannot cover every post.

So, are there any papers on this topic?

**Tags:** reference-request
> **Comments:**
> **username_1**: The most obvious approach would be to learn some topic model ([LSA](http://en.wikipedia.org/wiki/Latent_semantic_analysis), [LDA](http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation), [Semantic Hashing](http://www.utstat.toronto.edu/~rsalakhu/papers/semantic_final.pdf)) on your documents and then use knn on the latent document representations to find similar documents.  
> **username_2**: I believe your question needs clarification: what is "semantically identical"? What do you mean by "post" and why should a training set cover every post. But more than that, your question is not theoretical computer science, it's more practical data mining, so I'd consider asking at stats@SE or CS@SE (once you clarify the vague points). Finally, google topic modeling.  
> **username_0**: @SashoNikolov, in order: Has the same intent, for example something like a body of text, and a paraphrased version. Post just means some text. A training set should cover every post because how else will it know anything about what will match with that post? It is not really about data mining, I am looking for papers that relate to this (not how to implement the papers for example).  
> **username_3**: Related questions on [cs.SE]: [1](http://cs.stackexchange.com/q/2955/98), [2](http://cs.stackexchange.com/q/2922/98).  

---

## 0 answers

