<p>I think you misunderstood, in a lot of ways, what Haag &amp; Nagel (2000; what you call Paper 1) did and how Arens, Gerber &amp; Nagel (2008; Paper 2) extended it. Fig. 1 of AGN08 is a good summary of HN00. What HN00 did was build a system that could watch a video of an intersection, detect cars, and translate the car behavior into a conceptual framework. As inspiration for their system, they used their idea of how humans represent the task:</p>

<blockquote>
  <p>ﬁve levels of representation seem to be involved: (i) a representation of the geometry of spatiotemporal developments in the road traﬃc scene, comprising both a 2D-one in the image plane and a 3D-one relating to the depicted scene, (ii) a representation of driving maneuvers closely coupled to particular traﬃc situations, (iii) a conceptual representation of visible bodies, their attributes, and their elementary movements, (iv) generic conceptual representations of spatiotemporal body conﬁgurations and their expected temporal developments, and (v) one or more versions of a natural language representation of developments centred around the current point in time.</p>
</blockquote>

<p>In other words, the goal of HN00 was to look at a 2D image of an intersection, from it construct a 2D/3D representation of the scene. In that scene identify and label objects and describe them in a conceptual language called SIT++. Once in that conceptual representation (as situation trees) they could conduct logical inference (using Fuzzy Metric
Temporal Horn Logic) on their representation in order to decide what the agents they identified are trying to do.</p>

<p>Note that HN00 involved no natural language processing (NLP) at all. Although they did have to use lots of pattern recognition and various machine learning algorithm that would be familiar to NLP practitioners. However, their domain was transforming a visual scene into a conceptual (not natural language) internal representation.</p>

<p>How did AGN08 extend beyond this? They changed what they wanted to do. Their task was not simply to view a scene and transform it into an internal representation, but to then output that internal representation in a <strong>natural language description</strong>. Thus, they were adding a <a href="http://en.wikipedia.org/wiki/Natural_language_generation" rel="nofollow noreferrer">natural language generation</a> system to HN00. Generating natural language from an internal representation is obviously an important part of NLP.</p>

<p>In the process of adding this functionality, AGN08 had to extend the internal representation in several ways. This was due to the fact that more internal information was required to generate good natural language output, and because they wanted to deal with more complex scenes than HN00. The paper focuses on this aspect of the work (extending the internal representation) and only tangentially touches on natural language output. They go into detail of the natural language output in:</p>

<p>R. Gerber, Naturlichsprachliche Beschreibung von Straßenverkehrsszenen durch Bildfolgenauswertung. Dissertation,Fakultat fur Informatik der Universitat Karlsruhe (TH), Karlsruhe, January 2000</p>

<p>Unfortunately, I am not willing to learn German and read a whole thesis in order to give you a more complete answer about the details. Before you try to do that yourself (hopefully you already know German) or look into more recent papers, I recommend learning some basics of NLP. A good source is the following question:</p>

<p><a href="https://linguistics.stackexchange.com/q/1320/33">Looking for a good beginners reference to learn computational linguistics</a></p>

CC BY-SA 3.0
2012-06-11T18:48:41.723Z
Thank you for taking the trouble in explaining in such detail.As I assume, these papers are diffucult to implement for my task and are not related to what I am interested in. Would you suggest,in general how should I start or what should be the starting point and steps in building a surveillance system for situational assessment based on natural language.
chk
CC BY-SA 3.0
2012-06-09T02:58:13.473Z
2012-06-09T02:58:13.473Z
2017-04-13T12:54:11.280Z
Artem Kaznatcheev
<p>I am stuck trying to learn how to use video processing as explained in the linked papers in the area of human behavior detection or traffic surveillance (any kind of monitoring activity). In particular, I do not know how to formulate the problem using any available models.</p>
<p>I have found two papers (Haag &amp; Nagel (2000; Paper 1) and Arens, Gerber, &amp; Nagel (2008; Paper 2)) which illustrate, with an example, using natural language for traffic monitoring by using image sequences of videos. My question are</p>
<ol>
<li>Under what assumptions are the actions generated?What is the advantage of nlp here or in any application apart from the fact that it gives a hierarchical representation of information/knowledge?</li>
<li>How are the actions and graphs (labelled as Situation Graph Trees by the author) generated (under what assumptions) in Fig2 of Paper 1 and Fig1 of Paper 2?</li>
<li>Has the Table5 in paper1: Incremental recognition of traffic situations from video image sequences and subsequent tables been generated using some programming tools?</li>
</ol>
<h3>References</h3>
<p>Haag, M., &amp; Nagel, H.-H. (2000) &quot;<a href="http://publica.fraunhofer.de/documents/N-7542.html" rel="nofollow noreferrer">Incremental recognition of traffic situations from video image sequences</a>&quot;. <em>Image and Vision Computing</em> 18(2): 137-153.</p>
<p>Arens, M., Gerber, R., &amp; Nagel H.-H. (2008) &quot;<a href="http://www.arens-online.info/publications/MA_RG_HHN_IVC2008.pdf" rel="nofollow noreferrer">Conceptual representations between video signals and natural language descriptions</a>&quot;. Image and Vision Computing 26: 53–66.</p>

CC BY-SA 3.0
2012-06-05T19:22:50.580Z
I can't understand what you are asking. Also, if you don't what "stuff" you are doing, how can you ask for open-source tools for doing it? Figure out what you want to do and how to phrase it precisely and accurately in the form of one question, then based on what you learn in your initial search ask a specific well-focused questions. At this point I have to vote to close as "not a real question".
Artem Kaznatcheev
CC BY-SA 3.0
2012-06-05T20:27:17.817Z
Your question is a confusing because of some unclean grammar as well as lack of focus. I think you should first try to separate this into two questions. One about the visual classification (maybe you want to look into sign language translation, visual recognition of gestures etc.). After you find something more interesting you want focus on, it will be easier to ask about open source tools for it.
Vielle
CC BY-SA 3.0
2012-06-05T20:34:31.113Z
For example work on gesture recognition http://www.sciencedirect.com/science/article/pii/S0262885606002897
Vielle
CC BY-SA 3.0
2012-06-06T19:13:35.327Z
I edited the question to improve grammar and formatting so that OP could then edit it to make it more precise. There are actually two questions, in both the scope of desired robot cognition being overly broad. Some more limited scope of robot cognition would be IMHO a more fruitful point of departure for such a difficult matter.
nrz
CC BY-SA 3.0
2012-06-06T21:14:53.177Z
OP, I think you should familiarize yourself with the basics of NLP. For some introductory resources, see [this question and answers on ling.SE](http://linguistics.stackexchange.com/q/1320/33)
Artem Kaznatcheev
CC BY-SA 3.0
2012-06-06T21:20:21.127Z
Also, Fig 1 of Paper 2, is just a summary (in figure form) of the key ideas of Paper 1. I am not sure what you are asking.
Artem Kaznatcheev
<cognitive-modeling><computational-modeling><agent-based-modeling><nlp>
Using natural language processing for traffic monitoring from video
chk
# Using natural language processing for traffic monitoring from video

(Asked by: username_0 on 2012-06-05)

I am stuck trying to learn how to use video processing as explained in the linked papers in the area of human behavior detection or traffic surveillance (any kind of monitoring activity). In particular, I do not know how to formulate the problem using any available models.

I have found two papers (Haag & Nagel (2000; Paper 1) and Arens, Gerber, & Nagel (2008; Paper 2)) which illustrate, with an example, using natural language for traffic monitoring by using image sequences of videos. My question are

1. Under what assumptions are the actions generated?What is the advantage of nlp here or in any application apart from the fact that it gives a hierarchical representation of information/knowledge?
2. How are the actions and graphs (labelled as Situation Graph Trees by the author) generated (under what assumptions) in Fig2 of Paper 1 and Fig1 of Paper 2?
3. Has the Table5 in paper1: Incremental recognition of traffic situations from video image sequences and subsequent tables been generated using some programming tools?

### References

Haag, M., & Nagel, H.-H. (2000) "[Incremental recognition of traffic situations from video image sequences](http://publica.fraunhofer.de/documents/N-7542.html)". *Image and Vision Computing* 18(2): 137-153.

Arens, M., Gerber, R., & Nagel H.-H. (2008) "[Conceptual representations between video signals and natural language descriptions](http://www.arens-online.info/publications/MA_RG_HHN_IVC2008.pdf)". Image and Vision Computing 26: 53–66.

**Tags:** cognitive-modeling, computational-modeling, agent-based-modeling, nlp

> **Comments:**
> **username_1**: I can't understand what you are asking. Also, if you don't what "stuff" you are doing, how can you ask for open-source tools for doing it? Figure out what you want to do and how to phrase it precisely and accurately in the form of one question, then based on what you learn in your initial search ask a specific well-focused questions. At this point I have to vote to close as "not a real question".  
> **username_2**: Your question is a confusing because of some unclean grammar as well as lack of focus. I think you should first try to separate this into two questions. One about the visual classification (maybe you want to look into sign language translation, visual recognition of gestures etc.). After you find something more interesting you want focus on, it will be easier to ask about open source tools for it.  
> **username_2**: For example work on gesture recognition http://www.sciencedirect.com/science/article/pii/S0262885606002897  
> **username_3**: I edited the question to improve grammar and formatting so that OP could then edit it to make it more precise. There are actually two questions, in both the scope of desired robot cognition being overly broad. Some more limited scope of robot cognition would be IMHO a more fruitful point of departure for such a difficult matter.  
> **username_1**: OP, I think you should familiarize yourself with the basics of NLP. For some introductory resources, see [this question and answers on ling.SE](http://linguistics.stackexchange.com/q/1320/33)  
> **username_1**: Also, Fig 1 of Paper 2, is just a summary (in figure form) of the key ideas of Paper 1. I am not sure what you are asking.  

---

## 1 answers

### Answer by username_1 on 2012-06-09. Score: 3

I think you misunderstood, in a lot of ways, what Haag & Nagel (2000; what you call Paper 1) did and how Arens, Gerber & Nagel (2008; Paper 2) extended it. Fig. 1 of AGN08 is a good summary of HN00. What HN00 did was build a system that could watch a video of an intersection, detect cars, and translate the car behavior into a conceptual framework. As inspiration for their system, they used their idea of how humans represent the task:

> ﬁve levels of representation seem to be involved: (i) a representation of the geometry of spatiotemporal developments in the road traﬃc scene, comprising both a 2D-one in the image plane and a 3D-one relating to the depicted scene, (ii) a representation of driving maneuvers closely coupled to particular traﬃc situations, (iii) a conceptual representation of visible bodies, their attributes, and their elementary movements, (iv) generic conceptual representations of spatiotemporal body conﬁgurations and their expected temporal developments, and (v) one or more versions of a natural language representation of developments centred around the current point in time.

In other words, the goal of HN00 was to look at a 2D image of an intersection, from it construct a 2D/3D representation of the scene. In that scene identify and label objects and describe them in a conceptual language called SIT++. Once in that conceptual representation (as situation trees) they could conduct logical inference (using Fuzzy Metric
Temporal Horn Logic) on their representation in order to decide what the agents they identified are trying to do.

Note that HN00 involved no natural language processing (NLP) at all. Although they did have to use lots of pattern recognition and various machine learning algorithm that would be familiar to NLP practitioners. However, their domain was transforming a visual scene into a conceptual (not natural language) internal representation.

How did AGN08 extend beyond this? They changed what they wanted to do. Their task was not simply to view a scene and transform it into an internal representation, but to then output that internal representation in a **natural language description**. Thus, they were adding a [natural language generation](http://en.wikipedia.org/wiki/Natural_language_generation) system to HN00. Generating natural language from an internal representation is obviously an important part of NLP.

In the process of adding this functionality, AGN08 had to extend the internal representation in several ways. This was due to the fact that more internal information was required to generate good natural language output, and because they wanted to deal with more complex scenes than HN00. The paper focuses on this aspect of the work (extending the internal representation) and only tangentially touches on natural language output. They go into detail of the natural language output in:

R. Gerber, Naturlichsprachliche Beschreibung von Straßenverkehrsszenen durch Bildfolgenauswertung. Dissertation,Fakultat fur Informatik der Universitat Karlsruhe (TH), Karlsruhe, January 2000

Unfortunately, I am not willing to learn German and read a whole thesis in order to give you a more complete answer about the details. Before you try to do that yourself (hopefully you already know German) or look into more recent papers, I recommend learning some basics of NLP. A good source is the following question:

[Looking for a good beginners reference to learn computational linguistics](https://linguistics.stackexchange.com/q/1320/33)

> **Comments:**
> **username_0**: Thank you for taking the trouble in explaining in such detail.As I assume, these papers are diffucult to implement for my task and are not related to what I am interested in. Would you suggest,in general how should I start or what should be the starting point and steps in building a surveillance system for situational assessment based on natural language.  

---

