<p>This question cannot be answered in the form in which you asked it both because of the limits of current neuroscientific theories and methodologies when it comes to determining the structures of complex neural representations (although we have made headway in a few cases such as place cells and grid cells), and because neural representations are not really analogous to our colloquial concepts of simple mappings from numbers to images (ex. pixel, vector, wavelet, etc. bases for digital image representation).</p>

<p>I would encourage you to think, not only in terms of these representational formats privileged by Von Neumann style computer architectures (i.e. normal computers) with separate processing and digital memory systems, but in terms of a neural computer with distributed computation and memory. Neuroscientists typically think of neural representation in visual perception as occurring at a number of stages of increased abstraction. <a href="https://en.wikipedia.org/wiki/Retinal_ganglion_cell" rel="noreferrer">At the retina</a>, the image is represented roughly in terms of what you might think of as pixels, but it is really just a set of photosensors and attached neurons that are activated due to light hitting the retina in different places at different frequencies and amplitudes. In <a href="https://en.wikipedia.org/wiki/Visual_cortex#Primary_visual_cortex_.28V1.29" rel="noreferrer">V1 or primary visual cortex</a>, the representation is in terms of what can be thought of as "edge detectors." In V2, the cells are tuned for a variety of slightly more complex properties such as  such as orientation, spatial frequency, color, and <a href="https://en.wikipedia.org/wiki/Binocular_disparity" rel="noreferrer">binocular disparity</a> (an important piece of information for 3D perception.) As you go up through V3, V4, V5, and V6, the representations becomes more and more complex until they contain information about the structure of the underlying concept itself. </p>

<p>The 3-dimensionality does not come in until the visual information from both eyes is combined with top-down information about how things tend to be in the world (which comes from other senses and experiences as well) in order to infer the likely shape of the object. It is unclear whether there is any place in the brain where an image is represented explicitly as a 3D model of a visual object. More likely the raw perceptual data such as colors and patterns is represented in one area and is associated with the more object-centric structural representation in another area and this association causes the co-activation of all the neurons relevant to the perception of the object in it's three dimensional form. The information is neither represented in one area of the brain nor in one scheme but is distributed across a large number of neurons with different degrees of activation. </p>

<p>We can also think of this in terms of the contemporary theory of enactive perception, which states that the brain represents information in terms of state-action-observation contingencies. In this theory, there is no "image" present at all, but only a set of neural activations which correlate with prior experience (where the correlations are stored in terms of associative links between neurons creating a causal activation path along which "information" travels) and relationships between potential actions (either low level motor actions of the retina or other muscles, or higher level actions such as taking a step forward) and their likely consequences (both low level perceptual consequences such as the expected changes in the sensory data when an action such as moving the head is taken and higher level perceptual and conceptual consequences such as a cup falling when you choose to open your hand and the associated experiences.) In this more comprehensive picture, visual information cannot be strictly separated from from conceptual and multi-sensory information and so three-dimensional visual information is represented both in terms of neural activations causally associated with the structure of the perceived object as well as the activations associated with all the other conceptual beliefs and perceptual expectations that relate.</p>

<p>If you want to read more, these sources should be a good start: <a href="http://s3.amazonaws.com/academia.edu.documents/39939727/Transient_and_sustained_activity_in_a_di20151112-5796-vgowyp.pdf?AWSAccessKeyId=AKIAJ56TQJRTWSMTNPEA&amp;Expires=1466455672&amp;Signature=duYuMAjKs9%2B8UnJE6t8CgU1ZdXU%3D&amp;response-content-disposition=inline%3B%20filename%3DTransient_and_sustained_activity_in_a_di.pdf" rel="noreferrer" title="Neuroscience paper">distributed neural memory</a>, <a href="http://www.jneurosci.org/content/25/19/4806.full" rel="noreferrer" title="Distributed neural representation of a higher order process">distributed neural representation of a higher order process</a>, <a href="https://mitpress.mit.edu/books/action-perception" rel="noreferrer" title="Action in perception">perception as associations between actions and observations</a></p>

<p>There are also a few sources that offer direct experimental evidence towards your question, but do not provide as much of a conceptual introduction to neural computation and representation. See <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2883007/" rel="noreferrer">"Neural computations underlying depth perception"</a>, <a href="http://www.nature.com/nrn/journal/v8/n5/full/nrn2131.html" rel="noreferrer">Binocular depth perception and the cerebral cortex</a> as a start.</p>

<p>Your question also makes reference to rotating an object "in the mind's eye." There is also a very large body of research on this phenomenon, which is called "mental rotation." <a href="https://en.wikipedia.org/wiki/Mental_rotation" rel="noreferrer">The wikipedia article</a> is a good place to start but there is a lot more to say about this interesting phenomenon, so feel free to ask a related question if you are curious!</p>

CC BY-SA 3.0
2016-06-20T23:51:53.293Z
This is a fantastic answer, and I especially like the enactive perception interpretation, which would imply, for example, that the visual data attached to a human being is a combination of 2D data, 3D data, and general information about the shapes of humans in general, which can be used to build a cued rough (and yet quite accurate) mental model of structures that aren't stored in their entirety in 3D visuals as one would naïvely think. The links are also quite excellent. Feel free to add more!
TheEnvironmentalist
CC BY-SA 3.0
2016-06-22T21:14:37.913Z
Thank you! I appreciate the feedback and I'm glad to be able to contribute to your thoughts about this. I may be writing a blog post of this sort in the near future. I'll link you if and when I do.
Larry Muhlstein
CC BY-SA 4.0
2020-10-21T20:55:08.183Z
It seems I'm still waiting on the science to catch up :) Did you ever write that blog post?
TheEnvironmentalist
CC BY-SA 3.0
2016-06-20T20:13:33.900Z
2016-06-20T20:20:41.523Z
2016-06-20T20:20:41.523Z
Larry Muhlstein
<p>Adobe Illustrator has taken over five minutes (and counting) to render a vector 2D image rotated 18° in 3D on my computer. And yet, I and nearly anyone else can easily visualize the subject rotated almost instantaneously, and with little effort rotate the object continuously in real time in the mind's eye.</p>

<p><a href="https://i.sstatic.net/NI8Rc.gif" rel="noreferrer"><img src="https://i.sstatic.net/NI8Rc.gif" alt="Spinning Flamingo"></a></p>

<p>I'm not asking how the brain <em>stores</em> representations of objects, as that's clearly up for debate. But how does the brain <em>structure</em> its internal representation of 3D visual data?</p>

<p>It's almost definitely not in some pixel-based format, as can be shown by simply visualizing an object of some sort, and then zooming in mentally on some detail and noticing the image retains its sharpness. It's probably not breaking down objects into geometric shapes either, because at least I personally don't visualize my friends as stick figures. It could be a <a href="http://searchwindevelopment.techtarget.com/definition/vector-graphics" rel="noreferrer">vector format</a>, but then it should be easier to visualize complex shapes that are mathematically simple, like this one:</p>

<p><a href="https://i.sstatic.net/OTbUE.png" rel="noreferrer"><img src="https://i.sstatic.net/OTbUE.png" alt="Complex but mathematically simple shape"></a></p>

<p>So it would seem that the brain uses some other format. To the best knowledge of modern cognitive, how does this work?</p>

CC BY-SA 3.0
2016-06-13T21:07:24.280Z
Our visual image is in 2D, similar to a normal camera, but there are many factors that let us interpret the world as 3D. These are, among others, size of the object(objects farther are usually smaller), ordering of object (some object being behind something else) and relative movement (objects closer by move faster in the 2D pane than objects farther away). Given our many specialized visual areas (V1-V6 etc), this in most cases happens unconsciously. P.s. The fact that you dont use stick figures consciously, does not mean your brain doesn't.
Robin Kramer-ten Have
CC BY-SA 3.0
2016-06-13T21:20:55.017Z
@RobinKramer I'm not really asking about vision, the question is about how the brain holds visual data. Picture your best friend. You can probably spin him or her around, raise or lower arms, legs and head, and otherwise construct movies in your head with this person. That kind of grasp of 3D objects is difficult for computers to achieve, and yet we (and likely at least a few mammals) can do it naturally, effortlessly, and effectively instantly. Without any need to go into the electrochemical process by which objects are *stored* (unless you really want to), in what "format" is this data?
TheEnvironmentalist
CC BY-SA 3.0
2016-06-14T13:15:26.100Z
Just because you think you are mentally simulating a 3d object doesn't mean that the object is actually 3d in your head. That would require a large amount of perceptual information. Typically speaking the majority of information stored is based on global precepts and local precepts allowing for more specific identification. ( global perceptive bias) So given that the visual information you're manipulating is not structurally accessible (as in drawing) you are manipulating the precepts that you have previously stored (as required to identify the object)
Dog
CC BY-SA 3.0
2016-06-14T18:52:18.177Z
@Dog the notion that visual data is mentally 2D but interpreted using rules accustomed to 3D would seem contrary to simple experience. It's very easy to picture an object and describe it's shape in three dimensions, but describing it's shape in two dimensions from any particular angle is difficult (think of a car, easy to visualize in three dimensions, hard to outline in two). I would have to say I disagree
TheEnvironmentalist
CC BY-SA 3.0
2016-06-14T23:48:59.760Z
The *ultimate* answer for this would be that it stores is as a (highly-complex) weighted network structure. I'm sorry if this answer is disappointing but that really *the* answer. I can explain more about how the brain processes visuals, from the retina, through feature detection and top-down feedback, but I'm not sure this will answer your question.
Izhaki
CC BY-SA 3.0
2016-06-15T10:56:30.800Z
Nice question! I think the speculation and mystery of how we do this is a good example of why human vision/imagery is such a hard problem for cognitive science. A couple of points: (1) Introspection is not great evidence for how this happens in the brain. It doesn't feel pixel/shape based to you, but these mechanisms are probably not open to conscious introspection. (2) The type of "format"/representation is a key question in models of object recognition, so you might like to look at work by Tarr and Biederman and the debate between "view dependant" and "view independent" models.
splint
CC BY-SA 3.0
2016-06-15T11:00:44.650Z
Actually, you should ask yourself whether there is any empirical evidence that people *can* rotate things smoothly, like you suggest. There are some classic papers on "mental rotation" that you might like to look at for this. I doubt we are as good as you think (I have no idea what the back of my friend's head looks like).
splint
<cognitive-neuroscience><vision><cognitive-modeling><visualization>
How does the brain structure 3D visual data?
TheEnvironmentalist
# How does the brain structure 3D visual data?

(Asked by: username_0 on 2016-06-13)

Adobe Illustrator has taken over five minutes (and counting) to render a vector 2D image rotated 18° in 3D on my computer. And yet, I and nearly anyone else can easily visualize the subject rotated almost instantaneously, and with little effort rotate the object continuously in real time in the mind's eye.

[![Spinning Flamingo](https://i.sstatic.net/NI8Rc.gif)](https://i.sstatic.net/NI8Rc.gif)

I'm not asking how the brain *stores* representations of objects, as that's clearly up for debate. But how does the brain *structure* its internal representation of 3D visual data?

It's almost definitely not in some pixel-based format, as can be shown by simply visualizing an object of some sort, and then zooming in mentally on some detail and noticing the image retains its sharpness. It's probably not breaking down objects into geometric shapes either, because at least I personally don't visualize my friends as stick figures. It could be a [vector format](http://searchwindevelopment.techtarget.com/definition/vector-graphics), but then it should be easier to visualize complex shapes that are mathematically simple, like this one:

[![Complex but mathematically simple shape](https://i.sstatic.net/OTbUE.png)](https://i.sstatic.net/OTbUE.png)

So it would seem that the brain uses some other format. To the best knowledge of modern cognitive, how does this work?

**Tags:** cognitive-neuroscience, vision, cognitive-modeling, visualization

> **Comments:**
> **username_1**: Our visual image is in 2D, similar to a normal camera, but there are many factors that let us interpret the world as 3D. These are, among others, size of the object(objects farther are usually smaller), ordering of object (some object being behind something else) and relative movement (objects closer by move faster in the 2D pane than objects farther away). Given our many specialized visual areas (V1-V6 etc), this in most cases happens unconsciously. P.s. The fact that you dont use stick figures consciously, does not mean your brain doesn't.  
> **username_0**: @RobinKramer I'm not really asking about vision, the question is about how the brain holds visual data. Picture your best friend. You can probably spin him or her around, raise or lower arms, legs and head, and otherwise construct movies in your head with this person. That kind of grasp of 3D objects is difficult for computers to achieve, and yet we (and likely at least a few mammals) can do it naturally, effortlessly, and effectively instantly. Without any need to go into the electrochemical process by which objects are *stored* (unless you really want to), in what "format" is this data?  
> **username_2**: Just because you think you are mentally simulating a 3d object doesn't mean that the object is actually 3d in your head. That would require a large amount of perceptual information. Typically speaking the majority of information stored is based on global precepts and local precepts allowing for more specific identification. ( global perceptive bias) So given that the visual information you're manipulating is not structurally accessible (as in drawing) you are manipulating the precepts that you have previously stored (as required to identify the object)  
> **username_0**: @username_2 the notion that visual data is mentally 2D but interpreted using rules accustomed to 3D would seem contrary to simple experience. It's very easy to picture an object and describe it's shape in three dimensions, but describing it's shape in two dimensions from any particular angle is difficult (think of a car, easy to visualize in three dimensions, hard to outline in two). I would have to say I disagree  
> **username_3**: The *ultimate* answer for this would be that it stores is as a (highly-complex) weighted network structure. I'm sorry if this answer is disappointing but that really *the* answer. I can explain more about how the brain processes visuals, from the retina, through feature detection and top-down feedback, but I'm not sure this will answer your question.  
> **username_4**: Nice question! I think the speculation and mystery of how we do this is a good example of why human vision/imagery is such a hard problem for cognitive science. A couple of points: (1) Introspection is not great evidence for how this happens in the brain. It doesn't feel pixel/shape based to you, but these mechanisms are probably not open to conscious introspection. (2) The type of "format"/representation is a key question in models of object recognition, so you might like to look at work by Tarr and Biederman and the debate between "view dependant" and "view independent" models.  
> **username_4**: Actually, you should ask yourself whether there is any empirical evidence that people *can* rotate things smoothly, like you suggest. There are some classic papers on "mental rotation" that you might like to look at for this. I doubt we are as good as you think (I have no idea what the back of my friend's head looks like).  

---

## 1 answers

### Answer by username_5 on 2016-06-20. Score: 6

This question cannot be answered in the form in which you asked it both because of the limits of current neuroscientific theories and methodologies when it comes to determining the structures of complex neural representations (although we have made headway in a few cases such as place cells and grid cells), and because neural representations are not really analogous to our colloquial concepts of simple mappings from numbers to images (ex. pixel, vector, wavelet, etc. bases for digital image representation).

I would encourage you to think, not only in terms of these representational formats privileged by Von Neumann style computer architectures (i.e. normal computers) with separate processing and digital memory systems, but in terms of a neural computer with distributed computation and memory. Neuroscientists typically think of neural representation in visual perception as occurring at a number of stages of increased abstraction. [At the retina](https://en.wikipedia.org/wiki/Retinal_ganglion_cell), the image is represented roughly in terms of what you might think of as pixels, but it is really just a set of photosensors and attached neurons that are activated due to light hitting the retina in different places at different frequencies and amplitudes. In [V1 or primary visual cortex](https://en.wikipedia.org/wiki/Visual_cortex#Primary_visual_cortex_.28V1.29), the representation is in terms of what can be thought of as "edge detectors." In V2, the cells are tuned for a variety of slightly more complex properties such as such as orientation, spatial frequency, color, and [binocular disparity](https://en.wikipedia.org/wiki/Binocular_disparity) (an important piece of information for 3D perception.) As you go up through V3, V4, V5, and V6, the representations becomes more and more complex until they contain information about the structure of the underlying concept itself.

The 3-dimensionality does not come in until the visual information from both eyes is combined with top-down information about how things tend to be in the world (which comes from other senses and experiences as well) in order to infer the likely shape of the object. It is unclear whether there is any place in the brain where an image is represented explicitly as a 3D model of a visual object. More likely the raw perceptual data such as colors and patterns is represented in one area and is associated with the more object-centric structural representation in another area and this association causes the co-activation of all the neurons relevant to the perception of the object in it's three dimensional form. The information is neither represented in one area of the brain nor in one scheme but is distributed across a large number of neurons with different degrees of activation.

We can also think of this in terms of the contemporary theory of enactive perception, which states that the brain represents information in terms of state-action-observation contingencies. In this theory, there is no "image" present at all, but only a set of neural activations which correlate with prior experience (where the correlations are stored in terms of associative links between neurons creating a causal activation path along which "information" travels) and relationships between potential actions (either low level motor actions of the retina or other muscles, or higher level actions such as taking a step forward) and their likely consequences (both low level perceptual consequences such as the expected changes in the sensory data when an action such as moving the head is taken and higher level perceptual and conceptual consequences such as a cup falling when you choose to open your hand and the associated experiences.) In this more comprehensive picture, visual information cannot be strictly separated from from conceptual and multi-sensory information and so three-dimensional visual information is represented both in terms of neural activations causally associated with the structure of the perceived object as well as the activations associated with all the other conceptual beliefs and perceptual expectations that relate.

If you want to read more, these sources should be a good start: [distributed neural memory](http://s3.amazonaws.com/academia.edu.documents/39939727/Transient_and_sustained_activity_in_a_di20151112-5796-vgowyp.pdf?AWSAccessKeyId=AKIAJ56TQJRTWSMTNPEA&Expires=1466455672&Signature=duYuMAjKs9%2B8UnJE6t8CgU1ZdXU%3D&response-content-disposition=inline%3B%20filename%3DTransient_and_sustained_activity_in_a_di.pdf "Neuroscience paper"), [distributed neural representation of a higher order process](http://www.jneurosci.org/content/25/19/4806.full "Distributed neural representation of a higher order process"), [perception as associations between actions and observations](https://mitpress.mit.edu/books/action-perception "Action in perception")

There are also a few sources that offer direct experimental evidence towards your question, but do not provide as much of a conceptual introduction to neural computation and representation. See ["Neural computations underlying depth perception"](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2883007/), [Binocular depth perception and the cerebral cortex](http://www.nature.com/nrn/journal/v8/n5/full/nrn2131.html) as a start.

Your question also makes reference to rotating an object "in the mind's eye." There is also a very large body of research on this phenomenon, which is called "mental rotation." [The wikipedia article](https://en.wikipedia.org/wiki/Mental_rotation) is a good place to start but there is a lot more to say about this interesting phenomenon, so feel free to ask a related question if you are curious!

> **Comments:**
> **username_0**: This is a fantastic answer, and I especially like the enactive perception interpretation, which would imply, for example, that the visual data attached to a human being is a combination of 2D data, 3D data, and general information about the shapes of humans in general, which can be used to build a cued rough (and yet quite accurate) mental model of structures that aren't stored in their entirety in 3D visuals as one would naïvely think. The links are also quite excellent. Feel free to add more!  
> **username_5**: Thank you! I appreciate the feedback and I'm glad to be able to contribute to your thoughts about this. I may be writing a blog post of this sort in the near future. I'll link you if and when I do.  
> **username_0**: It seems I'm still waiting on the science to catch up :) Did you ever write that blog post?  

---

