<p>As other people have mentioned, due to network latency issues, I think that full rendering on a cloud server and streaming video to a client device is unlikely to be workable. Even if the latency can be kept low enough in a best-case scenario, it would only work for a very limited number of users with very high-quality internet connections. (Don't forget that e.g. in the USA, there are still tens of millions of people on dialup or DSL, with no access to cable internet, let alone fiber. It will be decades still before fiber is widespread.) Also, note that many <a href="https://en.wikipedia.org/wiki/Cloud_gaming">cloud gaming services</a> do exist, but that according to players, input latency is noticeably worse on these services than playing locally on a PC.</p>

<p>However, all that being said, rendering entirely on a cloud server is not the only way to approach the problem. The most latency-sensitive aspect of VR is head tracking. Oculus's <a href="https://developer.oculus.com/blog/asynchronous-timewarp-examined/">"timewarp"</a> technology shows that you can quickly and cheaply update a previously rendered image to respond to head rotation, as a 2D image operation: if the user turns their head to the right, you shift the image to the left, and so on. With a depth buffer, you can do positional timewarp as well, although this creates various artifacts with parallax because it's based on a single-layered 2D image.</p>

<p>Timewarp is certainly cheap enough to run on a mobile device. So, one approach is to render frames in the cloud, stream the video to the client, then timewarp the video on the client device to obtain very low-latency head tracking. This wouldn't help the latency of other inputs such as keyboard/mouse/controller inputs, but those don't induce nausea when they're delayed, the way delayed head-tracking does.</p>

<p>There are also other possibilities along the lines of splitting rendering between the cloud and the client. For example, object-space lighting and shading could be done on the cloud and streamed to the client as textures; then the client only needs to render simple textured geometry. This would work better than positional timewarp for getting correct parallax when you move your head around, though you'd still have some artifacts with specular shading and reflections. There are a lot of possibilities for offloading rendering to the cloud in VR-friendly ways.</p>

CC BY-SA 3.0
2016-04-07T06:08:09.030Z
Hi Nathan, nice to meet you here. I have read some of your presentations on Nvidia Gameworks VR. "Timewarp" can reduce the latency but with image quality loss. And I think, if the head rotated very fast,  it's very hard to warp the 2D image to cheat the eyes. Motion blurring may be help in this case. Local client or remote server could send an unclear image to the display in very low latency. You mentioned Cloud could do some offloadings on VR. That's a cool. What's the requirement for the network bandwidth and latency? Does it still need high bandwidth and low latency? Thanks.
Hao Zhang
CC BY-SA 3.0
2016-04-07T06:48:39.943Z
You are right, current internet access are mostly based on dialup or DSL. But, in the near future, e.g. 5-10 years later, could something be changed? Google has deployed 1Gbps fiber in some cities and I heard Google also tried to introduce 10Gbps to home.
Hao Zhang
CC BY-SA 3.0
2016-04-07T07:23:32.787Z
haha, Nathan, I got some information from your blog: (http://www.reedbeta.com/blog/2014/04/03/vr-and-multi-gpu/#more-591), "That being said, what if the high-spec dGPUs were far away from you? Cloud-based rendering is a topic that’s been getting interest lately, and the same kind of latency reduction strategies could be applicable there. Imagine having your VR headset driven by the little GPU in your phone, but streaming down source frames or shading cache updates from a dGPU in the cloud (or your home PC)! "
Hao Zhang
CC BY-SA 3.0
2016-04-06T17:07:41.003Z
2016-04-06T17:07:41.003Z
Nathan Reed
<p>Strictly speaking I think that the latency coming from transferring data over the network will not be low enough for VR. The device will need to send all the head tracking data and the user input over the network to the cloud based rendering farm. Then that data and input would need to be processed, rendered and the video streamed back to the user. Can you imagine playing a game and the the actual game stopping for buffering?</p>

CC BY-SA 3.0
2016-04-06T01:00:01.760Z
I added more detail info above. That explains what I'm thinking
Hao Zhang
CC BY-SA 3.0
2016-04-05T11:06:05.663Z
2016-04-05T11:06:05.663Z
Uri Popov
<p>I always feel gaming, particularly AAA gaming (e.g., FPS), is and will be the last application for cloud rendering, regardless being VR or not. VR simply makes the problem/requirement harder. </p>

<p>Because of the tight loop of interactivity, latency is the the blocking factor as others have pointed out -- and we know that unlike bandwidth, latency is a much harder problem to solve practically in a commercial world. Though technically solvable (say deploying a group of game servers in every county, which is well within the Geo-boundary range of your back-of-envolop calculation above; also your matchmaking process needs to do the right player grouping), it really needs justifiable business case.</p>

<p>Having said that, some non-AAA titles could well fall into cloud side rendering.</p>

CC BY-SA 3.0
2016-08-02T20:33:34.053Z
2016-08-04T15:40:27.483Z
2016-08-04T15:40:27.483Z
chen
<p>Network latency will not be low enough VR in near future. Mobile VR is a key driver for Cloud Rendering. However, the mobile network latency is much higher. Split rendering with future micro-cloud based Gamelets architecture (<a href="http://ieeexplore.ieee.org/document/6799051/" rel="nofollow noreferrer">http://ieeexplore.ieee.org/document/6799051/</a>) is one possible solution for Cloud Rendering for VR. </p>

CC BY-SA 3.0
2017-01-10T01:49:10.983Z
2017-01-10T01:49:10.983Z
Baddu
<h2>I do this already.</h2>

<p><strong>TL;DR: The latency between services decelerate the enthusiasm.</strong></p>

<p>I have a steam-vr-client supporting a wide range of Helmets and the vr-client is connecting to a local server (Currently connectors for Java exsits only) and then further cloud-services. </p>

<p>As most games have two threads who must be synchronized</p>

<ol>
<li>Frontend: The visualization of the gamelogic</li>
<li>Backend: The gamelogic, model, rules....</li>
<li>(Optional) Multiplayer Thread: Synchronizing gamestate between contributors.</li>
</ol>

<p>In VR there will be a growing latency between Frontend and Backend the more work is done in the Clouded-Backend. So we have to estimate what contribution to the VR is a lightweight-transported information and do the rest in the cloud.</p>

<ol>
<li>Frontend (locally): The low-level visualisation. Calculating shadows and "depth of field blur", but not calculating ambient occlusion, not calculating raytracing, not calculating Bumpmapping.</li>
<li>Backend (locally): Collecting results from the cloud, collect Bumpmaps, transport Mesh, UV-Coodinates, Normal-Mappings, Bumpmappings, Raytrace-Results and send it to Frontend.</li>
<li>Cloud (web/internet). Calculating raytrace, generate Bumpmaps, Calculate Mesh resolution, calculate UV-Coordinates and Normals.</li>
</ol>

<p>The key to seperate work for different cloud services is to understand what work can be done packed autark.</p>

<p>My project: <a href="https://github.com/enexusde/VRServer" rel="nofollow noreferrer">https://github.com/enexusde/VRServer</a></p>

CC BY-SA 4.0
2020-06-01T13:56:12.403Z
2020-06-01T13:56:12.403Z
Grim
<p>According to the following assumptions, I think Cloud based VR would be the future:</p>

<p>1) Real world scene rendering needs ray tracing or other complex computing graphics technologies. It needs a lot of computing power.</p>

<p>2) Single GPU can't afford enough GFLOPS for that. And the powerful GPU is costly and hot. Since the moore's law is ending soon, the process of GPU will also end evolution, that means GPU can't get much faster than needed.</p>

<p>3) Mobility is the king. But mobile GPU speed will be limited by the power and process. It won't get much faster too.</p>

<p>4) Cloud gaming would be promise for that. The network bandwidth and latency would be the biggest issues. Since Google Fiber has been deployed, I don't think network is the issue in the future.</p>

<p>5) Rendering a real world needs unlimited computing power, but human eye resolution is limited. Limited resolution means limited network bandwidth consumption. For example, 1Gbps is enough to transmite 8K/16K H265 streaming video in stereo format.</p>

<p>So, I believe Cloud based VR (Rendering in Cloud and stream to the client) would be the future. Am I correct?</p>

<hr>

<p>added more detail @20160406</p>

<p>I did some research on it. The motion-to-photon has to be less than 20ms. Maybe only 5ms is left for round trip network. So that the Cloud DC has to be very close to the client. In case optical fiber access, only two hops ( switch and BRAS ) to local router and then to local Cloud DC. (Transmission speed in optical is 200K KM/s, so 100KM radius would take only 0.5ms. That means Cloud DC would cover 30K square KM area.)If no network congestion with good QoS policy, the network latency would be very low. At the mean time, use distributed GPU clusters to render the VR content and then stream it to the client in parallel to reduce the time further. Let's count the time consumptions during each stages. 1.5ms for head tracking and local cpu processing. 1.5ms for network transmission to Cloud. 1.5 ms for distributed GPU cluster rendering and encoding. 5ms for network transmission to client. 1ms for decoding with special decoder since current video decoder designed for video playing not for VR. 5ms for transmission to monitor (equal 200Hz monitor, or very high bandwidth transmission as display port 1.4). The total time is 1.5+1.5+1.5+5+1+5 = 15.5ms. Of course, if using Foveated Rendering technology, addition 3ms for eye tracking delay, but the network bandwidth and latency will be reduced.</p>

CC BY-SA 3.0
2016-04-05T10:18:00.380Z
For VR you need super low latency, any type of network hop will be too long.
ratchet freak
CC BY-SA 3.0
2016-04-05T12:59:35.223Z
Only if the cloud is very close to the vr gear like, within tens of meters. But given that everything and their dad is going to be a computer that is certainly possible.
joojaa
CC BY-SA 3.0
2016-04-06T00:59:47.590Z
@ratchetfreak,  I added more detail info above. That explains what I'm thinking
Hao Zhang
CC BY-SA 3.0
2016-04-06T07:18:05.233Z
@HaoZhang well if your calculations that the total time to display a frame to the user from the cloud is 15.5ms then things should work out. In order to have a game running at 60FPS you need to display a frame on the screen every 16.6ms. Are you sure its going to take 15.5ms to display both frames(one for each screen of the VR system)?
Uri Popov
CC BY-SA 3.0
2016-04-06T08:45:59.250Z
I give it 5ms for transmission to OLED display of 2K*2@200Hz, that would need at least 40Gbps interface and cable, as display port 1.4. (https://en.wikipedia.org/wiki/DisplayPort) supports. Yes it would be a little costly but it is achievable.
Hao Zhang
CC BY-SA 3.0
2016-04-07T16:21:10.270Z
Although this question as phrased appears to be broad and opinion based, the answers show that the possibilities are not as arbitrary as they may at first appear, and objective restrictions can be placed on what is likely to happen.
trichoplax is on Codidact now
CC BY-SA 3.0
2016-08-03T04:43:36.190Z
I'm not sure Moore's law applies to GPU, you can keep adding cores and using wider buses.
PaulHK
CC BY-SA 3.0
2016-08-04T03:14:42.310Z
@PaulHK, the main issue is power and thermal. It's very hard to make a very big chip, especially for mobile system.
Hao Zhang
<gpu><virtual-reality>
Cloud based VR would be the future?
Hao Zhang
# Cloud based VR would be the future?

(Asked by: username_0 on 2016-04-05)

According to the following assumptions, I think Cloud based VR would be the future:

1) Real world scene rendering needs ray tracing or other complex computing graphics technologies. It needs a lot of computing power.

2) Single GPU can't afford enough GFLOPS for that. And the powerful GPU is costly and hot. Since the moore's law is ending soon, the process of GPU will also end evolution, that means GPU can't get much faster than needed.

3) Mobility is the king. But mobile GPU speed will be limited by the power and process. It won't get much faster too.

4) Cloud gaming would be promise for that. The network bandwidth and latency would be the biggest issues. Since Google Fiber has been deployed, I don't think network is the issue in the future.

5) Rendering a real world needs unlimited computing power, but human eye resolution is limited. Limited resolution means limited network bandwidth consumption. For example, 1Gbps is enough to transmite 8K/16K H265 streaming video in stereo format.

So, I believe Cloud based VR (Rendering in Cloud and stream to the client) would be the future. Am I correct?

---

added more detail @20160406

I did some research on it. The motion-to-photon has to be less than 20ms. Maybe only 5ms is left for round trip network. So that the Cloud DC has to be very close to the client. In case optical fiber access, only two hops ( switch and BRAS ) to local router and then to local Cloud DC. (Transmission speed in optical is 200K KM/s, so 100KM radius would take only 0.5ms. That means Cloud DC would cover 30K square KM area.)If no network congestion with good QoS policy, the network latency would be very low. At the mean time, use distributed GPU clusters to render the VR content and then stream it to the client in parallel to reduce the time further. Let's count the time consumptions during each stages. 1.5ms for head tracking and local cpu processing. 1.5ms for network transmission to Cloud. 1.5 ms for distributed GPU cluster rendering and encoding. 5ms for network transmission to client. 1ms for decoding with special decoder since current video decoder designed for video playing not for VR. 5ms for transmission to monitor (equal 200Hz monitor, or very high bandwidth transmission as display port 1.4). The total time is 1.5+1.5+1.5+5+1+5 = 15.5ms. Of course, if using Foveated Rendering technology, addition 3ms for eye tracking delay, but the network bandwidth and latency will be reduced.

**Tags:** gpu, virtual-reality

> **Comments:**
> **username_1**: For VR you need super low latency, any type of network hop will be too long.  
> **username_2**: Only if the cloud is very close to the vr gear like, within tens of meters. But given that everything and their dad is going to be a computer that is certainly possible.  
> **username_0**: @username_1,  I added more detail info above. That explains what I'm thinking  
> **username_3**: @username_0 well if your calculations that the total time to display a frame to the user from the cloud is 15.5ms then things should work out. In order to have a game running at 60FPS you need to display a frame on the screen every 16.6ms. Are you sure its going to take 15.5ms to display both frames(one for each screen of the VR system)?  
> **username_0**: I give it 5ms for transmission to OLED display of 2K*2@200Hz, that would need at least 40Gbps interface and cable, as display port 1.4. (https://en.wikipedia.org/wiki/DisplayPort) supports. Yes it would be a little costly but it is achievable.  
> **username_4**: Although this question as phrased appears to be broad and opinion based, the answers show that the possibilities are not as arbitrary as they may at first appear, and objective restrictions can be placed on what is likely to happen.  
> **username_5**: I'm not sure Moore's law applies to GPU, you can keep adding cores and using wider buses.  
> **username_0**: @username_5, the main issue is power and thermal. It's very hard to make a very big chip, especially for mobile system.  

---

## 5 answers

### Answer by username_6 on 2016-04-06. Score: 9

As other people have mentioned, due to network latency issues, I think that full rendering on a cloud server and streaming video to a client device is unlikely to be workable. Even if the latency can be kept low enough in a best-case scenario, it would only work for a very limited number of users with very high-quality internet connections. (Don't forget that e.g. in the USA, there are still tens of millions of people on dialup or DSL, with no access to cable internet, let alone fiber. It will be decades still before fiber is widespread.) Also, note that many [cloud gaming services](https://en.wikipedia.org/wiki/Cloud_gaming) do exist, but that according to players, input latency is noticeably worse on these services than playing locally on a PC.

However, all that being said, rendering entirely on a cloud server is not the only way to approach the problem. The most latency-sensitive aspect of VR is head tracking. Oculus's ["timewarp"](https://developer.oculus.com/blog/asynchronous-timewarp-examined/) technology shows that you can quickly and cheaply update a previously rendered image to respond to head rotation, as a 2D image operation: if the user turns their head to the right, you shift the image to the left, and so on. With a depth buffer, you can do positional timewarp as well, although this creates various artifacts with parallax because it's based on a single-layered 2D image.

Timewarp is certainly cheap enough to run on a mobile device. So, one approach is to render frames in the cloud, stream the video to the client, then timewarp the video on the client device to obtain very low-latency head tracking. This wouldn't help the latency of other inputs such as keyboard/mouse/controller inputs, but those don't induce nausea when they're delayed, the way delayed head-tracking does.

There are also other possibilities along the lines of splitting rendering between the cloud and the client. For example, object-space lighting and shading could be done on the cloud and streamed to the client as textures; then the client only needs to render simple textured geometry. This would work better than positional timewarp for getting correct parallax when you move your head around, though you'd still have some artifacts with specular shading and reflections. There are a lot of possibilities for offloading rendering to the cloud in VR-friendly ways.

> **Comments:**
> **username_0**: Hi username_6, nice to meet you here. I have read some of your presentations on Nvidia Gameworks VR. "Timewarp" can reduce the latency but with image quality loss. And I think, if the head rotated very fast,  it's very hard to warp the 2D image to cheat the eyes. Motion blurring may be help in this case. Local client or remote server could send an unclear image to the display in very low latency. You mentioned Cloud could do some offloadings on VR. That's a cool. What's the requirement for the network bandwidth and latency? Does it still need high bandwidth and low latency? Thanks.  
> **username_0**: You are right, current internet access are mostly based on dialup or DSL. But, in the near future, e.g. 5-10 years later, could something be changed? Google has deployed 1Gbps fiber in some cities and I heard Google also tried to introduce 10Gbps to home.  
> **username_0**: haha, username_6, I got some information from your blog: (http://www.reedbeta.com/blog/2014/04/03/vr-and-multi-gpu/#more-591), "That being said, what if the high-spec dGPUs were far away from you? Cloud-based rendering is a topic that’s been getting interest lately, and the same kind of latency reduction strategies could be applicable there. Imagine having your VR headset driven by the little GPU in your phone, but streaming down source frames or shading cache updates from a dGPU in the cloud (or your home PC)! "  

---

### Answer by username_3 on 2016-04-05. Score: 3

Strictly speaking I think that the latency coming from transferring data over the network will not be low enough for VR. The device will need to send all the head tracking data and the user input over the network to the cloud based rendering farm. Then that data and input would need to be processed, rendered and the video streamed back to the user. Can you imagine playing a game and the the actual game stopping for buffering?

> **Comments:**
> **username_0**: I added more detail info above. That explains what I'm thinking  

---

### Answer by username_7 on 2016-08-02. Score: 2

I always feel gaming, particularly AAA gaming (e.g., FPS), is and will be the last application for cloud rendering, regardless being VR or not. VR simply makes the problem/requirement harder.

Because of the tight loop of interactivity, latency is the the blocking factor as others have pointed out -- and we know that unlike bandwidth, latency is a much harder problem to solve practically in a commercial world. Though technically solvable (say deploying a group of game servers in every county, which is well within the Geo-boundary range of your back-of-envolop calculation above; also your matchmaking process needs to do the right player grouping), it really needs justifiable business case.

Having said that, some non-AAA titles could well fall into cloud side rendering.


---

### Answer by username_8 on 2017-01-10. Score: 0

Network latency will not be low enough VR in near future. Mobile VR is a key driver for Cloud Rendering. However, the mobile network latency is much higher. Split rendering with future micro-cloud based Gamelets architecture (<http://ieeexplore.ieee.org/document/6799051/>) is one possible solution for Cloud Rendering for VR.


---

### Answer by username_9 on 2020-06-01. Score: 0

## I do this already.

**TL;DR: The latency between services decelerate the enthusiasm.**

I have a steam-vr-client supporting a wide range of Helmets and the vr-client is connecting to a local server (Currently connectors for Java exsits only) and then further cloud-services.

As most games have two threads who must be synchronized

1. Frontend: The visualization of the gamelogic
2. Backend: The gamelogic, model, rules....
3. (Optional) Multiplayer Thread: Synchronizing gamestate between contributors.

In VR there will be a growing latency between Frontend and Backend the more work is done in the Clouded-Backend. So we have to estimate what contribution to the VR is a lightweight-transported information and do the rest in the cloud.

1. Frontend (locally): The low-level visualisation. Calculating shadows and "depth of field blur", but not calculating ambient occlusion, not calculating raytracing, not calculating Bumpmapping.
2. Backend (locally): Collecting results from the cloud, collect Bumpmaps, transport Mesh, UV-Coodinates, Normal-Mappings, Bumpmappings, Raytrace-Results and send it to Frontend.
3. Cloud (web/internet). Calculating raytrace, generate Bumpmaps, Calculate Mesh resolution, calculate UV-Coordinates and Normals.

The key to seperate work for different cloud services is to understand what work can be done packed autark.

My project: <https://github.com/enexusde/VRServer>


---

