Abstract:

Online streaming of Virtual Reality and 360-degree videos is rapidly growing, as more and more major content providers and news outlets adopt the format to enrich the user experience. We characterize 360-degree videos by examining several thousand YouTube videos across more than a dozen categories. 360-degree videos, at first sight, seem to pose a challenge for the network to stream because of their substantially higher bit rates and larger number of resolutions. However, a careful examination of video characteristics reveals that there are significant cant opportunities for reducing the actual bit rate delivered to client devices based on the user’s field of view. We study the bit rate and the motion in 360-degree videos, and compare them against regular videos by investigating several important metrics. We find that 360-degree videos are less variable in terms of bit rate, and have less motion than regular videos. Our expectation is that variability in the bit rates due to the motion of the camera in regular videos (or switching between cameras) is now translated to responsiveness requirements for end-to-end 360-degree streaming architectures.

Abstract:

We propose a VR video conferencing system over named data networks (NDN). The system is designed to support real-time, multi-party streaming and playback of 360 degree video on a web player. A centralized architecture is used, with a signaling server to coordinate multiple participants. To ensure real-time requirement, a protocol featuring prefetching is used for producer-consumer communication. Along with the native support of multicast in NDN, this design is expected to better support large amount of data streaming between multiple users. As a proof of concept, a protoype of the system is implemented with one-way real-time 360 video streaming. Experiments show that seamless streaming and interactive playback of 360 video can be achieved with low latency. Therefore, the proposed system has the potential to provide immersive VR experience for real-time multi-party video conferencing.

Abstract:

360 degree video compression and streaming is one of the key components of Virtual Reality (VR) applications. In 360 video streaming, a user may freely navigate through the captured 3D environment by changing her desired viewing direction. Only a small portion of the entire 360 degree video is watched at any time. Streaming the entire 360 degree raw video is therefore unnecessary and bandwidth-consuming. One the other hand, only streaming the video in the user’s current view direction will introduce streaming discontinuity whenever the user changes her view direction. In this work, a two-tier 360 video streaming framework with prioritized buffer control is proposed to effectively accommodate the dynamics in both network bandwidth and viewing direction. Through simulations driven by real network bandwidth and viewing direction traces, we demonstrate that the proposed framework can significantly outperform the conventional 360 video streaming solutions.

Abstract:

Online VR streaming faces great challenges such as the high throughput and real time interaction requirement. In this paper, we propose a novel ultra wide view based method to stream high quality VR on Internet at low bandwidth and little computation cost. First, we only transmit the region where user is looking at instead of full 360 degree view to save bandwidth. To achieve this goal, we split the source VR into small grid videos in advance. The grid videos are able to reconstruct any view flexibly in user end. Second, according to the fact that users generally interact at low speed, we expand the view that user requested to meet the real time interaction requirement. Besides, a low resolution full view stream is supplied to handle exceptional cases such as high speed view change. We test our solution in an experimental network. The results show remarkable bandwidth saving of over 60

Abstract:

In this paper, we conduct a reality check for Augmented Reality (AR) on mobile devices. We dissect and measure the cloud-offloading feature for computation-intensive visual tasks of two popular commercial AR systems. Our key finding is that their cloud-based recognition is still not mature and not optimized for latency, data usage and energy consumption. In order to identify the opportunities for further improving the Quality of Experience (QoE) for mobile AR, we break down the end-to-end latency of the pipeline for typical cloud-based mobile AR and pinpoint the dominating components in the critical path.

Abstract:

VR/AR is rapidly progressing towards enterprise and end customers with the promise of bringing immersive experience to numerous applications. Soon it will target smartphones from the cloud and 360° video delivery will need unprecedented requirements for ultra-low latency and ultra-high throughput to mobile networks. Latest developments in NFV and Mobile Edge Computing reveal already the potential to enable VR streaming in cellular networks and to pave the way towards 5G and next stages in VR technology. In this paper we present a Field Of View (FOV) rendering solution at the edge of a mobile network, designed to optimize the bandwidth and latency required by VR 360° video streaming. Preliminary test results show the immediate benefits in bandwidth saving this approach can provide and generate new directions for VR/AR network research.

Abstract:

We study the delivery of 360^∘-navigable videos to 5G VR/AR wireless clients in future cooperative multi-cellular systems. A collection of small-cell base stations interconnected via back-haul links are sharing their caching and computing resources to maximize the aggregate reward they earn by serving 360^∘ videos requested by VR/AR wireless clients. We design an efficient representation method to construct the 360^∘ videos such that they only deliver the remote scene viewpoint content genuinely needed by the VR/AR user, thereby overcoming the present highly inefficient approach of sending a bulky 360^∘ video, whose major part comprises scene information never accessed by the user. Moreover, we design an optimization framework that allows the base stations to select cooperative caching/rendering/streaming strategies that maximize the aggregate reward they earn when serving the users, for the given caching/computational resources at each base station. We formulate the problem of interest as integer programming, show its NP-hardness, and derive a fully-polynomial-time approximation solution with strong performance guarantees. Our advances demonstrate orders of magnitude operational efficiency gains over state-of-the-art caching and 360^∘ video representation mechanisms and are very promising. This is a first-of-its-kind study to explore fundamental trade-offs between caching, computing, and communication for emerging VR/AR applications of prospectively broad societal impact.

Abstract:

Deep learning could make Augmented Reality (AR) devices smarter, but few AR apps use such technology today because it is compute-intensive while front-end devices often cannot deliver sufficient compute power. We propose a distributed framework that ties together front-end devices with more powerful backend “helpers” that allow deep learning to be executed locally or to be offloaded. The framework shall be able to intelligently use current estimates of network conditions and backend server loads, and in conjunction with application’s requirements to determine an optimal offload strategy.

This work reports our preliminary investigation in implementing this framework, in which the front-end is assumed to be smartphones. Our specific contributions include (1) development of an Android application that performs real-time object detection, either locally on the smartphone or remotely on a server; and (2) characterization of the tradeoffs between object detection accuracy, latency, and battery drain, based on the system parameters of video resolution, CNN model size, and offloading decision.