Generating virtual camera views using generative networks

This project will explore how advances in generative networks and deep-learning approaches to inpainting can be combined to create virtual camera views.  A unified generative deep learning framework for viewpoint interpolation and inpainting, should effectively blend the observed viewpoints and hallucinated information, to produce a virtual camera feed from otherwise impractical vantage points. The developed technology is of interest as a potential extension to the capabilities of the Ed system currently being developed by BBC R&D.

The Ed system is an automated system for creating edited coverage of live events. Its inputs are high-resolution locked-off cameras that are deployed at venues with live audiences. The system then creates “virtual camera” views by cropping these raw camera feeds and cuts between them to produce output. In real-world deployments, camera positioning is commonly constrained to suboptimal positions. The ability to synthesise more favourable camera views that were physically untenable could improve the output of the Ed system.

Program of research

The first 15 months will be based at the University of Surrey, and will focus on integrating deep generative viewpoint interpolation with Surrey’s existing research on deep inpainting. The initial research will focus primarily on static environments. This period will also include extensive personal development and training opportunities covering technical and professional skills.

The following 18 months will be undertaken at the BBC North site in Manchester, and will explore problem specific extensions to the developed framework, such as enforcing temporal consistency and the insertion of artificial motion blurring for dynamic objects. The majority of the data collection will also happen during this period, with radio theatre as a likely use-case.

The remainder of the PhD is expected to be based primarily at BBC North, but some flexibility may be possible, subject to the requirements of the project and the student. The research during this period will focus on preliminary feasibility studies for potential follow-on research, such as initial explorations of developing (moving) shots from virtual cameras, and the interaction of lighting with the virtual camera.

Key highlights of this studentship

This studentship should appeal particularly to scientists and engineers who are interested in working with the creative industries, or have a particular interest in cutting edge and next generation vision technologies.  The project will be very interdisciplinary, with exposure to both academic and industrial research centres. Furthermore, this studentship could potentially benefit from the University of Surrey’s Centre for Doctoral Training in Audio-Visual Machine Perception. This will constitute a rigorous fully-funded programme of personal development, including professional research and software skills, interdisciplinary hackathon events, and a co-located peer-support network of approximately 100 other PhD students in related areas. Student’s receiving their doctorate through this scheme are expected to be exceptionally well-qualified for a subsequent career in research.

The ideal candidate for this studentship should have a strong academic background in software development from studying Computer Science, Electronic Engineering or a related subject. Furthermore, the candidate should have a keen interest in AI/machine learning. Prior experience (including formal study, project work or MOOCs) is advantageous but not required. Candidates would be expected to have (by October 2019) either a 2:1 or First class honours undergraduate degree or a Masters in one of the areas highlighted above.

Due to the nature of funding the studentship is available to UK citizens. Overseas applications will not be considered.

The PhD scholarship is offered for the duration of three and a half years, starting on 1 October 2019, and will cover full university fees and a stipend. No external funds are required from the student.

Non-native speakers of English will normally be required to have IELTS 6.5 or above (or equivalent) with no sub-test of less than 6.

• Centre for Vision, Speech and Signal Processing
• AI in production - BBC R&D
• Disentangling structure and aesthetics for style-aware image completion.