Anatomie trifft Kino
Modern imaging techniques look deep into the human body. Yet the images they provide are often difficult for anyone without a medical background to interpret. Is there a way to enhance medical images to such an extent that they appear natural and are easily understood?
Dr. rer. nat. Dipl.-Inf. (Univ.) Klaus Dieter Engel, Prof. Dr. med. Franz A. Fellner and Dr.-Ing. Robert Schneider have developed a technology that does just that. It is based on an approach popularized in the production of realistic animated films. The technique employs near-natural artificial lighting that imitates the complex propagation and changes in light. The three nominated scientists have refined the technology for medical applications. Clinical images from inside the body can be depicted in a fully new "hyperrealistic" form. Klaus Dieter Engel and Robert Schneider are leading experts in visualization at Siemens Healthineers in Erlangen, Germany, Franz A. Fellner is head of the Central Radiological Institute at Kepler University Hospital in Linz, Austria.
Imaging techniques like computer tomography and magnetic resonance imaging in which "pictures" of the body are produced using X-rays or magnetic fields are able to provide finely detailed images of organs or tissue structures at a resolution of less than one millimeter. The images that are produced slice by slice reveal internal injuries or the characteristics of a disease to physicians. Yet for laypeople, who are not as familiar with human anatomy, the images are often incomprehensible. This makes communication between radiologists and referring physicians but above all between doctors and patients difficult.
To overcome this obstacle, Klaus Dieter Engel and Robert Schneider, both Siemens scientists, were inspired by a method already in use in animated film production: during the rendering process, realistic lighting effects make virtual figures appear to fit naturally into the scene in the film environment. The process is very complex since it takes into account physical effects such as the reflection, absorption and color change of numerous light beams which are influenced by surrounding objects. Rendering in the studio is generally a long and involved post-production process of the film. In medical applications, however, the processed images must be made available quickly.
The scientists at Siemens Healthineers succeeded by using a specially developed mathematical algorithm that simulates only those beams of light in calculating photorealistic lighting which actually reach the eye of the beholder. Thanks to numerous other optimizations of the algorithm, the computing time is cut so dramatically that the process can be used on most computers. As a result, "Cinematic Rendering" delivers three-dimensional and photorealistic imaging of the body, vividly revealing even the finest structures and an unprecedented degree of plasticity. By combining these images with data from the PET CT, for example, they can also visualize functional information such as metabolic activity.
The very graphic and natural visualization of clinical data makes it easier for doctors to explain to their patients the findings regarding disease or injury as well as the procedure of a scheduled operation. The technique can also be used in the education and further training of medical students, therapists and healthcare personnel because it provides what had until recently been unimaginable insight into anatomical details – and is even capable of visualizing fibers in the brain with an otherwise impossible fineness. It is also possible to turn realistic 3D images of a living human body in any direction and process them as needed for a specific case.
Siemens Healthineers already offer doctors and scientists software products that use Cinematic Rendering for the visualization of clinical images. Franz A. Fellner has been using the technology since 2015 at the Ars Electronica Center –- a high-tech museum for future technologies in Linz– for impressive presentations in which the audience delves virtually into the depths of the human body as well as to train and educate doctors. The technology is also already being used at the Linz University Hospital as a special tool to prepare for operations. Since summer 2017 Cinematic Rendering is also being tested at the University Clinic Erlangen as a tool in the planning of surgical procedures of the liver, gall bladder or pancreas.
Dr. rer. nat. Dipl.-Inf. (Univ.) Klaus Dieter Engel
- Born in Nördlingen, Germany
- 1976 – 1980
- Elementary school, Nördlingen/Löpsingen
- 1980 – 1989
- Theodor-Heuss Gymnasium, Nördlingen
- Abitur (High School diploma)
- 1989 – 1990
- Military service
- 1990 – 1993
- Undergraduate studies in computer science at the University of Würzburg
- Intermediate diploma in computer science at the University of Erlangen-Nürnberg
- 1993 – 1997
- Graduate studies in computer science at the University of Erlangen-Nürnberg
- Degree in computer science at the University of Erlangen-Nürnberg
- Computer scientist at 3SOFT GmbH in Erlangen
- 1998 – 1999
- Doctoral student at the University of Erlangen-Nürnberg
- 1999 – 2002
- Doctoral student at the University of Stuttgart
- PhD in Natural Sciences (Dr. rer. nat.) from the University of Stuttgart
- 2003 – 2006
- Researcher at Siemens Corporate Research Inc., Princeton, NJ, USA
- 2006 – 2007
- Researcher at Siemens Corporate Technology, Erlangen
- 2006 – 2014
- Researcher at Siemens Healthcare, Components and Vacuum Technology
- Appointment as Principal Key Expert for visualization
- Since 2014
- Principal Key Expert for visualization at Siemens Healthineers, Strategy and Innovation
- Best Paper Award, Eurographics/SIGGRAPH Workshop on Graphics Hardware
- Best Paper Award, Eurographics/SIGGRAPH Workshop on Graphics Hardware
- Inventor of the Year in the category “Single Outstanding Invention” at Siemens AG
Prof. Dr. med. Franz A. Fellner
- Born in Passau, Germany
- 1972 – 1976
- Schalding r. d. D. elementary school, Passau
- 1976 – 1985
- Gymnasium Leopoldinum, Passau
- 1985 – 1987
- Study of human medicine (pre-clinic), University of Regensburg
- 1987 – 1991
- Study of human medicine (clinic), Technical University of Munich
- 1989 – 1992
- Tutor for neuroanatomy, University of Regensburg
- 1992 – 1993
- Resident, radiology practice at Klinikum Ingolstadt
- License to practice medicine
- 1993 – 1994
- Resident for neurology, Landes-Nervenklinik in Linz, Austria; scientific collaboration with the LNK Institute for Radiology
- 1994 – 1995
- Resident at the Institute of Diagnostic Radiology, Klinikum Ingolstadt
- 1995 – 1996
- Resident at the Institute of Radiology at Landes-Nervenklinik in Linz, Austria
- Doctorate in Medicine, Technical University of Munich
Dr. med. degree (degree in medicine) / grade: magna cum laude
- 1996 – 2000
- Resident at the Institute of Diagnostic Radiology, University of Erlangen-Nürnberg
- Certification as a medical specialist in diagnostic radiology
- Postdoctoral qualification in diagnostic radiology
- Awarded the venia legendi (lecturing qualification), University of Erlangen-Nürnberg
- 2001 – 2002
- Senior physician of the Institute of Diagnostic Radiology and Deputy Director of the Institute, University of Erlangen-Nürnberg
- 2002 – 2005
- Senior physician at the Institute of Neuroradiology, Landes-Nervenklinik in Linz, Austria
- 2005 - 2015
- Director of the Central Radiology Institute, General Hospital of the city of Linz, Austria
- Since 2007
- Extraordinary professor at the University of Erlangen-Nürnberg
Member of the faculty council
- Since 2016
- Director of the Central Radiology Institute, Kepler Universitätsklinikum, Linz, Austria (formerly General Hospital Linz)
- 2005 – 2013
- Member of the research & development advisory board of the Universities of Applied Sciences Upper Austria
- 2007 – 2017
- Member of the supervisory board of Ars Electronica Linz GmbH
- 2012 – 2014
- Spokesman of the council of senior physicians of the General Hospital Linz
- 2014 – 2015
- President of the Medical Association for Upper Austria
- 2015 – 2016
- Past president of the Medical Association for Upper Austria
- 2005 – 2009
- Lecturer at the University of Applied Sciences Upper Austria for medical technology, Linz
- 2005 - 2010
- Lecturer at the medical training center of General Hospital Linz
- 2012 - 2016
- Lecturer at the Technical and Science Faculty of the Johannes Kepler University of Linz
Member of the advisory board of the medical technology cluster of Business Upper Austria agency in Upper Austria
Member of the advisory board of Ars Electronica Linz GmbH & Co KG
Lecturer at the University of Applied Sciences Upper Austria for health professions
Lecturer at the Medical Faculty of Johannes Kepler University of Linz
Organization and/or management of 25 national and international conventions and symposiums
Organization and/or management and implementation of more than 150 advanced training courses
Planning and development of the curriculum of human medicine for Johannes Kepler University of Linz (in the areas of radiology, medical technology, and virtual anatomy)
Reviewer for 11 national and international peer-reviewed professional medical journals
More than 1,700 national and international publications in journals and presentations as well as book contributions and academic works
1 book (“Fashion & Technology”)
12 art exhibitions on the topic of radiology and art (“Ars Intrinsica”) in Austria and Germany
Innovative clinical applications of magnetic resonance imaging
3D post-processing techniques for radiological cross-sectional imaging methods
New teaching methods in medicine and general education taking state-of-the-art digital possibilities into account
- Poster prize of the annual meeting of the European Society of Magnetic Resonance in Medicine and Biology (ESMRMB), Geneva
- Poster prize of the joint annual meeting of the associations for vascular and endovascular surgery (Germany, Switzerland, Austria), Hamburg
- Scientific award from the Medical Association of Upper Austria (Pilgerstorfer Prize)
- Editor’s Recognition Award from the European Journal of Radiology
- Editor’s Recognition Award from the European Journal of Radiology
- Best Scientific Contribution: “Challenges in Imaging,” awarded by the President of the European Society of Radiology
Dr.-Ing. Robert Schneider
- Born in Neumarkt in der Oberpfalz, Germany
- 1977 - 1981
- Elementary school, Neumarkt in der Oberpfalz
- 1981 - 1983
- Willibald-Gluck Gymnasium, Neumarkt in der Oberpfalz
- 1983 - 1987
- Secondary school, Neumarkt in der Oberpfalz
- 1987 - 1989
- Maximilian Kolbe Fachoberschule, Neumarkt in der Oberpfalz
- Bayerisches Hochbegabtenstipendium (Bavarian scholarship for highly gifted students)
- 1989 - 1990
- Military service
- 1990 - 1992
- Study of mathematics at Regensburg University of Applied Sciences
- Joined Siemens Studentenkreis
- 1992 - 1994
- Undergraduate studies in mathematics at the University of Erlangen-Nürnberg
- Intermediate diploma in mathematics at the University of Erlangen-Nürnberg
- 1994 - 1997
- Graduate studies in mathematics at the University of Erlangen-Nürnberg
- Degree in mathematics at the University of Erlangen-Nürnberg
- 1997 - 1999
- Doctoral student at the University of Erlangen-Nürnberg at the computer graphics lab
- 1999 - 2001
- Doctoral student at the Max-Planck Institute for Informatics in Saarbrücken in the computer graphics department
- PhD in engineering (Dr. Ing.) from Saarland University
- 2001 - 2007
- Computer scientist at Siemens Healthcare, Syngo
- Appointment as a Senior Key Expert for visualization and multicore
- 2007 - 2011
- Senior Key Expert for visualization and multicore at Siemens Healthcare, Syngo
- 2011 - 2014
- Innovation Manager at Siemens Healthcare, Syngo
- Appointment as a Principal Key Expert for visualization and multicore
- Since 2014
- Principal Key Expert for visualization and multicore at Siemens Healthineers, Diagnostic Imaging
Dr. rer. nat. Dipl.-Inf.(Univ.) Klaus Dieter Engel
Siemens Healthineers - Siemens Healthcare GmbH
Strategy and Innovation, Technology Center, Medical Imaging Tech
Phone: + 49 (0) 9131 / 84 60 65
Mobile: + 49 (0) 173 / 97 66 044
Siemens Healthineers - Siemens Healthcare GmbH
Phone: +49 (0)9131 84-3473
Fax: +49 (0)9131 84-3047
Mobile: +49 162 24 33 492
A description provided by the institutes and companies regarding their nominated projects
To the layperson, the output of magnetic resonance imaging (MRI) or computed tomography (CT) scans is often nothing more than a meaningless jumble of forms in different shades of grey. It takes a trained eye to identify exactly what the images show. What if the inside of a human body could be visualized in photographic detail – complete with shadows and depth, as vividly as in real life?
Such a notion is no longer the preserve of science fiction thanks to Dr. Klaus Engel, Dr. Robert Schneider, and Professor Dr. Franz Fellner. Cinematic Rendering is a visualization technique that was developed by Siemens Healthineers experts together with clinical partners, and which uses raw data from CT and MRT scans to produce photo- and hyper-realistic 3D images of the patient using the diagnostic radiology software Syngo.via.
Siemens Healthineers researchers took their inspiration from the film industry and used this as a basis for developing the new medical visualization technology: An animated figure like Gollum in the film adaptation of the “Lord of the Rings” trilogy, for example, is entirely lifelike rather than a foreign object, despite having been digitally modeled and then retroactively inserted into the scenes in which he appears. This is achieved via a technique known as image-based rendering. In this process, a spherical panorama is captured using a reflective sphere, which records the current light environment for subsequent application to image datasets.
At the core of this innovation is the physics of light. Rays of light are made up of particles called photons. Photons interact with their environment: When light encounters matter, it is reflected, bouncing off in various directions. In some places it is absorbed, resulting in shadows. Up to now, medical 3D imaging has not leveraged these special characteristics of light. Previously, a simple ray casting model was used that produced images without precisely depicted depths and tissue structures leading to less realistic 3D volume visualizations.
For the first time, cinematic rendering enables photorealistic medical images – in this case, CT and MRI images – using techniques from physics: An algorithm simulates the complex interactions between photons and the scanned images of the patient’s body. A randomized fraction of the most important of all the possible light paths is simulated using a Monte Carlo algorithm, and high dynamic range (HDR) light maps are used to define the light environment to generate realistic lighting effects. To make particular anatomical structures visible, the colors and transparencies of the MRI and CT acquisition data can be adjusted using a transfer function.
Dahinter stecken ein enormer Rechenaufwand und optimierte Algorithmen, denn es müssen Hunderte oder sogar Tausende Photonen-Wechselwirkungen pro Pixel berechnet werden – je nach Bildqualität. Im Gegensatz zur Filmindustrie wird in der medizinischen Anwendung nicht nur berechnet, wie das Licht an der Körperoberfläche reflektiert wird, sondern wie das Licht in das Gewebe eindringt und dort in die verschiedenen Richtungen streut. So können auch Effekte wie die sogenannte Umgebungsverdeckung modelliert werden, die beispielsweise die Tiefe einer Fraktur berücksichtigt: Je tiefer, desto weniger Licht kann eindringen und stärkere Schatten entstehen. Das Ergebnis sind eine nahezu realistische Abbildung von Frakturen, deutlich zu erkennende Organe und Gefäßverästelungen ─ abgesetzt voneinander durch Schattierungen und Tiefenwirkung.
The process requires vast amounts of computing power and highly optimized algorithms, as hundreds or even thousands of light paths per pixel are calculated depending on the image resolution. The difference compared with the animation industry is that, besides calculating how light is reflected off the surface of the body, cinematic rendering also takes into account how light penetrates tissue and is scattered in different directions. In this way effects such as ambient occlusion can be modeled, whereby the depth of a fracture is taken into account. The deeper the fracture, the less light is able to penetrate, resulting in a range of shadows. The result is an almost perfect depiction of fractures and clearly defined organs and blood vessels ─ easily discernible from each other thanks to the inclusion of subtle shading and depth effects.
Since the beginning of 2017, cinematic rendering has found uses beyond research. Radiologists have access to the technology with the latest version of the Syngo.via imaging platform, allowing photorealistic clinical images to be generated from any CT or MRI scan. This does not cause any additional radiation exposure to the patient as rendering occurs in the postprocessing stage: with just a few clicks, a patient’s images can be displayed on the monitor. Existing imaging data and images that have been generated using third-party scanners can also be displayed using cinematic rendering.
Using the new technology, a physician can access specific details about a patient’s anatomy and pathology, facilitating a high degree of precision when planning the procedure. There are obvious uses for cinematic rendering in other medical disciplines, too: A 3D view of CT and MRI scans provides surgeons with key spatial information, for example. This is particularly true for surgeons specializing in musculoskeletal disorders, who can achieve insights into the topography of more complex fractures from 3D images. It will also be very useful for the planning and follow-up of vascular, neurological, and craniofacial surgery procedures and interventions. Better visualization enables surgeons to plan their interventions with more confidence, reducing the risk of complications in some areas.
The technology can also be used to assist communication between the physician and patient – especially when it comes to explaining the procedure and risks of a planned intervention. Photorealistic images make it possible to illustrate clearly how a fracture develops or how a tumor grows.
The technology also opens up new possibilities for medical students to learn more about the body – rather than dissecting cadavers, they can now use cinematic rendering to become familiar with the inside of the human body. Students as well as medical personnel and research assistants can gain a clearer understanding of the bronchial tree in the lungs, for example. The technology can generate very specific or comprehensive images of the body, from bone structures to soft tissue. Up to now, healthcare professionals such as nursing staff and physiotherapists have not had access to the insights that medical students gain through dissection: Cinematic rendering changes this, offering entirely new possibilities for studying human anatomy for people working in the medical industry.
Whatever the application, cinematic rendering generates photorealistic images of the human body with an unprecedented degree of plasticity. To explore the full potential of the innovation, Siemens Healthineers has created a prototype for surgical procedure planning that has special tools and a customized user interface. At different trials in European university hospitals, the added value of the technology is currently being trialed in different application areas. For instance, the Cardiff University Brain Research Imaging Centre (CUBRIC), a center for neurological imaging in Wales, is using cinematic rendering to research nerve fibers in the brain with a focus on the causes and development of multiple sclerosis. By visualizing nerve fibers and lesions, the researchers aim to gain a clearer insight into the adverse effects caused by the illness.
Molecular imaging is a further application area: PET/CT image data is used to visualize metabolic activity to track tumor development. Cinematic rendering highlights the increased metabolism of cancer cells, enabling physicians to identify tumors more clearly. The capacity to visualize uric acid at molecular level using Dual Source CT data is also useful, as cinematic rendering clearly indicates the emergence of gout in the hands. Dynamic processes like blood circulation in the body can also be visualized.
In addition, researchers from Siemens Healthineers are collaborating with clinical partners to determine potential application scenarios with augmented and virtual reality. For all further developments, the objective for the technology is to support clinical workflows to the greatest possible extent while ensuring a simple and efficient design and never losing sight of the most important factor: The patient.
Syngo.via can be used as a standalone device or together with a variety of Syngo.via-based software options, which are medical devices in their own right. Syngo.via and the Syngo.via based software options are not commercially available in all countries. Due to regulatory reasons its future availability cannot be guaranteed. Please contact your local Siemens organization for further details.
Siemens Healthineers is the separately managed healthcare business of Siemens AG enabling healthcare providers worldwide to meet their current challenges and to excel in their respective environments. A leader in medical technology, Siemens Healthineers is constantly innovating its portfolio of products and services in its core areas of diagnostic and therapeutic imaging and in laboratory diagnostics and molecular medicine. Siemens Healthineers is also actively developing its digital health services and enterprise services. To help customers succeed in today’s dynamic healthcare marketplace, Siemens Healthineers is championing new business models that maximize opportunity and minimize risk for healthcare providers. In fiscal 2016, which ended on September 30, 2016, Siemens Healthineers generated revenue of €13.5 billion and net income of over €2.3 billion and has about 46,000 employees worldwide. Further information is available at www.siemens.com/healthineers.
The Kepler University Hospital was founded in response to the decision to create a medical faculty in Linz, Austria. It emanated from the merger of three respected hospitals at two locations and forms a high-performance medical centre for some 1.5 million Upper Austrian inhabitants. With roughly 6,100 employees and approximately 1,830 beds, the Kepler University Hospital constitutes Austria’s second largest hospital and unites around fifty areas of medicine, as well as specialists from every health-related profession. For patients, the Med Campus location offers a complete range of surgical, conservative and diagnostic treatment, and is also home to a gynaecological and paediatrics centre with a full range of competences. Furthermore, over the years the Neuromed Campus locations has established a reputation as an international neuro-medical centre, treating patients with brain and brain-related diseases, disorders of the spinal cord and nervous system, and mental illnesses.
The right to nominate outstanding achievements for the Deutscher Zukunftspreis is incumbent on leading German institutions in science and industry as well as foundations.
The project "Anatomy Meets Cinema - Cinematic Rendering" was submitted by the Federation of German Industries, BDI.
Winner 2017 · TEAM 2