VR-Heritage: Chinese Cultural Heritage in the Digital Age

Abstract

VR-Heritage, developed by Peking University, is a database that records hundreds of cultural heritage sites in China. It is an educational application that is oriented by teaching and research needs. Each cultural heritage site in the database is documented digitally using the tools of panoramic photography, photogrammetry, and 3D scanning and modeling. The process of building the database provides training opportunities for students and professionals in the field of heritage preservation. VR-Heritage is also a platform that enables students and scholars to ask questions and solve problems using the information in the database. Cultural heritage sites that soon will be damaged or disappear in the real world have been saved in the virtual world of VR-Heritage. The developer plans to add a timeline to the database in addition to the current Geographic Information System, which will allow users to relearn information of the past in both spatial and temporal contexts.

Introduction

In light of the recent data flood caused by the rapid development of 3D modeling and virtual reality technologies, researchers and preservers of cultural heritage sites now are facing new opportunities and challenges. As with personal computers and smartphones, virtual simulation devices that employ VR (virtual reality) and AR (augmented reality) technologies have great potential to become the next indispensable computation platform. Ever since 2016, “virtual reality’s breakthrough year,” virtual simulation devices have been used in numerous applications to enhance online tours and public education for cultural heritage sites in China.[1] Based on previously completed digital documentation and 3D scanning, for example, 360° virtual tours of major Buddhist cave sites in China are now available.[2] These Buddhist cave projects are among the earliest applications of these technologies in China. Here we will introduce a comprehensive database that incorporates several hundred cultural heritage sites in China and explore how this database promotes education and research in the fields of art, architecture, archaeology, history, and cultural heritage.

VR-Heritage: Project and Methodology

The VR-Heritage website, also known as the Panorama Database of Chinese Cultural Heritage (Zhongguo wenhua yichan quanjing ku 中国文化遗产全景库), is a database developed in 2017 by Peking University’s Experimental Teaching Center for Virtual Reality and Simulation in Archaeology (see http://www.vr-heritage.com/tour/dab88f0bfc4b6b78 for an accessible sample of the subscription-only website).[3] The database aims to document all valuable cultural heritage sites in China through the use of panoramic photography, photogrammetry, 3D scanning and modeling, and other VR technologies. Drones installed with digital cameras also are employed to record and measure archaeological sites and ancient architecture (as well as portable antiquities) by taking aerial pictures of each object. After all of the data from every site is collected, each is located on a map using the Geographic Information System (GIS). VR-Heritage currently stores records of around three hundred and fifty heritage sites and buildings dating from the tenth to the early twentieth century. Every site is searchable by name, location, and keywords (such as a particular architectural feature). Users also can browse and see the distribution of all sites on a Baidu map (a desktop and mobile web mapping application and technology; fig. 1).

Figure 1. VR-Heritage homepage, with view of all sites on the Baidu map

As for documentation of an individual site, for example, a Peking University research team investigated Zhenguo Monastery (Zhenguosi 鎮國寺), located in Pingyao, Shanxi Province, in 2017.[4] The main hall of the monastery, the Hall of Ten Thousand Buddhas (Wanfodian 萬佛殿), is dated to 963. One of the oldest extant timber buildings in China, it represents construction features of the tenth century. Both the wooden architecture and the sculptures in the main hall are appreciated greatly for their artistic and historical value (fig. 2). The research team used the aforementioned technologies to record the main hall, including the precious tenth-century wooden structure and its original Buddhist paintings and sculptures. First, drones were used to take pictures of the monastery complex from the air. Then panoramic photography was employed to record both the interior and exterior of each building in the monastery (fig. 3). Links to the panoramic photographs are pinpointed on the aerial picture of the site below according to the real locations where the photos were taken (fig. 4).

Figure 2. Front view of the Hall of Ten Thousand Buddhas (963) at Zhenguo Monastery, Pingyao, Shanxi Province

Figure 3. Panoramic photograph taken inside the Hall of Ten Thousand Buddhas at Zhenguo Monastery, Pingyao, Shanxi Province

Figure 4. Aerial view of Zhenguo Monastery taken by drone, with labels (linking to URLs) showing where the panoramic photographs were taken

In order to virtually reconstruct the structure of the main hall and its murals and sculptures, the team took hundreds of high-resolution photographs and used photogrammetric modeling to create a 3D model of the main hall with surface texture and color information. In this process, through utilization of the large number of high-resolution photographs, the objects are recognized by computer as tens of millions of points, each point with a location; these points are called the point cloud. The computer then generates meshes based on the point cloud. In this way, the 3D modeling of the surface is completed. Then, the high-resolution photos are applied as skins to precisely cover the surface of the 3D model of the building’s interior (fig. 5). The entire process is done automatically through use of commercial software. The links to the 3D models with surface texture also are displayed on the aerial view of the monastery.

Figure 5. The process of creating a virtual 3D model of the statues and architecture of the Hall of Ten Thousand Buddhas by use of commercial software called Agisoft Photoscan (now Metashape)

Enhancing Education with VR Technology and the Database

The VR-Heritage database is used not only for documentation; enhancing education is one of the developers’ primary motivations. Safe and comprehensive experiences in the field comprise an indispensable part of the teaching of art, architecture, archaeology, and cultural heritage. In the past, field trips were limited primarily by financial realities. With this database, however, students will be able to visit hundreds of sites virtually, without leaving the classroom. Moreover, even before a real-life field trip, they can preview the site in the database, make notes, and then make the best use of their time at the spot. The virtual tour in the database is also risk free. In the past, to see the roof frame or what is on top of the roof, students had to climb up to the beams or walk on the eaves. Now students can wear a VR headset connected to the database system and virtually stand on the ground or walk around a site. They can even jump onto a beam to view the paintings and the building from above (fig. 6). The database, which will be open access in the future, will enable all educators, regardless of their financial circumstances, to incorporate these virtual field trips into their classrooms.

Figure 6. Student wearing a VR headset (inset image at lower left) and virtually viewing a historic building in the database

In addition to teaching students about these cultural heritage sites, the VR-Heritage database also is useful for educating and training individuals to “preserve” more sites in the virtual-reality world. Under the circumstances of China’s rapid urbanization, numerous cultural heritage sites, especially those deemed not very important, are endangered. Preservers are racing to document the sites before they disappear, which is why students, professionals, and local administrators are involved in each process of recording a new site.

Along with education in the classroom, VR technologies can encourage public education, for example, by allowing museums to redesign and upgrade traditional exhibitions. In the spring of 2017, the Sackler Museum at Peking University exhibited high-resolution life-size photocopies of the wall paintings from Kaihuasi 开化寺, a Buddhist monastery in Shanxi Province dating from the Song dynasty (960–1279). Using this system, in addition to viewing the paintings in two dimensions, visitors were able to wear a VR headset and immerse themselves in the virtual environment of the monastery (as created by photogrammetric modeling) to observe paintings and architectural structures in their original spatial context.[5]

On the one hand, the VR-Heritage database will enhance educational programs; on the other, it helps to protect historical architecture from potential damage by flocking visitors. Thus, the technology also can minimize the risk that neglectful visitors may harm ancient art and architecture. With this tool, we can strike a balance between promoting awareness of these cultural heritage sites and reducing disturbance to sites where conditions are already fragile.

Compared to the traditional methods of surveying historical architecture or sites, digital documentation is advantageous in terms of efficiency, accuracy, and the minimization of physical intervention for endangered objects. It must be added, however, that in-person observations are still indispensable even with the technologies that we have today. For the purpose of deep learning, we still need to take notes, make drawings, and take photos of details on site, as no technology will allow us to touch the traces left on the buildings or feel the textures of the materials. The visuality that we acquire through small screens cannot replace the materiality that we experience onsite.

Relearning the Past

Admittedly, the VR technologies available today, either panoramic photography or photogrammetric modeling, are no longer on the cutting edge. It is now much easier than ever to digitize ancient architecture and other cultural heritage with the booming of VR and AR technologies and compatible devices, and with the popularity of drones, 360° panoramic photography, photogrammetry, and 3D scanning technologies in research institutes in China. The questions of how to intensively adapt the digital data into the professional and academic environments, however, and how to combine the information into research resources that will promote the scholarship of art and architectural history are critical challenges for all of the developers.

One immediate benefit of this database for art and architectural historians is to track the spread and development of certain architectural or artistic features in China. The database, which has recorded hundreds of historic Chinese buildings, provides a tool for using tags to mark styles and features of carpentry construction in these buildings. These tags connect all buildings that share the same style or feature, enabling users to search for and locate them quickly. Like tagging one or more friends on Facebook, we can tag one or more features on a building (fig. 7). For example, if we search for the term xiegong 斜栱 (oblique arms), a projecting bracket on top of a column at an oblique angle of thirty to forty-five degrees, we will be able to see all buildings in the database with this feature distributed on a map. Furthermore, on this map, we can observe that xiegong often may be found in northern, northeastern, and southwestern China (fig. 8). It is much easier and more efficient for the user to discover and understand a certain trend through the database than by reading multiple articles to learn about individual discoveries.

Figure 7. Tags for multiple features of timber construction on the Middle Buddha Hall (971) of Chongming Monastery, Gaoping, Shanxi Province

Figure 8. Distribution of all buildings with xiegong on the Baidu map (with inset photo showing the feature itself)

If more and more keywords are recorded and tagged through scholarly collaboration, the VR-Heritage database can help scholars, professors, and students to discover new areas of inquiry and generate new research topics. Users can group their learning objects according to dynasties or geographic regions. For example, several research themes already have been developed during the process of selecting objects, such as “Song–Jin architecture in southeast Shanxi,” “Yuan–Ming architecture in Sichuan,” and “Liao pagodas in Inner Mongolia and Liaoning.” The use of such themes bears similarities to some other digital humanities databases that employ GIS, such as the Chronicle Atlas of Tang–Song Literature and the Digital Atlas of Roman and Medieval Civilizations (DARMC).[6] Both projects serve as good examples of how to integrate isolated pieces of historical information into one geographic system. Although we do believe that in-person field trips are still indispensable for advanced researchers, the database, first, helps researchers to make travel plans with a clear purpose and smart route; second, allows them to use their time most efficiently at each site; and, lastly, creates a community of scholars who are interested in the same topic.

VR technology, incorporated with GIS, further promotes interdisciplinary studies of art and architecture. As we have seen in the example of Zhenguo Monastery, this technology can combine architecture, paintings, sculptures, inscriptions, and other archeological materials in the virtual space. One research topic can be searched and studied across different fields.

Conclusion: Challenges and Prospects

The availability of digitizing tools has increased rapidly; even in our daily lives, everyone easily can take hundreds of high-resolution pictures with a smartphone. The more these technologies are available, the more data are created. Some (if not all) of the data could be isolated fragments of information. Our scholarship and the material environment all will be digitized as fragments, and we will be lost unless this information is organized effectively according to its spatial and temporal context. The more fragmentized and digitized the data is, the more specific the temporal and spatial framework needs to be. Placing works in the context of time and space is not a new idea for art and architectural historians. This is also the basic concept and tool for research in many disciplines, such as archaeology and historical geography. In these disciplines, such training has been emphasized in teaching and conducting research. In the future of digital humanities, when VR and AR technologies are used widely, we may no longer have to stress this training. By that time, the temporal and spatial framework will be too necessary to be ignored, as most databases will be built on this concept.

In anticipation of these future circumstances, the developers of the VR-Heritage database plan to add a timeline to integrate the spatial and temporal information for every site. The future prospect of digital humanities is to contextualize visual or archaeological materials in a visualized geographic environment. It is essential to integrate spatial as well as temporal information from the macro to the micro level. For instance, the macro level could be the geographic position of a building or cave, and the micro level could be the accurate position of a stele, painting, construction element, or other component in a monastery. Such a comprehensive view also would include visual data and historical textual data (i.e., text from inscriptions on a stele, on a building, or from local gazetteers and personal journals), placed in a temporal and spatial framework.[7]

Non-commercial developers do not use VR technologies with the purpose of ambitiously occupying the market (comprised largely of schools, museums, and other educational institutions). Educational applications always should prioritize the production of high-quality content, instead of merely relying on or showing off VR devices and technologies with superficial content. Sometimes it is easier to invest money in equipment (such as devices and software) than in scholarly production and curatorial practice. The VR-Heritage database, on the contrary, is an excellent example of how new technologies can solve teaching and research problems.

VR-Heritage will facilitate scholarly collaboration and publication. The primary purpose of the database is to serve students and scholars. The database provides a friendly community for people who are interested in the same object and who are willing to contribute their knowledge about particular objects. For the future, we also envision VR-Heritage as a reliable platform where scholars can publish their firsthand data from trips to the field, such as images of building structures, steles, and wall paintings. In other words, art and architectural historians will not have to rely merely on information acquired from their own experience or to wait for a publication to become available in print. Even now, they can collaborate with people in the VR-Heritage community and accelerate the publication of their new research findings. Moreover, given the GIS information built into the database and the timeline that will be added in the near future, this database is especially useful for transregional and transdynastic studies on heritage sites in China.

In addition to the numerous benefits discussed above, challenges remain for the application of VR and other digital technologies, and questions and concerns still need to be addressed. Preservers may contemplate how to balance user convenience with safety, and whether publishing graphic data online could raise the risk of theft and vandalism at critical cultural heritage sites. Today’s art and architectural historians must learn and keep up with the rapid advancements in digital technology. Most importantly, the question of how digital tools and data eventually can promote, rather than kill, modern scholarship will continue to be debated.

Jianwei Zhang, PhD (Southeast University, Nanjing, China), 2013, is an assistant professor and deputy dean of the School of Archaeology and Museology, and associate director of the Experimental Teaching Center for Virtual Reality and Simulation in Archaeology at Peking University. He is the author of Chinese Ancient Metal Architecture (Southeast University Press, 2015) and dozens of scholarly articles on archaeology, architectural history, and heritage preservation. Zhang also serves as the principal investigator for multiple cultural heritage projects that have received national funds. E-mail: [email protected]

Lala Zuo, PhD (University of Pennsylvania), 2010, is an assistant professor of art history at NYU Shanghai. She is the author of Diversity under the Great Unity: Yuan Regional Architecture (University of Hawai‘i Press, 2019) and multiple scholarly articles about Chinese architecture and cultural heritage. E-mail: [email protected]

Notes

1. Mindi Chahal, “Why 2016 will be virtual reality’s breakthrough year,” January 28, 2016, https://www.marketingweek.com/2016/01/28/why-2016-will-be-a-breakthrough-year-for-virtual-reality/ (last accessed January 22, 2020).

2. These Buddhist cave sites include the Mogao Caves in Dunhuang, Gansu Province; the Yungang Caves in Datong, Shanxi Province; and the Longmen Caves in Luoyang, Henan Province.

3. The Center, known as Beijing daxue kaogu xuni fangzhen (guojiaji) shiyan jiaoxue zhongxin 北京大学考古虚拟仿真（国家级）实验教学中心, was established in 2016. The database is currently open to individuals and institutions that have subscriptions, which are free of charge; inquiries and applications for subscriptions should be sent to the Center by e-mail, to [email protected]. The current interface of the database is only available in Chinese, but plans are in place to provide an English-language interface and to make the database available to the public through Peking University’s library by 2025.

4. Zhenguosi is located fifteen kilometers northeast of Pingyao, Shanxi Province. The monastery, which attracts many tourists for its architecture and Buddhist sculptures, was listed as a World Cultural Heritage site in 1997 as part of the ancient city of Pingyao. Zhenguosi now consists of the Gate Hall, the Bell Tower, the Drum Tower, the Hall of Ten Thousand Buddhas (the main hall), the Guanyin Hall, the Dizang Hall, and the Three Buddha Hall. For details of the monastery and its Hall of Ten Thousand Buddhas, see Nancy Steinhardt, Liao Architecture (Honolulu: University of Hawai‘i Press, 1997), 76–80.

5. For the digital project about Kaihuasi, see Jianwei Zhang and Lala Zuo, “Kaihuasi: Buddhist Art and Architecture in Virtual Reality,” IIAS Newsletter 79 (March 2018): 19; https://www.iias.asia/the-newsletter/newsletter-79-spring-2018 (last accessed February 19, 2020). The same objects were shown again at the exhibition “Virtual Journeys: Chinese Buddhist Art and Architecture in the Digital Era” at Stony Brook University in New York from September 27 to December 15, 2018.

6. The Chronicle Atlas of Tang–Song Literature has been developed by Professor Wang Zhaopeng of South-Central University for Nationalities, Wuhan, China. See https://sou-yun.com/poetlifemap.html (last accessed January 23, 2020). The Digital Atlas of Roman and Medieval Civilizations has been developed by Harvard University, Cambridge, Massachusetts. See https://darmc.harvard.edu/ (last accessed January 23, 2020).

7. In fact, we can observe the same course during the development of Google Earth and Facebook. Early versions of Google Earth only had geographic information without a timeline. Later on, a timeline was added so that viewers could see the earth at different points in time. Similarly, Facebook added a timeline several years after the social network platform was established.