Kyoto AR
Tour Guide App
Summary
To complete my Major Qualifying Project, WPI's undergraduate capstone project, I spent three months in Kyoto, Japan working with two other WPI students to develop a platform for augmented reality (AR) walking tour experiences.
Kinkaku-ji, the Temple of the Golden Pavilion, is one of Kyoto's most visited tourist attractions. However, most people who visit the site do not have the opportunity to experience the rich art and culture that are part of its history. We worked with our sponsor, Kyoto VR, to create an app that would allow visitors of Kinkaku-ji to better experience these aspects of the site. We used Unity to create an audio tour app that provides historical insight on the location, displays galleries of art throughout the visit, and offers accessibility features through AR. We also developed a web application to aid in the design and creation of these audio tours.
Background
In July 2019 I went to Kyoto, Japan for three months with six other WPI students. We stayed at Ritsumeikan University's Biwako-Kusatsu Campus (technically in Shiga prefecture, but close enough to Kyoto), and had access to their labs and faculty. My team consisted of three Seniors: myself, a computer science major; Cole Granof, computer science major; and Will Campbell, computer science and interactive media and game development double major.
In Japan we fulfilled the requirements for the Major Qualifying Project,
a WPI graduation requirement in which students solve a real-world
problem through practical project-based work. Our project was sponsored
by Kyoto VR, a startup founded in
2016 to "document and preserve the culture and heritage of the city and
communicate the richness of Japan's Old Capital to the world."
ARuko—An AR Walking Tour App
My team worked directly under Kyoto VR founder and CEO Atticus Sims to develop and test a mobile app to deliver an audio tour, incorporating various augmented reality features to enhance the experience. The app, which we named ARuko (from 歩こ aruko, roughly "let's walk," and AR) was created in Unity using the Vuforia Augmented Reality SDK. This allowed us to quickly develop a professional-level application, and easily deploy to both iOS and Android.
We focused on designing a tour of Kinkaku-ji for the app, but made it extensible so Kyoto VR could create tours of other locations using the same app in the future.
GPS Functionality
The core experience of ARuko is the walking tour. By walking the given route, the user enters and exits specific geographic regions that invoke changes in the app, such as playing a portion of the audio tour related to a specific area and displaying relevant images. We provided a robust system for defining polygons of arbitrary size and shape using GPS coordinates to represent the various points of interest along the tour.
Each region is a set of three or more latitude and longitude points that define a closed polygon. Using the phone's GPS location, the app needs to periodically check to see if the user is standing within one of these polygons.
To do this, we used an algorithm that draws a ray from the user's location to infinity in a random direction. The origin of the ray is inside a polygon if the ray passes through an odd number of the polygon's line segments (see the figure below).
This algorithm is rather straightforward and runs in O(n) time, where n is the total number of vertices of all polygons being tested.
However, there is an edge case that could have given us issues. If the ray passes exactly through a vertex, the collision algorithm will count two intersections instead of just one. Even though this was extremely unlikely, we avoided the edge case by subtracting one from the intersection count if any vertices were exactly colinear with the ray.
Translation Feature
ARuko uses augmented reality to translate text on the many Japanese-only signs at Kinkaku-ji. We visited the site and took dozens of high quality photos of every sign around the walking tour path. A Japanese student named Hikaru Inoue provided English translations of each of them. We then used the photos to generate Vuforia image targets.
When the user points the app at one of these signs, a blue "info" button will be overlaid on the image. If the user clicks this button, the translation is displayed.
This turned out to be a very intuitive way of delivering high-quality translations, and allowed non-Japanese speakers to understand more of the cultural and historical significance of Kinkaku-ji.
Map Overlay
Near the entrance to Kinkaku-ji is a large illustrated map. Much like the signage around the site, all of the text on this map is in Japanese only. Using the AR camera within the app we can overlay the walking path and translations of the labels.
The blue path and English text in the figure are not present in the real world, they are superimposed onto the camera image by ARuko in real time.
Editour—The Tour Editor
Alongside ARuko, Cole and I developed a web application called Editour to allow a non-technical user to design AR audio tours. Editour allows the user to draw arbitrarily-shaped geographic regions onto a map, edit those regions, and assign media to each of those regions.
The user can upload an audio file and any number of image files for each region. The server-side of the application organizes and version-controls these files and a definition of the various regions in the tour. It serves the most recent version of the tour either as a zipped archive or as an editable web page.
The user can draw and edit regions of any shape by drawing directly on an OpenStreetMap map. Through the menu on the left the user can edit the media files associated with each region, as well as uploading new ones.
Editour Back-End
I wrote the back-end of the Editour web application using Node.js with ECMAScript 2015 (also known as ES6). This allowed us to utilize some of the useful new features of the standard, such as modules and promises. I wrapped all of the asynchronous methods of the back-end in promises, which made error-handling and control flow much easier.
Chaining promises with consecutive then() calls is a
powerful asynchronous technique that is easy to read and understand.
Consider this snippet from the "get tours"
endpoint:
// read files from the tours directory
pHelpers
.getTours(constants.toursLoc)
.then(files => {
// find the full name of the tour
return pHelpers.findFileName(files, req.params.name);
})
.then(zipFile => {
logger.log("Sending file: " + zipFile);
res
.status(200)
.download(
constants.toursLoc + zipFile,
req.params.name + ".zip",
err => {
if (err) {
throw err;
}
}
);
})
.catch(errObj => {
// send errors back to the client
constants.returnError(res, errObj.status, errObj.message);
});
If an error is thrown at any point in the chain, control drops down to
the catch() clause and an error is returned to the client.
Tours are defined by a JSON metadata file, which contains the GPS coordinates, transcript, name, and files associated with each region. This JSON and each of the files is stored in a zip file on the server.
Extensibility
The main goal of Editour is to allow ARuko to be extensible. A non-technical user, such as our project sponsor, can use Editour to easily design new walking tours for other locations, and automatically load them into ARuko. We designed the mobile app to accept and parse the same zip files generated by Editour, so the integration is seamless.
We planned to have ARuko automatically request new tours from the Editour server, but didn't have time to implement this feature because our sponsor wanted to focus on just the Kinkaku-ji tour.
Conclusion
In three months we were able to create a functional and feature-complete application for delivering AR-enhanced audio tours. It was a great experience to work on a practical, real-world project for a real sponsor, and I learned a lot about the software development process.
It was also great to spend three months in Japan, learning about a foreign culture's approach to software development and computer science.
You can read our full 74 page paper here, and check out the source code for ARuko and Editour on GitHub.