The System
Dead Reckoning
Custom Software / Interactive Audiovisual Installation2026
The System
i) System Overview
Dead Reckoning is built as a navigational system in which a projected voyage in the sea, a physical boat's wheel controller, sound, and a circle of illuminated photographs respond to the same movement. Standing at its center, the player steers the wheel. The current carries it forward automatically, while the wheel controls only its heading. This partial transfer of authority reflects the work’s model of recollection: the player can steer through the photographs and actively shape the connections between incomplete fragments, but cannot stop, reverse, or determine an arrival because the past to which those images refer can no longer be returned to or verified.In the system, a rotary encoder and Arduino transmit the wheel’s movement to Unity, where the photographic sequence is progressively revealed along the steered route. In the installation, five original iCloud thumbnails—the images that led me to photograph the in-betweens—are framed and suspended around the wheel as physical positions within the sequence. Responsive LED lights indicate which part of the sequence the player is approaching, while the score follows the speed of revelation. Together, movement, image, light, and sound form a multisensorial perceptual experience.
Fig. 1.1. System Flowchart
ii) The Sequence: Five Thumbnails and Their In-betweens
An iCloud malfunction erased my cellphone photographs from 2024–2025, leaving only five low-resolution thumbnails. The accident exposed both the fragility of this photographic infrastructure and my reliance on it as an external support for memory within my research into migratory recollection.Throughout 2025 and 2026, I returned to the five locations, photographed their routes and surroundings, and re-enacted fragments of the original moments in places that had changed unevenly. Informed by David Hockney’s photographic “joiners,” I arranged these fragments through compositional continuity into a cyclic sequence—a ring of impossible narrative—concealing the transitions until recorded evidence and reconstructed memory became difficult to distinguish. When a reconstructed past becomes plausible enough to navigate, does it also begin to feel real?Fig. 2.1. Sequencing in Unity Engine
iii) Software Development
a) SpawningThe first prototype spawned flat image panels edge to edge on the player’s left, placing the sequence beside rather than ahead of the voyage. Attempts to add depth and overlap caused the panels to fan apart during turns, so I redesigned the revealing of the sequence as a coplanar strip of curved meshes that follows the steered centerline, the Ribbon.Fig. 3.1.1. Sketch for the fanning model and the Ribbon
Fig. 3.1.2. Test on the Ribbon in Unity Engine
b) The Sailing SimulationEarly tests mapped steering input directly to heading, producing abrupt sawtooth. I did research on how boat reacts to waters and replaced this with a three-layer simulation: wheel input moves a virtual rudder, rudder angle and speed generateyaw torque through a lift-and-stall curve, and hull inertia and drag integrate that force into heading. Turns therefore build, stall, and settle over time.
Fig. 3.2.1. Simplified Free-Body Diagram of the Vessel
Fig. 3.2.2. (top) Docking Stimulating
Fig. 3.2.3. (bottom) Left turn clamped by yaw torque and brought back by hull inertia.
c) Overlook State The Overlook state closes the voyage by shifting the player from navigator to observer, allowing the accumulated route to be viewed as a whole. During navigation, sampled positions are stored as a breadcrumb buffer; then a EndGame script interpolates through this cached path in reverse to create a smooth trace-back without recalculating the journey.
Fig. 3.4. Overlook state showing the completed route from an elevated viewpoint.
iv) Physical Computing
a) The WheelData Transmission
FabricationThe controller was built around a found, full-size ship’s wheel. Most units were 3D printed for connecting its standard-size shaft to the rotary encoder. Cut wood panels was constructed to make a waist-high, arcade-style stand.
Fig. 4.1.1. 3D printed shaft holder, adaptors in UltiMaker Cura
Fig. 4.1.2. fabrication details of the wheel
b) The LED MapTo make the cyclic structure of the sequence perceptible and clue the player in on a sense of their position, five spotlights are installed at the base of the wooden pedestal, illuminating the five suspended iCloud thumbnails according to the current in-game progress. Unity maps the current position to five brightness values and sends them to the Arduino through USB serial. (e.g. at 30% of the distance from image A to B, their brightness values are 70% and 30% respectively.) The Arduino converts these values into high-frequency PWM signals, while MOSFETs control the power of the 12-volt spotlights, producing a continuous crossfade between neighboring images.
Fig. 4.2.1. Demo on one image, blinking at peak
Fig. 4.2.2. Complete Arduino diagram
v) Installation
a) SoundThe soundscape was developed through a “reversed VJ” method: rather than making the image react to music, the score by Yixiao Liu was composed in proportion to the spatial length of each section of the photographic ribbon. As steering changes the speed of revelation, Unity accelerates or slows the main soundtrack, while reciprocal pitch correction preserves its original pitch.Fig. 5.1.1. Custom Unity Sound Map aligning the sequence with the soundtrack waveform
b) Final InstallationDead Reckoning was installed in a 6 × 4 m room (approximately 20 × 13 ft) in Suzhou, China. The five original iCloud thumbnails were mounted in frames and suspended from ceiling hooks with fishing line, forming a circular arrangement around the helm. The LED light map was mounted at the base of the pedestal illuminated each image, while the speaker connected the projected voyage to the surrounding physical space.
Fig. 5.2.1. Final installation of Dead Reckoning (projector off, LED lights all on at rest)
The player enters the centre of the installation and generates a spatial audiovisual experience. The interaction itself then becomes live performance, allowing surrounding visitors to experience the navigation.
Fig. 5.2.2. A player steering
back to work list