Embodied Kinetic Orchestra. A research-and-development journey into how a living body — through movement, sound and light — can summon a responsive world in real time. This report documents what we set out to learn, what we built, and what we found.
EKO began from a single unifying idea: frequency. Sound is frequency. Light and colour are frequency. Movement is rhythm and oscillation. If everything vibrates, then a body in motion can become the instrument that tunes a whole environment — visuals, geometry and sound responding to it as one.
The name EKO — Embodied Kinetic Orchestra — captures the intent: the performer is not operating the work, they are the work. A gesture becomes a particle trail, a velocity becomes a grain of sound, a pose becomes a field of sacred geometry. The theme is deliberately universal — it crosses language, age, gender and culture — and is rooted in the shared human instinct that the world was sung, spoken or vibrated into being.
Phase One was never about a finished outcome. It was about proving the relationships are real, the tools are within reach, and the experience is worth building.
From our first conversations with The Civic Theatre, EKO was framed as three deliberate stages — each one earning the next.
Open research and development with no fixed outcome expected. Permission to experiment, fail, and discover what is actually possible.
A planned, public-facing installation built directly on the findings of the exploratory journey — the discoveries made tangible.
The seed of an annual festival — a breeding ground for further exploratory projects like EKO and a home for their findings.
A small, complementary team — one body, one ear, one hand on the technology.
The living input. Matt's choreography is the signal that drives the entire system, and the test of whether the response feels intuitive to a performer.
The sonic world. Jack shapes how gesture becomes sound — synthesis, grain, rhythm and the responsive sonic field that closes the loop.
The systems and the spectacle. Seán builds the capture pipeline, the generative visuals and the architecture that ties every experiment together.
Establish that movement, sound and visuals can be bound together in real time, with no perceptible lag between a body and its response.
Test whether a live, performer-reactive system is achievable with everyday equipment — a webcam and a laptop — rather than specialist motion-capture rigs.
Discover a coherent aesthetic — particles, trails and sacred geometry — that reads as a single, intentional system rather than a bag of effects.
Explore how the wider community could engage with the work — leading us to the idea of a heritage augmented-reality trail through the civic quarter.
Create a technical architecture where every experiment feeds the next, so the journey compounds instead of starting over each time.
Understand which artists, partners and audiences EKO should grow with as it moves toward a public installation.
Thirty-three pieces of exploration, in the order we made them. Each one's video loads automatically from its file in this folder; you can also override any clip in edit mode.
A study of frequency as the connective tissue beneath everything: light, sound, brainwaves, colour, shape, psychology and cymatics. This established the conceptual spine of EKO — the idea that movement, sound and visuals are all expressions of the same underlying vibration — and produced a reference mapping the full spectrum, from the visible and audible through to Solfeggio tones, chakra associations and the geometric figures of cymatics.
An investigation into visuals and sound driven in real time by movement. A single webcam registers the hands and facial expressions, turning the fingers into keys that play different instruments while simple visual effects fire alongside each note — the first proof that a body alone, with no controller, can perform.
A study of visuals driven purely by hand movement. The webcam tracks the hand and turns a fingertip into a brush or pencil that paints patterns and trails across the screen, testing how cleanly a gesture can be translated into persistent, expressive mark-making.
Building on the earlier studies, we created a virtual theremin to learn how sound can be manipulated and warped by the hands in front of a webcam — pitch, volume and timbre all bending to gesture, with no physical instrument involved.
A more poetic, experiential piece. A sleeping realm exists beneath the surface of our world; it responds to vibration — to dance, to rhythm, to the frequency of a person's presence. The audience moves their hands to paint with light — an early test of EKO's emotional register as much as its technology.
A study of sacred geometry — the patterns that cultures across the world have, independently, recognised as the architecture beneath things. We explored animated forms such as the Flower of Life, Metatron's Cube and the Sri Yantra, and how a field of particles can morph between them, building a visual language that frames the performer in meaningful, recognisable structure.
An investigation into FACS, the Facial Action Coding System. The webcam reads facial features and micro-expressions, which are mapped to emotions and in turn to colour frequencies — a route to letting a performer's emotional state, not just their movement, drive the visuals and palette.
Understanding full-body landmarks in the webcam feed: reading positional values and capturing real-time source data so those values can be mapped to audio and visuals in a controlled, predictable way. A grid overlay and trails on key landmarks made the raw data legible — the groundwork for everything that followed.
An exploration of narrative through movement, with the four seasons as the theme and a flower's journey through the cycle of life — from seed to bloom to decay and renewal. This gave EKO a story arc any audience can follow, independent of language, age or culture.
Investigating how to drive a rigged 3D object in real time through movement. A Blender flower — stem and petals — was rigged with drivers attached to pre-recorded motion-capture data, simulating the real-time workflow and proving that a performer's movement can animate a virtual object convincingly.
Investigating how to animate 3D characters directly from video motion capture, using Reallusion's iClone video mocap. Matt's recorded performance was retargeted onto a 3D avatar — a path toward populating the work with expressive virtual performers without a dedicated mocap stage.
Investigating how to displace and deform 3D geometry through movement in real time — a performer's motion physically reshaping a virtual object (here, a seed) rather than simply animating a rig, opening up more organic, sculptural responses to the body.
An early exploration into making culture and heritage more accessible by creating virtual replicas and bringing them to the audience. Using Reality Scan for camera alignment and photogrammetry, we produced a first, deliberately rough replica of the site as a proof of concept.
The first step of the heritage AR trail: an initial exploration of how text can be transformed into a form usable in augmented reality, so words and signage already in the real space can become interactive triggers and storytelling elements.
A rough block-out of an AR experience, using Blender's 3D environment to plan one of the trail's stations. The concept: a window display of books in the library opens up to expose a portal into the past.
A working prototype of the library window — a QR-activated AR application that uses the real window display as its image target. Crucially, lighting from the real world affects the 3D objects through AR light estimation, grounding the virtual content in the physical space.
A sample of existing artwork from the library walls coming alive once triggered in AR — a candidate station on the trail where a section of the narrative is uncovered through a piece the building already holds.
Drone footage of a park sculpture, captured to produce a high-quality 3D replica that can later be triggered at a station on the trail.
Using Reality Scan to align the captured images for a 3D scan, we explored 3D Gaussian Splatting as a format for a high-fidelity yet highly optimised replica of the sculpture, training a model to reconstruct the form from the aligned imagery.
Exploring the camera-culling process to achieve a portal effect, where the Gaussian-splat replica is only visible from inside the portal. The aim is a genuine sense of wonder — particularly for a younger audience stepping 'through' into another space.
Exploring AI tools to produce 3D characters that can be animated and used to help communicate the narrative at a station — a faster route to populating the trail with guides and storytellers.
A further exploration of narrative through movement, returning to the four seasons and the flower's journey through the cycle of life — this first phase developing the choreography and the staging of the piece.
The second phase of the four-seasons performance study, refining the movement and tightening its relationship to the unfolding narrative.
An alternative, full run of the four-seasons performance, bringing the developed phases together into a single complete piece for review.
The first working version of our custom mapping tool: a Unity game-engine application that reads around 1,800 face and body movements, with the ability to convert them into audio triggers and live manipulation. This is the engine the entire performance system is built on.
Developing the mapping system with visual grid overlays and feedback trails on certain landmarks, so that we — and ultimately the performer — can see exactly what the system is reading and responding to.
Extending the mapping beyond audio to additional game-object properties: a simple cube's transform and material colours, plus light intensity and colour — proving movement and expression can drive almost any parameter in the scene.
Nailing the mapping system so beats and instruments stay consistent and predictable — including tap dancing with a kick drum driven by the foot striking the floor, a satisfying and legible cause-and-effect for an audience.
Fine-tuning the system to work as a genuine performative piece: the hands controlling pitch and reverb, complemented by dance and full-body expression — the moment it crosses from instrument into performance.
A simple but effective study using the grid overlay and trails to understand the boundaries and limitations of the audio effects — mapping where the system is reliable and where it begins to break down.
Exploring narrative through full-body movement and comedic, slapstick action while manipulating the audio — both a creative test and a way to probe the system's limits under fast, exaggerated motion.
Adding visuals and real-time manipulation of game-object properties alongside the music, to understand how a complete spectacle would be performed through movement — audio, visuals and, potentially, stage lighting all driven by one body.
Exploring conversion of the mapping tool to Unreal Engine 5 — a AAA game engine used extensively in live performance, with native DMX lighting control and capabilities — positioning EKO to drive full stage productions.
As part of the journey, we asked how the wider community could step inside this work — not as spectators but as participants. The answer was a heritage augmented-reality trail that animates the spaces already around The Civic.
We explored four stations, each one triggering a narrative by using the existing spaces, artwork and visually appealing fixtures already in the area. Location and image-target AR triggers summon virtual characters, props and effects — a self-guided walk that doubles as a treasure hunt where the prize is the place itself.
Performance & myth — Partholón's arrival staged with virtual characters and a dramatic soundscape, anchored to the theatre's own frontage.
Knowledge & record — AR triggers surface archive material and the documented history of Tallaght from the library's existing features.
Contemporary art — a creative challenge from the artist group's space, linking the founding story to today's making and exhibition.
Civic future — the trail closes on community and place at the SDCC building, inviting visitors to leave their own mark.
Drawing the threads together, four conclusions stand out across the whole journey.
Sub-40ms capture-to-render keeps the visuals glued to the dancer. Bodies drive the work with no perceptible lag — the single most important proof of concept.
MediaPipe holds up to roughly four simultaneous skeletons with re-identification across frame drops — enough for collaborative, ensemble interaction.
Particles, trails and sacred geometry read as a single, intentional system rather than a collection of effects — the aesthetic holds together.
Sight and sound move together, born from the same gestures. Gesture-to-audio mapping feels authored, not automated — the loop closes.
Phase One was R&D with no required deliverable — yet it produced a substantial body of working tools and proofs.
Our immediate aim is to demonstrate the findings of this exploratory journey to The Civic Theatre. From there, the intention is to seek broader funding so that EKO can move into Phase Two: a planned public installation built directly on what we have learned.
Beyond that, the ambition is for EKO to scale up into a more full-time practice for those involved, drawing in other creatives and artists to extend the concept and encourage future collaboration. The longer arc — Phase Three — is the launch of an annual digital arts festival: a recurring breeding ground for exploratory projects like this one, and a public home for their findings.
Phase One asked whether any of this was real. The answer is yes — and the next chapters are about making it matter to a community.