In celebration of Singapore’s National Day, The Straits Times created Sounds of Singapore, an interactive article that pays homage to the nation’s iconic sounds using a fun quiz format.

Venturing across the island, we captured nine distinct sounds and scenes from diverse Singaporean soundscapes — ranging from bustling public housing estates to the lively underpasses of shopping districts. We filmed them in portrait orientation and recorded the sounds using multiple mics set up around the scene.

The audio files recorded were processed with three custom-built sound visualiser programmes, generating animated shapes based on the rhythm, texture, and musical tones of the sound data. Then we combined the audio, video, and animated shapes into 15-second video collage quizzes, and players had to guess the correct answer before the time was up.

Pushing the boundaries of sound visualisation

While most sound visualisers rely on volume-based waveforms, we took a creative leap by using the p5.js creative coding library and the machine learning tool ml5.js to develop three unique visualisers.

Here’s how they work:

No.1: Musical tones

This visualiser maps musical tones and pitches to colours and shapes, creating visuals that are vibrant and reminiscent of sheet music.

At its core, the programme listens to the audio’s frequency spectrum, breaking down sounds into their constituent pitches. Each pitch corresponds to a specific frequency, which drives the visual parameters. When a pitch is detected, shapes of the corresponding colour appear, with the shape’s width reflecting the duration of the pitch.

No. 2: Texture

Some sounds are more textural and complex — like the crackling of roasting chestnuts. This visualiser translates sound texture into radiating starburst shapes that evolve based on the audio’s amplitude. The louder the sound, the more energetic and intricate the starbursts become. To mimic the organic nature of sound, the lengths of the starburst rays are randomised.

Elements of symmetry, randomness, and responsiveness are combined to create a dynamic representation of sound texture that shifts and changes with the music. In this way, the visualiser portrays sound not just as a sequence of tones or beats, but as an evolving, multidimensional texture.

No. 3: Rhythm

This visualiser captures the rhythm of sound by generating floating shapes that appear in sync with peaks in the audio waveform.

The programme analyses the audio file’s peaks — where the sound’s intensity spikes — and when a peak surpasses a set threshold, a shape emerges and floats upward. From small chirps to hard chopping, five hard-edged shapes are designed to represent beats of varying intensity.

Micro-interactions and sound in UI

The article is enriched with micro-interactions, many of which are tied to sound. For example, enabling sound triggers a needle scratch and the familiar pin-drop sound of a record player starting.

The quiz buttons also feature animated shapes identical to those in the videos. When a correct or incorrect answer is selected, the corresponding sound is fed into the visualisers, and the resulting animation is overlaid onto the buttons. This not only enhances the feedback experience but also ensures a cohesive art direction throughout the piece.