CC-Tweaked/doc/guides/speaker_audio.md

module: [kind=guide] speaker_audio see: speaker.playAudio Play PCM audio using a speaker. see: cc.audio.dfpwm Provides utilities for encoding and decoding DFPWM files.

Playing audio with speakers

CC: Tweaked's speaker peripheral provides a powerful way to play any audio you like with the @{speaker.playAudio} method. However, for people unfamiliar with digital audio, it's not the most intuitive thing to use. This guide provides an introduction to digital audio, demonstrates how to play music with CC: Tweaked's speakers, and then briefly discusses the more complex topic of audio processing.

A short introduction to digital audio

When sound is recorded it is captured as an analogue signal, effectively the electrical version of a sound wave. However, this signal is continuous, and so can't be used directly by a computer. Instead, we measure (or sample) the amplitude of the wave many times a second and then quantise that amplitude, rounding it to the nearest representable value.

This representation of sound - a long, uniformally sampled list of amplitudes is referred to as Pulse-code Modulation (PCM). PCM can be thought of as the "standard" audio format, as it's incredibly easy to work with. For instance, to mix two pieces of audio together, you can just add samples from the two tracks together and take the average.

CC: Tweaked's speakers also work with PCM audio. It plays back 48,000 samples a second, where each sample is an integer between -128 and 127. This is more commonly referred to as 48kHz and an 8-bit resolution.

Let's now look at a quick example. We're going to generate a Sine Wave at 220Hz, which sounds like a low monotonous hum. First we wrap our speaker peripheral, and then we fill a table (also referred to as a buffer) with 128×1024 samples - this is the maximum number of samples a speaker can accept in one go.

In order to fill this buffer, we need to do a little maths. We want to play 220 sine waves each second, where each sine wave completes a full oscillation in 2π "units". This means one seconds worth of audio is 2×π×220 "units" long. We then need to split this into 48k samples, basically meaning for each sample we move 2×π×220/48k "along" the sine curve.

local speaker = peripheral.find("speaker")

local buffer = {}
local t, dt = 0, 2 * math.pi * 220 / 48000
for i = 1, 128 * 1024 do
    buffer[i] = math.floor(math.sin(t) * 127)
    t = (t + dt) % (math.pi * 2)
end

speaker.playAudio(buffer)

Streaming audio

You might notice that the above snippet only generates a short bit of audio - 2.7s seconds to be precise. While we could try increasing the number of loop iterations, we'll get an error when we try to play it through the speaker: the sound buffer is too large for it to handle.

Our 2.7 seconds of audio is stored in a table with over 130 thousand elements. If we wanted to play a full minute of sine waves (and why wouldn't you?), you'd need a table with almost 3 million. Suddenly you find these numbers adding up very quickly, and these tables take up more and more memory.

Instead of building our entire song (well, sine wave) in one go, we can produce it in small batches, each of which get passed off to @{speaker.playAudio} when the time is right. This allows us to build a stream of audio, where we read chunks of audio one at a time (either from a file or a tone generator like above), do some optional processing to each one, and then play them.

Let's adapt our example from above to do that instead.

local speaker = peripheral.find("speaker")

local t, dt = 0, 2 * math.pi * 220 / 48000
while true do
    local buffer = {}
    for i = 1, 16 * 1024 * 8 do
        buffer[i] = math.floor(math.sin(t) * 127)
        t = (t + dt) % (math.pi * 2)
    end

    while not speaker.playAudio(buffer) do
        os.pullEvent("speaker_audio_empty")
    end
end

It looks pretty similar to before, aside from we've wrapped the generation and playing code in a while loop, and added a rather odd loop with @{speaker.playAudio} and @{os.pullEvent}.

Let's talk about this loop, why do we need to keep calling @{speaker.playAudio}? Remember that what we're trying to do here is avoid keeping too much audio in memory at once. However, if we're generating audio quicker than the speakers can play it, we're not helping at all - all this audio is still hanging around waiting to be played!

In order to avoid this, the speaker rejects any new chunks of audio if its backlog is too large. When this happens, @{speaker.playAudio} returns false. Once enough audio has played, and the backlog has been reduced, a @{speaker_audio_empty} event is queued, and we can try to play our chunk once more.

Storing audio

PCM is a fantastic way of representing audio when we want to manipulate it, but it's not very efficient when we want to store it to disk. Compare the size of a WAV file (which uses PCM) to an equivalent MP3, it's often 5 times the size. Instead, we store audio in special formats (or codecs) and then convert them to PCM when we need to do processing on them.

Modern audio codecs use some incredibly impressive techniques to compress the audio as much as possible while preserving sound quality. However, due to CC: Tweaked's limited processing power, it's not really possible to use these from your computer. Instead, we need something much simpler.

DFPWM (Dynamic Filter Pulse Width Modulation) is the de facto standard audio format of the ComputerCraft (and OpenComputers) world. Originally popularised by the addon mod Computronics, CC:T now has built-in support for it with the @{cc.audio.dfpwm} module. This allows you to read DFPWM files from disk, decode them to PCM, and then play them using the speaker.

Let's dive in with an example, and we'll explain things afterwards:

local dfpwm = require("cc.audio.dfpwm")
local speaker = peripheral.find("speaker")

local decoder = dfpwm.make_decoder()
for chunk in io.lines("data/example.dfpwm", 16 * 1024) do
    local buffer = decoder(chunk)

    while not speaker.playAudio(buffer) do
        os.pullEvent("speaker_audio_empty")
    end
end

Once again, we see the @{speaker.playAudio}/@{speaker_audio_empty} loop. However, the rest of the program is a little different.

First, we require the dfpwm module and call @{cc.audio.dfpwm.make_decoder} to construct a new decoder. This decoder accepts blocks of DFPWM data and converts it to a list of 8-bit amplitudes, which we can then play with our speaker.

As mentioned above, @{speaker.playAudio} accepts at most 128×1024 samples in one go. DFPMW uses a single bit for each sample, which means we want to process our audio in chunks of 16×1024 bytes (16KiB). In order to do this, we use @{io.lines}, which provides a nice way to loop over chunks of a file. You can of course just use @{fs.open} and @{fs.BinaryReadHandle.read} if you prefer.

Processing audio

As mentioned near the beginning of this guide, PCM audio is pretty easy to work with as it's just a list of amplitudes. You can mix together samples from different streams by adding their amplitudes, change the rate of playback by removing samples, etc...

Let's put together a small demonstration here. We're going to add a small delay effect to the song above, so that you hear a faint echo a second and a half later.

In order to do this, we'll follow a format similar to the previous example, decoding the audio and then playing it. However, we'll also add some new logic between those two steps, which loops over every sample in our chunk of audio, and adds the sample from 1.5 seconds ago to it.

For this, we'll need to keep track of the last 72k samples - exactly 1.5 seconds worth of audio. We can do this using a Ring Buffer, which helps makes things a little more efficient.

local dfpwm = require("cc.audio.dfpwm")
local speaker = peripheral.find("speaker")

-- Speakers play at 48kHz, so 1.5 seconds is 72k samples. We first fill our buffer
-- with 0s, as there's nothing to echo at the start of the track!
local samples_i, samples_n = 1, 48000 * 1.5
local samples = {}
for i = 1, samples_n do samples[i] = 0 end

local decoder = dfpwm.make_decoder()
for chunk in io.lines("data/example.dfpwm", 16 * 1024) do
    local buffer = decoder(chunk)

    for i = 1, #buffer do
        local original_value = buffer[i]

        -- Replace this sample with its current amplitude plus the amplitude from 1.5 seconds ago.
        -- We scale both to ensure the resulting value is still between -128 and 127.
        buffer[i] = original_value * 0.6 + samples[samples_i] * 0.4

        -- Now store the current sample, and move the "head" of our ring buffer forward one place.
        samples[samples_i] = original_value
        samples_i = samples_i + 1
        if samples_i > samples_n then samples_i = 1 end
    end

    while not speaker.playAudio(buffer) do
        os.pullEvent("speaker_audio_empty")
    end

    -- The audio processing above can be quite slow and preparing the first batch of audio
    -- may timeout the computer. We sleep to avoid this.
    -- There's definitely better ways of handling this - this is just an example!
    sleep(0.05)
end

:::note Confused? Don't worry if you don't understand this example. It's quite advanced, and does use some ideas that this guide doesn't cover. That said, don't be afraid to ask on GitHub Discussions or IRC either! :::

It's worth noting that the examples of audio processing we've mentioned here are about manipulating the amplitude of the wave. If you wanted to modify the frequency (for instance, shifting the pitch), things get rather more complex. For this, you'd need to use the Fast Fourier transform to convert the stream of amplitudes to frequencies, process those, and then convert them back to amplitudes.

This is, I'm afraid, left as an exercise to the reader.