Getting started with programmatic audio

37 down vote accepted

As some has pointed out in the comments, what you want to look into is PCM audio. In a nutshell, sound is a wave that travels through air. In order to capture that sound, we use amicrophone, which contains a membrane which will vibrate as the sound waves hit it. This vibration is translated into an electric signal, where the voltage goes up and down. This change in voltage is then changed into a digital signal by an analog-to-digital converter (ADC) by sampling a certain number of times a second ("sampling rate" - the 44 KHz, or 44,100 samples per second) and, in the current case, stored as a pulse-code modulated (PCM) audio data. A speaker works in opposite; the PCM signal is converted to digital by an digital-to-analog converter(DAC), then the analog signal goes to the speaker where it will vibrate a membrane which produces vibrations in the air which results in sound. Manipulating Audio There are many libraries out there for many languages that you can manipulate audio with, however you've marked this question as "language-agnostic", I'll mention a few simple ways (as that's all I know!) that you'll be able to manipulate audio in your preferred language. I'll present the code samples in pseudocode. The pseudocode will have each audio sample have an amplitude in the range of -1 to 1. This will be dependent on the data type you are using for storing each sample. (I haven't dealt with 32-bit floats before, so this may be different.) Amplification In order to amplify the audio, (therefore, increasing the volume of the sound) you'll want to make the vibration of the speakers to be larger so the magnitude of the sound wave is increased. In order to make that speaker move more, you'll have to increase the value of each sample: original_samples = [0, 0.5, 0, -0.5, 0] def amplify(samples): foreach s in samples: s = s * 2 amplified_samples = amplify(original_samples) // result: amplified_samples == [0, 1, 0, -1, 0] The resulting samples are now amplified by 2, and on playback, it should sound much louder than it did before. Silence When there are no vibrations, there is no sound. Silence can be achieved by dropping each sample to 0, or to any specific value, but does not have any change in amplitude between samples: original_samples = [0, 0.5, 0, -0.5, 0] def silence(samples): foreach s in samples: s = 0 silent_samples = silence(original_samples) // result: silent_samples == [0, 0, 0, 0, 0] Playing back the above should result in no sound, as the membrane on the speaker is not moving at all, due to a lack of change in amplitude in the samples. Speed Up and Down Speeding things up and down can be achieved in two ways: (1) changing the playback sampling rate or (2) changing the samples themselves. Changing the playback sampling rate from 44100 Hz to 22050 Hz will decrease the speed of playback by 2. This will make the sound slower and lower in tone. Going from a 22 KHz source and playing back at 44 KHz, the sound will be very fast and high pitched like birds chirping. Changing the samples themselves (and keeping a constant playback sampling rate) means that samples either (a) get thrown out or (b) are added in. To speed up the playback of the audio, throw out samples: original_samples = [0, 0.1, 0.2, 0.3, 0.4, 0.5] def faster(samples): new_samples = [] for i = 0 to samples.length: if i is even: new_samples.add(samples[i]) return new_samples faster_samples = faster(original_samples) // result: silent_samples == [0, 0.2, 0.4] The result of the above program is that the audio will speed up by a factor of 2, similar to playing back an audio that sampled at 22 KHz at 44 KHz. To slow down the playback of the audio, throw in a few samples: original_samples = [0, 0.1, 0.2, 0.3] def slower(samples): new_samples = [] for i = 0 to samples.length: new_samples.add(samples[i]) new_samples.add(interpolate(s[i], s[i + 1])) return new_samples slower_samples = slower(original_samples) // result: silent_samples == [0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3] Here, extra samples were added, thereby slowing down the playback. Here, we have aninterpolation function that makes a "guess" as to how to fill in that extra space that much be added. Spectrum Analysis and Sounds Modification by FFT Using a technique called Fast Fourier transform (FFT), the sound data in the amplitude-time domain can be mapped to the frequency-time domain to find out the frequency components of audio. This can be used to produce the spectrum analyzers that you might see on your favorite audio player. Not only that, since now you have the frequency components of the audio, if you change the amount of If you want to cut-off certain frequencies, you can use FFT to transform the sound data into the frequency-time domain, and zero-out the frequency components that are not desired. This is calledfiltering. Making an high-pass filter, which allows frequencies above a certain frequency can be performed like this: data = fft(orignal_samples) for i = (data.length / 2) to data.length: data[i] = 0 new_samples = inverse_fft(data) In the above example, all frequencies over the half-way mark is cutoff. So, if the audio could produce 22 KHz as the maximum frequency, any frequency above 11 KHz will be cut out. (For audio played back at 44 KHz, the maximum theoretical frequency that can be produced is 22 KHz. See Nyquist–Shannon sampling theorem.) If you want to do something like increase the low-frequency range (similar to the bass boost effect), take the lower-end of the FFT-transformed data and increase its magnitude: data = fft(orignal_samples) for i = 0 to (data.length / 4): increase(data[i]) new_samples = inverse_fft(data) This example increases the lower quarter of the frequency components of the audio, leading to the low frequencies to become louder. There are quite a few things that can be done to the samples to manipulate the audio. Just go ahead and experiment! It's the most exciting way to learn. Good luc

37 down vote accepted

As some has pointed out in the comments, what you want to look into is PCM audio. In a nutshell, sound is a wave that travels through air. In order to capture that sound, we use amicrophone, which contains a membrane which will vibrate as the sound waves hit it. This vibration is translated into an electric signal, where the voltage goes up and down. This change in voltage is then changed into a digital signal by an analog-to-digital converter (ADC) by sampling a certain number of times a second ("sampling rate" - the 44 KHz, or 44,100 samples per second) and, in the current case, stored as a pulse-code modulated (PCM) audio data. A speaker works in opposite; the PCM signal is converted to digital by an digital-to-analog converter(DAC), then the analog signal goes to the speaker where it will vibrate a membrane which produces vibrations in the air which results in sound. Manipulating Audio There are many libraries out there for many languages that you can manipulate audio with, however you've marked this question as "language-agnostic", I'll mention a few simple ways (as that's all I know!) that you'll be able to manipulate audio in your preferred language. I'll present the code samples in pseudocode. The pseudocode will have each audio sample have an amplitude in the range of -1 to 1. This will be dependent on the data type you are using for storing each sample. (I haven't dealt with 32-bit floats before, so this may be different.) Amplification In order to amplify the audio, (therefore, increasing the volume of the sound) you'll want to make the vibration of the speakers to be larger so the magnitude of the sound wave is increased. In order to make that speaker move more, you'll have to increase the value of each sample: original_samples = [0, 0.5, 0, -0.5, 0] def amplify(samples): foreach s in samples: s = s * 2 amplified_samples = amplify(original_samples) // result: amplified_samples == [0, 1, 0, -1, 0] The resulting samples are now amplified by 2, and on playback, it should sound much louder than it did before. Silence When there are no vibrations, there is no sound. Silence can be achieved by dropping each sample to 0, or to any specific value, but does not have any change in amplitude between samples: original_samples = [0, 0.5, 0, -0.5, 0] def silence(samples): foreach s in samples: s = 0 silent_samples = silence(original_samples) // result: silent_samples == [0, 0, 0, 0, 0] Playing back the above should result in no sound, as the membrane on the speaker is not moving at all, due to a lack of change in amplitude in the samples. Speed Up and Down Speeding things up and down can be achieved in two ways: (1) changing the playback sampling rate or (2) changing the samples themselves. Changing the playback sampling rate from 44100 Hz to 22050 Hz will decrease the speed of playback by 2. This will make the sound slower and lower in tone. Going from a 22 KHz source and playing back at 44 KHz, the sound will be very fast and high pitched like birds chirping. Changing the samples themselves (and keeping a constant playback sampling rate) means that samples either (a) get thrown out or (b) are added in. To speed up the playback of the audio, throw out samples: original_samples = [0, 0.1, 0.2, 0.3, 0.4, 0.5] def faster(samples): new_samples = [] for i = 0 to samples.length: if i is even: new_samples.add(samples[i]) return new_samples faster_samples = faster(original_samples) // result: silent_samples == [0, 0.2, 0.4] The result of the above program is that the audio will speed up by a factor of 2, similar to playing back an audio that sampled at 22 KHz at 44 KHz. To slow down the playback of the audio, throw in a few samples: original_samples = [0, 0.1, 0.2, 0.3] def slower(samples): new_samples = [] for i = 0 to samples.length: new_samples.add(samples[i]) new_samples.add(interpolate(s[i], s[i + 1])) return new_samples slower_samples = slower(original_samples) // result: silent_samples == [0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3] Here, extra samples were added, thereby slowing down the playback. Here, we have aninterpolation function that makes a "guess" as to how to fill in that extra space that much be added. Spectrum Analysis and Sounds Modification by FFT Using a technique called Fast Fourier transform (FFT), the sound data in the amplitude-time domain can be mapped to the frequency-time domain to find out the frequency components of audio. This can be used to produce the spectrum analyzers that you might see on your favorite audio player. Not only that, since now you have the frequency components of the audio, if you change the amount of If you want to cut-off certain frequencies, you can use FFT to transform the sound data into the frequency-time domain, and zero-out the frequency components that are not desired. This is calledfiltering. Making an high-pass filter, which allows frequencies above a certain frequency can be performed like this: data = fft(orignal_samples) for i = (data.length / 2) to data.length: data[i] = 0 new_samples = inverse_fft(data) In the above example, all frequencies over the half-way mark is cutoff. So, if the audio could produce 22 KHz as the maximum frequency, any frequency above 11 KHz will be cut out. (For audio played back at 44 KHz, the maximum theoretical frequency that can be produced is 22 KHz. See Nyquist–Shannon sampling theorem.) If you want to do something like increase the low-frequency range (similar to the bass boost effect), take the lower-end of the FFT-transformed data and increase its magnitude: data = fft(orignal_samples) for i = 0 to (data.length / 4): increase(data[i]) new_samples = inverse_fft(data) This example increases the lower quarter of the frequency components of the audio, leading to the low frequencies to become louder. There are quite a few things that can be done to the samples to manipulate the audio. Just go ahead and experiment! It's the most exciting way to learn. Good luc