Manipulating Audio And Video Streams For Streaming on Linux

What’s great about having the ability to do things like echo cancellation and creating fake backgrounds natively on your linux system is that it no longer matters if, oh, Zoom hasn’t gotten around to supporting it yet, or you need a fake background on an application that doesn’t have that functionality.

And yes, if you want a full GUI setup and more options (which comes with more overhead – something I have to pay attention to with my older hardware), OpenBroadcaster can do all of this as well. But if you want to be able to do it from the terminal – or at least have the option to, these instructions, along with the programs and scripts I discuss, will get you going.

Audio

Using the built-in capabilities of PulseAudio and the brilliant pulse-autoconf program (and a little fzf and sed elbow grease) we can easily do two cool things: We can perform echo-cancellation and/or mix in a second audio stream.

It’s very easy to install. Simply download the latest version from the releases page and run the script. If you want, you can move the .desktop file to $HOME/.local/share/applications to provide a launcher (or put it in $HOME/.config/autostart), or use the example systemd file if that’s your jam.

Mixing in another sound source is one of those things that’s pretty cool for a particular group of people. Whether it’s background music you’re listening to while talking with friends, or a soundtrack or sound effects for your role-playing game, or some other use I’ve not thought of yet, being able to play a sound in one program and have it feed directly into your Skype call or Roll20 game is pretty cool. It works pretty much flawlessly with pulse-autoconf.

The echo cancellation works rather well. I have external speakers placed a little further away which can cause a really annoying echo of the people talking to me. With the echo-cancellation profile going, it completely disappears. You can even have music playing and the people on the other end of the video chat won’t hear a thing.

I ran into two things to be aware of. When I have my whole house audio running and both my laptop and another computer in the same room playing the same thing, there’s enough of an audio delay that the echo cancellation has to work really hard, and even then is not as effective.

The second issue to be aware of is that your voice will probably sound tinny if you’re playing music loudly; it’s just because the echo cancellation can only do so much. Just don’t listen to your music as loudly and you should be fine.

I recorded a brief (minute and a half) bit of audio switching the sinks around to let you hear what I mean:

The third, and final, issue I have is with the way the configuration gets changed. Yes, you can run pulse-autoconf EditConfig and then edit the file, but if you’re just wanting to switch between configurations (say, because you finished a Discord chat and now want to listen to your whole-house audio again, cough cough) it’s a bit of a pain. So I created a small script that you can use – either with fzf or with command-line switches (or use those with an OpenBox menu – sample included in the file) to switch back and forth. You can find that script, pulse-fzf-autoconf on GitHub, GitLab, or my personal repository.

Video

It’s also possible – using v4l2 – to both create a virtual background or substitute one video source for your webcam. (There is another module that can do it, v4l2 is the less resource-demanding.)

You’ll need the v4l2loopback-dkms, ffmpeg, and uvcdynctrl packages. Debian/Ubuntu users can simply sudo apt install v4l2loopback-dkms ffmpeg uvcdynctrl. I’m not certain if a reboot is necessary afterward, but I did one to be sure.

That also takes care of all the prerequisites (except for special cases) for Linux Fake Background Webcam except for Python3. So you can just grab the repository, plop it in the directory of your choice and install it by running install.sh (which just runs pip3 install --user -r requirements.txt).

And that is where it can get complicated. So once again, I wrote a fzf based script – video-fzf-config (GitHub, GitLab, personal repository) – to automate a lot of the processes.

NOTE: You will need to edit the line LFBW="/home/steven/sourcecode/Linux-Fake-Background-Webcam/fake.py" to reflect whatever path you put Linux Fake Background Webcam in and the line BACKGROUND_PATH="/home/steven/documents/fake_backgrounds/" to reflect whatever path you have your fake backgrounds (and/or video streams to fake in) are stored in.

You run video-fzf-config with either --fake for a fake background or --replace to replace the video stream. If you put the full path and filename of the file to either use as a fake background or replacement video stream (respectively) as the second argument, it will just do the thing. If not, it will present a fzf powered selection from the directory you specified in BACKGROUND_PATH.

The script will automatically detect the real camera/webcam you have, and if there’s more than one, let you select which one it will work with. It will also determine which video output is the fake webcam loaded by v4l2, and then launch ffmpeg or LFBW with some settings that will let you "just use the thing", which was the point of writing this script. Once you’re done and exit either ffmpeg or LFBW, the script will also unload the l4v2 module once you’re done.

If you’d like more advanced options (say, having a fake foreground that goes with the background), then take a look at the usage section of LFBW’s README for the fake background image, or at Linux For Devices’ writeup about using ffmpeg for fake webcam streams.

You can get my video-fzf-config script to automate this stuff at GitHub, GitLab, and my personal repository

Featured Photo by Stanley Li on Unsplash