Very cool app. If you can crack multi-channel output formats (5.1, Atmos) I can see a lot of prosumers who would happily buy the product. Even the most basic tools for Dolby 5.1 are overpriced IMHO and Atmos encoder prices are either far beyond the reach of most DAW users or require use of Pro Tools.
One downside of selling into the pro audio market is piracy unfortunately. I learnt that the hard way and ended up having to use iLok.
I assume this can do sound rendering, like simulating a conversation on a subway platform while a subway passes by?
Or singing while walking through a tunnel?
Since it has capabilities that would be hard to replicate, rather than show the tool on the landing page, I would show the output. Remove the clutter and force people to listen to what the tool can produce.
The tool as it is now, is being marketed towards yourself, people that wanted to build that tool and know what it is. But everyone will know what it can do after listening to sample output.
Yes it can do all of those things.
You are right about the audio demos, I'll update the frontpage with this soon. Thanks!
It's funny to see this now because I've been for a couple weeks looking into audio spacialization. After a lot of research and even trying to write my own spatializer plugin, I found that Game Engines have probably the most complete toolset to do this task. (Specifically I'm using Godot with https://valvesoftware.github.io/steam-audio/).
Steam audio is pretty awesome in that regards because it supports HRTF and all the physical based goodies like occlusion/reflection and sound propagation. So you can get really really immersive spatial audio.
The only downside with this solution is that you can't do offline rendering. So my question is:
can Audiocube do offline rendering? seems like it would be one killer feature for my use case.
It sounds like you’re in a similar field to me so it would be great to connect! Totally agree, yeah Audiocube is built in unity as game engines made the most sense for what I wanted to achieve. Steam audio is super cool and was a lot of use to look at for writing my own stuff.
Offline rendering is at the top of my research list currently. It’s not in the app yet. It’s going to take a lot of work to build given that the physics engine calculations are tied to frame rate etc, so there is a lot of work to do - but I’m sure it’s possible.
Thanks!
This is great (in theory). During lockdown I got an ambisonic mic (Rode NF-SFW1) and used it to create Dolby Atmos experiences. The workflow - including sending it to Dolby's tool every time - was such a pain. Adding additional 3d elements was especially annoying and limiting.
Unfortunately that's no longer my hobby so can't test this for you but definitely scratches an itch for past me. Nice
That mic looks fun to play with. On their site there’s a link to a bunch of recordings that can be downloaded:
Thanks!
Yeah, frustration with the existing solutions was one of the main reasons I started this project.
I was working in creating immersive audio, and I just found that none of the plugins allowed the level of placement that I wanted. It was all fiddly, and you couldn't easily move the listening position around, while also moving sound sources.
I came from an audio engineering background, not programming, so it took me a while to even learn how to make software! But now I made the tool I always needed.
Very cool project! And well done on doing it without being a software eng!
How long have you been working on it out of interest?
Thanks! Yeah it was a steep learning curve at the start but I’m very confident now.
I started it in 2020 - here’s a blog post about the history https://www.audiocube.app/blog/audiocube-history
I’ve recently got it working on an iPad which was a frustrating grind but I think the concept is a lot of fun on touchscreen!
Lots of cool software was created by domain experts who taught themselves how to program. And quality is more than often is excellent.
This looks great! How small of an audio buffer have you been able to get down to? Any plans for an API?
I've been developing a VR spatial sound and music app for a few years with the Unity game engine, bypassing the game engine's audio and instead remote controlling Ambisonic VSTs in REAPER. I can achieve low latency with that approach but it's a bit limited because all the tracks and routing need to be setup beforehand. There's probably a way to script it on REAPER but that sounds like an uphill battle. It would be a lot more natural to interface with an audio backend that is organized in terms of audio objects in space.
What I'd like is more flexibility to create and destroy objects on the fly. The VSTs I'm working with don't have any sort of occlusion either. That would be really nice to play with. Meta has released a baked audio raytracing solution for Quest, and that's fun for some situations but the latency is a bit too much for a satisfying virtual instrument.
Here's my project for context: https://musicality.computer/vr
Hey, I’ve got down to under 20ms audio buffer, but it depends on the complexity of the scene.
What you’re working on sounds really cool, I’ll have a look at it!
It sounds like Audiocube offers the kind of features that you need, although it doesn’t have realtime audio input (yet, I’m working on it and have it partially working).
Hmm 20ms is quite a lot. I'm targeting <3ms, ideally.
Looking at your project it looks like it would be great to integrate the two somehow - Audiocube would be awesome in vr but I have no vr dev experience
There's a library Valve made for spatial audio for games (inc. VR). I've played around with it a bit, it's incredible. I'm surprised more games haven't adopted it.
Oh yeah, I forgot about this. I think it was a bit less developed last I looked. Not sure if there was another reason I wasn't considering it. Thanks!
edit: ah, it's because it is a plugin for Unity's audio system which I've already determined adds more latency than I want.
Thanks! I joined your Discord... And looks like I "liked" your Audiocube post on FB about 4 years ago :-)
Will there be a Linux version anytime soon!?
I don't have a Linux machine or much experience with it, but I can easily build to it. If you want to help test a Linux build email noah@audiocube.app
Great! Look for a private message.
YouTuber Benn Jordan would probably get a kick out of this. He's a major audiophile and did a series of ambisonic ambience.
Thanks, I think so too by the looks of his channel, I'll get in touch!
Really cool! I've been working on a side project that utilizes spatial audio and I've been pleasantly surprised by the quality I'm getting just using the WebAudioAPI HRTF spatialization. I'm sure this is leagues ahead but it was really nice to find that I didn't really need to do anything to get decent spatial audio other than set the panner node to HRTF mode.
Thanks! Yeah this is definitely a lot more complex and deep than the WebAudio API but that’s still a great tool for some situations.
You using Blender's viewport? (Great idea if so, nearly perfect recreation if no!)
Aha, yeah blender was a huge UI inspiration, it’s all handmade from scratch though!
Awesome work, you nailed it!
IIUC, I'd be able to take individual closely miked recording on multiple different instruments and mix them into a soundspace, such that when I listen on stereo headphones, I'd be able to "locate" the sounds on a virtual stage?
(asking because I listen to a lot of live jam music in stereo and noticed that they use a stereo mix with a virtual image)
Yes exactly - this was something I wanted to achieve.
For example, you could have an entire orchestra with each instrument close miked - and then arranged in audiocube in a vritual concert hall.
The user could even move around the space for full placement control - or you could simply render it to a binauralized stereo file for more immersive listening on headphones.
I'd like to develop a mobile "listening" app, which isn't for making projects, but listening to them and placing you in the virtual environment with control
That's how normal panning works, yes (or using a stereo pair of ambient mics, same thing but baked into the recording). But that's left-right only, I assume that these Atmos tools can do it in 3D space.
Does that work if you only have a pair of stereo headphones?
I've had quad speakers before (with a "delay" unit for the rear speakers) but I think it would be hard to do 3D in stereo (I know about https://en.wikipedia.org/wiki/3D_audio_effect and HRTFs but I have never noticed any 3d-ness to the sound I'm listening to in games).
Yes, on standard stereo headphones, a binauralized sound can give the impression of sound coming from all around your head. It mimics the filtering acoustic effects applied by your head and ears
Does it work with vst plugins?
Currently no, but I'd like to impliment it when I have more resources. See my explanation in reply earlier.
Neat interface the grouped ones with matching neighbor contours
thanks!
The free download requires account registration, which will discourage most people from trying it. Even third-party login might make it easier, but I didn't find that option.
Nice work! Can you export for multichannel playback or is it binaural / stereo?
I'm working on multichannel playback, I've had some experimental success, and plan on adding it into the next update. it just needs more testing on a range of audio interfaces etc.
Currently it only exports/plays back in binaural stereo
Ah. I think I misunderstood the purpose of the app. When viewing the main page, I had the impression that the output would be targeting large Atmos multi-speaker installations. Or that it would export to a format that can play spatially on AirPods.
So right now, is it possible to make something in Audiocube and export it to native spatial AirPod sound ?
I am working on multichannel output but it’s a bit of a headache with all the formats and interfaces etc - I want a simple solution, so that needs a bit of research.
Currently I believe apple spatial uses some kind of 5.1 or 7.1 format audio files mixed with a binauralizer to distribute on headphones. I’m building a method to export in surround formats but it needs a bit more research. Currently it exports as a binaural wav that sounds very 3d, but I’ll look more into apple Spatial Audio as I haven’t researched it much
Outputting 5.1 as a DTS-CD (or something I can convert to a DTS-CD) would give me a reason to buy it.
I'm researching the possibility of other export formats that can be encoded with more spatial information - it's highly requested
Very interesting! Did you write your own acoustic simulation engine for this?
Yes - while the app itself is built in unity, it doesn't use Unity's native audio simulation. I implemented a custom spatializer and acoustic engine that can simulate reflections, occlusion, and other stuff that wasn't native in unity
> As a standalone app, Audiocube offers tools, workflows, and processes that go beyond the capabilities of VST plugins.
But does it support VST/AU in order to load instruments rather than "samples"?
There is currently no VST/plugin support, although this is something I would like to include later on.
It's just too much development work at this stage - not only the VST integration itself, but building a full MIDI sequencer timeline, editor, etc to support instruments.
Other DAWs do this perfectly well so I'm focusing on the 3D virtual environment moreso than tradtional DAW functions. Audiocube isn't a tool to compete with the likes of Ableton and Logic, but something to run on the side.
Although, if I had a big enough team I would love to give it all of these features. I'm just doing it solo at the moment, so I need to have a narrower focus.
"Why You Should Not Write a DAW" (ADC 2024)
Yeah this is an interesting talk. A full DAW is maybe over ambitious for one developer, but I don’t think vst plugins would be able to achieve what I wanted. For example vsts can easily spatialise individual sounds, but there is no way to then easily move the listening position through the scene, without a disgusting amount of automation - hence building a virtual 3d environment daw with more placement flexibility
... or start with an open source DAW that already has all the rest, and add the placement flexibility you want ...
But I'm speaking out of self-interest here, and anyway for now your output is limited to binaural stereo which is not really a viable target for production workflows at this point.
That's the thing: if you have VST or ReWire support, you don't need to implement any of that. That's now something someone can use a VST/ReWire bridge for, taking all that work off your plate. "If you need MIDI sequencing, or multi-pattern input, or randomized pattern generation, just connect your DAW of choice as input".
(And of course, "and if you then want to consume the result in another DAW, you can". Chaining up multiple DAWs because they're all good at different things is one of the more fun things to do ;)
This seems intriguing but I'm genuinely confused.
It seems like "bakes in" spatial audio to binaural stereo?
But who is the market for that?
I love spatial audio on my AirPods but a big feature is that it moves with my head and can even be customized for my ears.
And I certainly don't want it applied when downsizing to a mono Bluetooth speaker.
It seems like you'd need to export your final product to surround/Atmos for the way people want to, and currently do, consume spatial audio? I assume the target here is Apple Music, short films, etc.?
I mean I think the concept of the 3D DAW is great. I just want to make sure there's a product-market fit here, so you can succeed. Or is there a market I'm overlooking?
Thanks for the intrigue, and yes these are all valid questions and things I'm trying to work out.
I think I need to narrow down the market a bit. Currently I'm aiming more at audio and sound design experimenters rather than the deep technical stuff - although that is a goal later.
I'd like to create a 'listener app' so people can have full immersion, control, and exploration of audio scenes.
The binaural process is more of a high powered, more flexible and controllable version of a binarual plugin, so this would be for making binaural mixes rather than slapping on everything.
I personally think the binauralizarion is one of the best sounding out there, it sounds great on headphones and has excellent sound localisation.
But you are completely right. I'm still working out product market fit, and I think it's why I'm finding a bit of friction reaching the right audience.
Thanks for your thoughts! Any feedback and input is very appreciated
In case it helps, I think producing binaural recordings, or using a custom listener app, would be appealing to "experimenters" if it were open source, but it doesn't strike me as something with appeal to consumers or professionals that would pay for it.
Since you're going for a paid approach, it does seem like you need to locate the audio professionals mixing for spatial audio on Apple Music and TV shows, to see if you can show them why your 3D editor results in a better surround/Atmos final result.
I will say that I've often wondered what it would be like to "walk through" a band or orchestra as they were playing, with the binaural audio adjusting accordingly, which would require a full 3D approach like you've created, not just angles. I once attended an art project of a choir singing in 360° in a circle around you, where you were invited to move around to change the "mix" of voices, and it was magical. I think that's what you must mean by "exploration of audio scenes" and it sounds really cool. On the other hand, it's hard for me to imagine how you'd turn that into a business. Especially since I assume that's already relatively easy to do with video game/VR engines, even if perhaps the binaural effect isn't quite as good?
Good luck! I think the concept is very cool, and I hope there's a way to monetize it as an independent product.
Just guessing, but even setting aside VR and whatever kind of stuff they are doing for theaters like imax, there does seem to be an interest in new kinds of immersive 3d audio/video. The Las Vegas sphere aimed to do this iirc, and the money behind it aimed to develop the IP and open similar venues in different places around the world. Supposedly media that’s under development actually uses a smaller copy of the target venue, so if you can do that with a software suite instead it’s probably a good thing.
I've even heard of them using this kind of tech in theatre performances, so there is definitely a bunch of use cases!
Not spatial audio, but reminded me of audioGL (2012, but a newer video posted in 2024): https://m.youtube.com/@AudioGL/videos
There was also a BeOS application with a 3D mixing interface: https://en.wikipedia.org/wiki/3dmiX
I believe the HRTF binauralization of object-based audio does qualify as spatial audio, but there are many different definitions floating around. AudioGL is cool though!
Is there an audio format that stores the 3D origins of each sound, so that you could theoretically play it through Airpods (or some other spatially-aware headphones) and hear the sound change as you tilt your head?
Dolby Atmos is literally that, I think there are several physical formats that can support it.
Ambisonics, especially higher-order ambisonics, also is capable of spatial panning, though it encodes the 3D soundfield rather than individual sound sources. Again, there are a few physical formats.
This is probably a very odd and roundabout way of doing something akin to what you are asking for, but my recently released VR game DodgeALL[0] has a form of modding support for (wav/mono) audio files which can be uploaded (via SideQuest) to the headset. These are then assigned to "obstacles" which are launched at the player i.e. you could move your head around and hear spatialized audio files as they fly around the arena. You don't even need special headphones, the ones on the Quest do a good enough job, and Meta's Spatial audio SDK does a wonderful job of spatializing audio. I was originally using SteamAudio[1] but switched mainly due to moving from PC-VR to Standalone.
Sorry if this sounds like some self-promotion - I specifically built this functionality[2] for my previous audio engineer so that he could quickly design and test sounds without the hassle of having to build/install an APK with every new audio file.
[0]: https://www.DodgeALL.com (only for Quest 2/3/3s/Pro headsets)
[1]: https://valvesoftware.github.io/steam-audio/
[2]: https://docs.google.com/document/d/1IFQjkroomHh8FocjfA-a55JP...
Another way to do this would be to make an app that uses positional information from the headphones and feeds the sound of your choice into a head related transfer function (HRTF) to give the illusion of spatial audio.
In addition to Dolby Atmos there is MPEG-H 3D Audio.
You could also just emulate what the music would sound like with a stereo system and support head tracking with that. dear reality does something like that with their dearVR digital audio workstation plugins.
And then I recall there was this one old Kickstarter project that made headphones with that idea. It had an IR transmitter for head tracking.
I think it might be quite difficult to track head movement with just earbuds and not have the tracking drift, though.
I sort of expected this sort of thing to be a nice use for VR, since it already needs to do the tracking. But VR generally didn’t seem to catch on, so…
I have a home theater which is 5.1 which is great for movies [1] and some music [2]. There are 'virtual surround' systems based on [3] of which Airpods are the most famous. Most of those don't work for me except in the sense that I learn 'sounds like I have a cold' means 'the sound is supposed to be above me'. The exception is my Hololens where if I use the web browser to play music and turn around it really sounds like the music is coming from behind me. I have yet to be impressed by spatial audio on my Meta Quest 3.
[1] Oddly the surround sound soundtracks that impress me the most are old VHS tapes encoded with Dolby Pro Logic
[2] let's see... Close to the Edge by Yes, Supernature by Goldfrapp, Maximum-Minimum by Kraftwerk
[3] https://en.wikipedia.org/wiki/Head-related_transfer_function
I'm working on a mobile app that lets you playback projects created in Audiocube and gives you full control over where your listening position.
It's definitely possible to have head tracking, I think it will just depend on what headphones and hardware offer it. It would be cool to give you that kind of VR listening experience but just on headphones
For those of us unfamiliar with the term DAW, I assume it’s Digital Audio Workstation.
At first glance I thought of: DEW
Digital Audio Workstation: https://en.wikipedia.org/wiki/Digital_audio_workstation
It’s a digital audio workstation.