at that time. Stream used ultrasonic sounds embedded in video files to transmit the IDs of products, which were then pulled from SAP Commerce and displayed on a tablet screen. Take a quick look, it’s really a bit like magic! It was (and is) a great example for a highly relevant Customer Experience technology that connects brands and their products to consumers – in a very innovative way!
Roughly 6 years later, the startup library we used at that time is probably long gone. But there is a great new company out there that offers even more options for ‘data-over-sound’ applications: the UK-based Chirp.
The purpose of this blog post to quickly introduce this exciting technology to you and to reason about potential applications in the Customer Experience space. As always, please share your ideas via comments or simply via Twitter! To make this a bit more fun, I’ve created a bunch of quick demos that should be fun to watch.
Data-over-Sound – the basics
Before we dive deeper, why would one choose data to communicate between devices? There are many options out there create a link between devices – scan via BLE/iBeacons, present QR codes or NFC Tags, via geofencing, data-over-light (like Electric Imp), etc. – so data-over-sound is definitely just one way to create that linkage. These are the characteristics:
- Data-over-Sound can use audio files in the hearable sound frequencies or ultrasonic frequencies, which most humans cannot hear. Using ultrasound, not all device / application combinations are easily supported. For example, the Chirp WebAssembly SDK is only interoperable with the 16kHz-mono frequency, which is in the hearable audio spectrum. For ultrasonic on a mobile smartphone, you’ll have to go the native way.
- It typically takes a few seconds to transmit a few bytes. This means data-over-sound is typically used to share identifiers: IDs to shared spaces (virtual game rooms, physical shopping locations, etc.), IDs to services or products, the ID of a service ticket, the ID to a payment transaction… there are also protocol restrictions which depend on the frequency chosen.
One recent new SDK caught my attention: Chirp for Arduino! There is now an official Chirp SDK for Arduino, which of course now makes it even easier to add data-over-sound features to physical creations.
Demo: sending IDs in all kind of flavors
Sender: Bare Conductive Touch Board
Receiver: Arduino Nano 33 BLE Sense
chirp-audio-write -x 01020304 output.wav
The result being a .wav file, I needed to convert the wav file to an mp3 for the Bare Conductive Touch Board via Audacity and the Lame mp3 encoder.
Note: unfortunately instagram misses some of the action due to the square format of the video – please enable sound and also make sure you view the full post/video.
Sender: Chirp command line on a MacBook Pro
Receiver: Arduino Nano 33 BLE Sense
Happy with the results, I was keen to spin it further. So next I replaced the Touch Board with a pure command line audio-generation. The receiving part stayed the same, the Arduino with built-in mic.
Redeiver: Arduino Nano 33 BLE Sense
While the command line is fine for demos, a more realistic use case is obviously to generate dynamic “chirps” in a web page. For example, an in-store product display would emit the sounds to help customers identify products or link to games/vouchers being advertised. Some interesting learning here: the usage of the Chirp JS SDK will fail, if it is not initialised with a customer action (e.g. a button press linked to the init function).
Sender: Touch Board, Web-Page with Chirp JS SDK, sound files…
Receiver: Chirp WebAssembly SDK in a locally served HTML-page
After so much fun and successes with the various Chirp SDKs, I had to give it a final spin. I ended the testing by integrating their WASM SDK , which allows you to listen to chirps and to decode the chirps right in the browser. Obviously this will ask for the microphone permission, which needs to be granted.
Ultrasonic frequencies – not with your browser
As quickly mentioned above, to play in the ultrasonic frequencies range, you’ll have to go native. Neither the Arduino C SDK nor the WASM SDK for the web will allow you to receive chirps via the ultrasonic (non-hearable) frequencies. For the web, this seems to be due to how the browsers pass the mic input down to the Web Assembly script and it is most likely a security setting which disables audio-tracking.
Chirp/Flutter integration failed on me
Needless to say, I wanted to try the Flutter integration which would have allowed me to try out the ultrasonic frequencies. Sadly, with my installation of the latest iOS/Android toolchains and the latest Flutter 1.9, I hit a hard stop here. The application compiles and starts up fine on my Pixel 3 Android Q, but fails upon receiving or generating a chirp. I ended stuck on this issue here, hopefully they come back to me on this quickly.
Applications and Use Cases for CX
I want to end this blog post by ideating a bit about the potential use cases for Customer Experience. While we’ve shown the applicability already in 2013 with the Stream Prototype, I think there might be many more both on the B2C and B2B area.
In the B2C IoT area, I can well imagine setting up consumer facing devices such as smart speakers or home hubs via sounds. While this is typically done by scanning and joining ad-hoc Wifi networks, the data-over-voice features of Chirp would allow the consumer to stay on the current WiFi network and not switch back and forth between both. Similar solutions are of course already applied via BLE or NFC.
With the widely played game Roblox using data-over-sound by Chirp, it seems all fun & engaging end-user interactions that depend on some kind of shared data such as a channel id (for chatting, for receiving vouchers, etc) could be a good fit, too. In the end, the chirps are pretty nice to listen to :-)
One thing to keep in mind is that for receiving data on a web or native application, the microphone permission needs to be granted. Due to this, the perfect sweet spot for this technology might even be in the B2B or professional area, where let’s say handymen have to check-in to a work setting. I’d also argue that there are environments, which have a totally polluted 2.4 GHz frequency spectrum (Wifi, BLE) where sound is an interesting alternative for data transmission. Keep in mind that for my testing I’ve used the hearable chirps, but especially with native Apps and custom hardware ultrasonic frequencies should not be an issue.
If you have more great ideas, please really let me know via Twitter or just leave a comment! I hope you enjoyed this trip into the data-over-sound space.