Very interesting idea. I remember reading that in visual spoken communications, only 20% is the actual words. The rest is tone of voice, body language, context, emphasis, expressions, ... all that stuff.
I don't know if 20% is correct, but I feel it's very close to it. I also think a lot of internet arguments happen as a direct result of miscommunication. Emojis are great, but they get abused to the point that HN filters them out. Perhaps allow readers to toggle if they want to see emojis or not?
The book Understanding Comics by Scott McCloud is a tremendous study in this area, Scott shows how you can add abstract meanings to words and pictures through illustration.
Something like this would be great for karaoke! Especially for the long held notes https://x.com/TheOisinMoran/status/1614435041764859907
Comic books already use changes in font, weight, size, of text and the shape of the word balloon to indicate tone and expression.
Emojis absolutely have their place here. They can add tone, nuance, and a bit of humanity where plain text can feel flat.
I've always wondered about this.
In Akan languages it is not difficult to conceive of how the same word can be written in different ways to convey another dimension.
Anyone who speaks an akan language will understand that each of these words below means good but with a slightly different emphasis.
papa papaaapa papapapapapa
What is the linguistic term for this concept?
Consider learning Polish. Kurwa sounds exactly as it looks.
Reminds me of how the captions were done in Tony Scott's Man on Fire (2004). It's a pretty great movie too.
Many moons ago I became quite obsessed with analyzing spectrograms on my computer.
I would load up audio files in Audacity and look at them to see how the audio "looked", as a function of how intense each frequency is over time.
You can even set a track to spectrogram while recording which allowed you to see the sound in real time.
Music also tends to be very beautiful in the spectrogram! And birdsong also. Sometimes I would see a bird first, and only afterwards notice it in my field of hearing.
I noticed while analyzing a podcast that I began to recognize common words like "you." I also noticed that I was able to easily distinguish between different people's voices.
I had to wonder if I were deaf, or if I become deaf, I would suddenly have a strong motivation to learn how to read these things. To develop some kind of device which would show them to me 24 hours a day.
I have not done this, but the project has remained in the back of my mind for over a decade.
Does anyone else know more about this? Does such a device exist?
I think that only some linguists learn how to read spectrograms. But it seems like something that might be extremely useful to any hearing impaired person?
Relating to the article, I think one could quickly learn to read them fluently (e.g. as subtitles, perhaps overlaid on real life), and of course you get the tonal information built in for free—that's what a spectrogram is!