Let’s build Metaverse with AI: What we have?

In the previous post about building metaverse with AI (link), I discussed the generic points of view, what we need and all the stuff like that. In this post, I am going to discuss about AI models we have which can be helpful in order to build the metaverse using AI and also possible pipelines.

Also, remember that in this particular post, I only will be discussing the AI models which I think can be helpful in building a virtual universe. So if your favorite AI isn’t in the list, accept my apologizes.

AI models for Metaverse

First of all, I think for building a virtual universe or metaverse using AI, we need these models:

Image generation models: These models will help us build everything imaginable. These are essential in pretty much every AI art project and of course, very useful in order to make the concept of our supposed Metaverse.
Music/SFX generation models: Imagine walking in a jungle. The landscape is pictured in your mind right? Now go a little deeper. You hear the sounds in your head, too. This is what we call soundscape in Ambient or minimalistic music (as I wrote about it before). Now let’s consider a metaverse we’re building, right? This newly made universe needs sounds. Without sounds, metaverse doesn’t mean anything. We need AI models in order to generate music, sounds and soundscapes for us.
Vision Language Models: These are important as well. In building the metaverse, we need everything to be as automated as possible. Basically, we need the matrix but in a good way. So a vision model can easily analyze a scene and generate respective prompts for sound generators.
3D Generation models: And the question is why not? We try to make a complete 3D universe and we need to make 3D objects which let people make their desired universe, right? With AI, this will be a reality.

Now, let’s dive a little more in depth and look at what models we have access to!

Image Generators

If you ask me, this is the easiest type of model to find for this particular project. We have tons of proprietary options such as Dall-E 3 or Midjourney or even FLUX Pro. Which are all considered the best in the business.

In the open source side, we’ve got Mann-E, Stable Diffusion and other useful models as well, right? This means with a small search on the web, we can find out the best way of visualizing our dreams of a made-up universe.

Also, due to my research about different models and hosting services, hosting models on replicate or modal is very easy. For other types of hosting we may explore possibilities on CivitAI or Runware as well.

Music and Sound Effects generators

This is also not a rare thing. Although I am not really familiar with the music generation space and I only know Stable Audio LM and Meta’s Music Gen in open space, and Suno AI in proprietary space, I guess we already have the best in the business.

Vision Models

Well, I personally use Open Router to find out about the possibilities of these models, and being honest, the best model I could find for vision task was nothing but GPT-4o.

Although there are good vision models out there, but most of them are very generic or very specific and GPT-4o is right at the middle. We can use this model in order to describe different scenes in our metaverse. Also, we may utilize this model in order to be a guide through the metaverse or just help us build 3D objects or soundscapes.

3D Generation Models

Well these models are currently the rarest models in the list. We may need two approaches for this specific task:

Text to 3D: very similar to text to image, you just describe your scene or object, and get the 3D object. Although it may be a little buggy, but it will be a fun experiment to implement a model or pipeline for text to 3D. It will help the residents of our metaverse to generate assets of their choice as easy as typing what they have in their minds.
Image to 3D: This is also a possibility. Currently, I use TripoSR a lot for making different 3D objects, but I still couldn’t find the best input images or the best settings or hyper-parameter tuning for getting the best results.

With 3D generators, our workflow will become much much easier than what you may think. So we need another step, right?

What’s next?

Well, in the previous post we discussed the whole idea of metaverse and what we need to build one. In this one, we just discovered the AI tools we may be able to utilize. The next will be a study on how we can make a metaverse AI model at all.

It will be the most challenging part of the project, but in my honest and unfiltered opinion, it is also the best part!

April 16, 2018

How to make video games like movies!

It was a long time that I did not write any thing in this blog. Now, I decided to write a topic about “video games” (as I wrote in my Persian blog). I was member of a game development team for about three months and I learned a lot. At least, I know the way they were doing the job was “How to not make a video game”. So, When I left the team, I decided to research about game development process. In this topic, I explain everything I found (experience and research result!)

When I was in the team…

It was in October (2017), a person sent me a message in Telegram, and the message was like this :

Mr. Haghiri, we need a musical composer for our game, we did a search and we found you. Please come here Wednesday 4:00 PM to talk about the project and your role.

Wednesday, I went to their office. That guy greeted me so nicely and started talking to me, about their project. I found the game is a horror game (horror games are popular in Iran, but there’s no “good” horror game “made in Iran”.) and it made me happy! Because it was the first time I heard about an indie team decided to make such a great game. They let me two weeks to research about “Sound and Music” in Unity Engine and I did it. I composed two pieces and also I tried to learn some tools for mixing and mastering sounds in the game (But it wasn’t actually my role, I just did that as a sample!).

After two and half months, they said “Mr. Haghiri, you don’t do what we wanted you to do”, anyway, they haven’t paid me even a “rial” in those days and expected me do great music composition, and they also wanted me to do what wasn’t my actual role. And this was not a good experience actually. But in those two months, I learned Unity game engine and I also met other “game developers”. And, I decided to publish my experience on my blog.

Why movies?

Recently, I read a book called Making Short Films : Complete Guide from Script to Screen, by Clifford Thurlow. In the book, I found great names like “Salvador Dali” or “Charlie Chaplin”, and also great movies and books also mentioned in the book, like “One Flew Over Cuckoo’s Nest”. Everything was perfect, In I think about process of making a video game! It’s so similar to process of making a movie!

But I realize game and movie have lots of differences. The biggest difference is that games are interactive and players interact with the environment or other characters, but movies are not. Anyway, the main procedure – I mean “writing” – is %90 the same! So, I decided to mention some movies I watched, then tell you how I make a video game like them!

Great movies gave me ideas!

An Andalusian Dog

Movie is written by Louis Bunuel and Salvador Dali. I think these names are enough. But, after I watched the movie (it’s now available on YouTube and other video-sharing platforms, you can easily find and watch it), I discovered “product of a melancholic and depressed mind”. And both “Melancholia” and “Depression” are good subjects for a story or game.

Phantom of The Opera (Musical)

The book is about a musician, in a better word, a “genius” and smart person who hided himself. Andrew Lloyd Webber, just made that sad and creepy story to a romantic story by his music! First time I watched the movie, I could not understand it well. Because it’s not following the book’s story, and I’m not a native English speaker! I watched that movie 4 times. I watched the live performance 2 or 3 times and finally, I got the concept.

It has “Misanthropy” and “Romance” at once. I think these two things are also good for people who want to make video games (Please go and check “INSIDE” by Playdead! It’s the Misanthropy! Pure Misanthropy!)

School of Rock

It’s a bit different, School of Rock is comedy, and it’s also attractive for children. Because the topic is about a guy who teaches a bunch of 4th grade children to ROCK! Yes, he teaches them how to play electric guitars, bass, drums and keys. I think the concept of “Music” can be a good idea, too! Specially if you plan for a game like “Guitar Hero”.

Only Lovers Left Alive

If you like Vampires, please watch this movie. This movie has no “teenager” content, but it still tells stories about two vampires who married for centuries. Movie is directed by Jim Jarmusch and he also made music for his movie, in his rock trio SQRL. This movie has two things for game developers. It’s and independent movie and it can be a good idea bank for indie developers and also, it has “Romance” and “Fear” and “History”. All of these three factors can make a game great!

Ok, I talked a lot about movies! Let’s find “how can I make a game like a movie?”

Finally, let’s make a game like a movie!

The most basic thing you must have for a movie is “plot”. Plot is the main idea, it’s developed and tells people your concept, but it’s not a completed “script”. Writing plot in both games and movies is the same. To write a good plot, you need to study and read books, scripts and other plots. I prefer printed (and Persian) books in this case.

You will need research on the topic you want to write a plot about. For example, if you want to write about a middle eastern civilization (for example : Sassanid Empire or Ottoman Empire), you have to read history of Iran, Turkey, Afghanistan, Iraq, etc. If you want to write a plot about Satan, you have to search about Satan’s role in Judaism, Christianity and Islam, and find which belief is closer to what you want. So, you need to have a background.

After you wrote the plot, you have to write the script. But, writing script is different here, you have to clarify where and when the scene is interactive and when and where is not. The best example, is “Bioshock Infinite”. You know why? Because when you’re not interacting with the environment, you still can move camera and see what happens around you. I’m not a good script writer (But I try to be!) and I will write a post about how to write script from plot, when I manage to do that.

After you wrote the script, you have to make your basic ideas in the engine. Please! Please! Please! Call an experienced game developer before doing that, because the experienced one can help you find experienced character/concept and environment artist and game-play designer. After that, I can say you’re ready to start making your game! With a good team, you can make a good game!

Finally, I wrote this article but it wouldn’t be the last article about games in my English blog. I try to continue this, because I even couldn’t find good articles about being “game script writer” or “game director” even in English! I hope you like my post 🙂

June 22, 2017

Composing using relative scales!

In this topic, I’m going to show you how to compose a minimal music piece using “relative” scales. But first, let’s talk about music theory. A relative scale, is simply “a scale with the same note as main scale”, there are two kinds of relatives, relative minor and relative major. So, if we consider this as D minor :

D – E – F – G – A – A# – C – D

We need another scale with the same notes. Let’s start from 3rd note of our main scale :

F – G – A – A# – C – D – E – F

This is “F Major” scale, and it’s “relative major” of our D minor scale. As you can see, if you start from 3rd note of a minor scale, you will have its relative major, and if you start from 6th note of a major scale, you’ll have its “relative minor”.

Now, let’s make some piece! In this topic, I use Ableton Live 9 as my DAW, and DSK Overture and ZynAddSubFX as my plugins. Let’s make music!

First, I play an orchestral piece in D minor using DSK overture, with “Strings section”, “Flute”, “Violin” and “Cello”. It will be D-F-A-F-D-A-E.

Sounds good, but It needs some decorations! We can do that using F major! in F major section, I want to use ZynAddSubFX. And my notes will be F – A- C – A – F – C – G. So, this is our relative major :

Now, we can mix them! This is the orchestral piece with its electronic relative major :

And, I repeat this piece for 4 or 5 times, then make a song (with some audio engineering ), here is our final song :

I hope you enjoy this topic and song! Remember that this “relative” technique is useful when you want to use Ostinatos, specially on piano, you can play the ostinato on right hand and its relative on left hand.

April 8, 2017June 17, 2017

A very short introduction to Ambient Music

In my Persian blog, I had written a lot about operating systems, computer architecture and digital electronics. I have plans for English blog and I’ll write about computer science and engineering in future, but, I decided to explain my experiences in music for now. In this article, I’m going to talk about ambient music, and how it’s produced. Of course, this is not a music theory or musical software tutorial.

Let’s talk about ambient music, what is ambient music? Ambient music is a minimalistic, modern and electronic genre, which is invented by “Brian Eno” in early 70’s. Ambient is actually a subgenre of electronic music, but after years of evolution, it’s known as an independent musical genre.

Characteristics

Ambient music, is highly dependent on the environment. You will realize this when you hear the name. Actually, this genre is based on John Cage’s theory, Everything we do is music and his 4’33” is one of the best ambient songs ever! Four minutes and thirty three seconds of silence, the composer asks you to listen to ambient noises. It means, you can record any sounds and then make it ambient music. This is true, but, not every sounds. A lot of ambient tracks are just recordings from nature, and a simple melody is played over that sounds. Some others are electronic productions, based on natural sounds and atmospheres. This means, ambient is some kind of avant-garde music, you are free to do everything you want!

Styles

You know every musical genre, can be played in different styles. In this section, I just explain some of ambient music styles. I’m sure there are more styles, but these styles are my favorites :

Dark Ambient :
This is one of the most known styles of the ambient music. Sometimes people think that dark ambient is a subgenre of ambient music, but it’s actually not. Because it’s the same concept, but with scary, depressing or dark atmosphere. Sounds like someone plays his/her music in an abandoned and haunted place 😀
Space Ambient :
This is another style. If you’re a fan of outer space life, science fictions and movies like Star Wars or Star Trek, this is your kind of music. In this style, musicians use effects which can make you feel aliens are in your home! And this is what makes this style awesome!

There are more, but I usually listen to these styles. So, I can explain these two better. For more information, you can find ambient musicians on YouTube, Soundcloud, Jamendo, etc. And ask them about their style!

Subgenres

And now, we are going to take a look on subgenres of ambient music. These genres are created to show us how minimal music can be perfect!

Drone :
My most favorite subgenre of ambient music, drone music is just sustained chord, note or sound. Also, artists may decorated the sustained sounds using small melodies, or a single melody is repeated on the drone sound. I’ll explain drone music in future.
Lowercase :
This is the most artistic form of ambient music. Artists record sounds from nature, or daily activities, and then, amplify them and edit them to make a melody. Lowercase music is one of the most minimalistic genres, and one of the most amazing ones, too!

In this article, we talked about ambient music and which kind of music we can call ambient. In future, I’ll explain more about making an ambient track and I’ll introduce my favorite ambient artists.

Good luck!