{"id":354,"date":"2024-11-22T21:46:00","date_gmt":"2024-11-22T18:16:00","guid":{"rendered":"https:\/\/haghiri75.com\/en\/?p=354"},"modified":"2024-11-22T21:46:00","modified_gmt":"2024-11-22T18:16:00","slug":"lets-build-metaverse-with-ai-what-we-have","status":"publish","type":"post","link":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-what-we-have\/","title":{"rendered":"Let&#8217;s build Metaverse with AI: What we have?"},"content":{"rendered":"<p>In the previous post about building metaverse with AI (<a href=\"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-introduction\/\">link<\/a>), I discussed the generic points of view, what we need and all the stuff like that. In this post, I am going to discuss about AI models we have which can be helpful in order to build the metaverse using AI and also possible pipelines.<\/p>\n<p>Also, remember that in this particular post, I only will be discussing the AI models which I think can be helpful in building a\u00a0<em>virtual universe.<\/em> So if your favorite AI isn&#8217;t in the list, accept my apologizes.<\/p>\n<p><a href=\"https:\/\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full\" src=\"https:\/\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png\" width=\"1456\" height=\"816\" \/><\/a><\/p>\n<h1>AI models for Metaverse<\/h1>\n<p>First of all, I think for building a virtual universe or metaverse using AI, we need these models:<\/p>\n<ul>\n<li><strong>Image generation models:\u00a0<\/strong>These models will help us build everything imaginable. These are essential in pretty much every\u00a0<em>AI art project<\/em> and of course, very useful in order to make the concept of our supposed\u00a0<em>Metaverse.<\/em><\/li>\n<li><strong>Music\/SFX generation models:\u00a0<\/strong>Imagine walking in a jungle. The landscape is pictured in your mind right? Now go a little deeper. You hear the sounds in your head, too. This is what we call\u00a0<em>soundscape\u00a0<\/em>in Ambient or minimalistic music (as I wrote about it <a href=\"https:\/\/haghiri75.com\/en\/a-very-short-introduction-to-ambient-music\/\">before<\/a>). Now let&#8217;s consider a metaverse we&#8217;re building, right? This newly made universe needs sounds. Without sounds, metaverse doesn&#8217;t mean anything. We need AI models in order to generate music, sounds and soundscapes for us.<\/li>\n<li><strong>Vision Language Models:\u00a0<\/strong>These are important as well. In building the metaverse, we need everything to be as automated as possible. Basically, we need\u00a0<em>the matrix\u00a0<\/em>but in a good way. So a vision model can easily analyze a scene and generate respective prompts for sound generators.<\/li>\n<li><strong>3D Generation models:\u00a0<\/strong>And the question is why not? We try to make a complete 3D universe and we need to make 3D objects which let people make their desired universe, right? With AI, this will be a reality.<\/li>\n<\/ul>\n<p>Now, let&#8217;s dive a little more in depth and look at what models we have access to!<\/p>\n<h2>Image Generators<\/h2>\n<p>If you ask me, this is the easiest type of model to find for this particular project. We have tons of proprietary options such as <a href=\"https:\/\/openai.com\">Dall-E 3<\/a> or <a href=\"https:\/\/midjourney.com\">Midjourney<\/a> or even <a href=\"https:\/\/blackforestlabs.ai\/\">FLUX Pro<\/a>. Which are all considered the best in the business.<\/p>\n<p>In the open source side, we&#8217;ve got <a href=\"https:\/\/mann-e.com\">Mann-E<\/a>, Stable Diffusion and other useful models as well, right? This means with a small search on the web, we can find out the best way of visualizing our dreams of a made-up universe.<\/p>\n<p>Also, due to my research about different models and hosting services, hosting models on <a href=\"https:\/\/replicate.com\">replicate<\/a> or <a href=\"https:\/\/modal.com\">modal<\/a> is very easy. For other types of hosting we may explore possibilities on <a href=\"https:\/\/civitai.com\">CivitAI<\/a> or <a href=\"https:\/\/runware.ai\">Runware<\/a> as well.<\/p>\n<h2>Music and Sound Effects generators<\/h2>\n<p>This is also not a rare thing. Although I am not really familiar with the music generation space and I only know Stable Audio LM and Meta&#8217;s Music Gen in open space, and Suno AI in proprietary space, I guess we already have the best in the business.<\/p>\n<h2>Vision Models<\/h2>\n<p>Well, I personally use <a href=\"https:\/\/openrouter.ai\">Open Router<\/a> to find out about the possibilities of these models, and being honest, the best model I could find for\u00a0<em>vision task\u00a0<\/em>was nothing but <a href=\"https:\/\/ai.com\">GPT-4o<\/a>.<\/p>\n<p>Although there are good vision models out there, but most of them are\u00a0<em>very generic\u00a0<\/em>or\u00a0<em>very specific<\/em> and GPT-4o is right at the middle. We can use this model in order to describe different scenes in our metaverse. Also, we may utilize this model in order to be a guide through the metaverse or just help us build 3D objects or soundscapes.<\/p>\n<h2>3D Generation Models<\/h2>\n<p>Well these models are currently the rarest models in the list. We may need two approaches for this specific task:<\/p>\n<ul>\n<li><strong>Text to 3D:\u00a0<\/strong>very similar to text to image, you just describe your scene or object, and get the 3D object. Although it may be a little buggy, but it will be a fun experiment to implement a model or pipeline for text to 3D. It will help the residents of our metaverse to generate assets of their choice as easy as typing what they have in their minds.<\/li>\n<li><strong>Image to 3D:\u00a0<\/strong>This is also a possibility. Currently, I use\u00a0<em>TripoSR\u00a0<\/em>a lot for making different 3D objects, but I still couldn&#8217;t find the best input images or the best settings or hyper-parameter tuning for getting the best results.<\/li>\n<\/ul>\n<p>With 3D generators, our workflow will become much much easier than what you may think. So we need another step, right?<\/p>\n<p><a href=\"https:\/\/mann-e-images.storage.c2.liara.space\/c13a1511-1364-4398-b1a2-c145e95ea2d3_part_0001.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full\" src=\"https:\/\/mann-e-images.storage.c2.liara.space\/c13a1511-1364-4398-b1a2-c145e95ea2d3_part_0001.png\" width=\"1456\" height=\"816\" \/><\/a><\/p>\n<h1>What&#8217;s next?<\/h1>\n<p>Well, in the previous post we discussed the whole idea of metaverse and what we need to build one. In this one, we just discovered the AI tools we may be able to utilize. The next will be a study on how we can make\u00a0<em>a metaverse AI model\u00a0<\/em>at all.<\/p>\n<p>It will be the most challenging part of the project, but in my honest and unfiltered opinion, it is also the best part!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the previous post about building metaverse with AI (link), I discussed the generic points of view, what we need and all the stuff like that. In this post, I am going to discuss about AI models we have which can be helpful in order to build the metaverse using AI and also possible pipelines. &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-what-we-have\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Let&#8217;s build Metaverse with AI: What we have?&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[4],"tags":[36,17,18,23,37,11,22,6,45,24,25],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8BkKn-5I","jetpack-related-posts":[{"id":344,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-introduction\/","url_meta":{"origin":354,"position":0},"title":"Let&#8217;s build Metaverse with AI : Introduction","author":"prp-e","date":"November 21, 2024","format":false,"excerpt":"It was 2021, the whole products under the flag of Facebook, went down for a few hours. I remember that most of my friends just started messaging me on Telegram instead of WhatsApp and also no new post or story was uploaded on Instagram. A few hours passed, everything went\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":362,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-we-need-to-talk-about-3d\/","url_meta":{"origin":354,"position":1},"title":"Let&#8217;s build Metaverse with AI: We need to talk about 3D","author":"prp-e","date":"November 24, 2024","format":false,"excerpt":"In the previous post about building metaverse with AI, we discussed different possibilities and AI models we can access in order to make the virtual world. Although I personally am a big fan of 2D worlds, but let's be honest, a 2D world is basically a perfect choice for a\u00a0low\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":391,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-building-asset-generator\/","url_meta":{"origin":354,"position":2},"title":"Let&#8217;s build Metaverse with AI: Building asset generator","author":"prp-e","date":"November 27, 2024","format":false,"excerpt":"Look at this: How do you think this apple has been made? Excellent question. After the previous post, I said we should put LLMs out of the picture for now. Also we needed to talk about 3D, because it is important in whole metaverse space, right? Today I just did\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/120c3554-8326-4625-a129-50e80bb215da_part_0003.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/120c3554-8326-4625-a129-50e80bb215da_part_0003.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/120c3554-8326-4625-a129-50e80bb215da_part_0003.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/120c3554-8326-4625-a129-50e80bb215da_part_0003.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":377,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-llama-mesh-is-out-of-picture\/","url_meta":{"origin":354,"position":3},"title":"Let&#8217;s build Metaverse with AI : LLaMA Mesh is out of picture","author":"prp-e","date":"November 25, 2024","format":false,"excerpt":"In the previous post I mentioned that I could not get LLaMA Mesh to work, right? So I could and in this particular post, I am going to explain what happened and why LLaMA Mesh is not a good option at all. First, I will explain the workflow of the\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/1175f08e-c7d9-437e-95fa-88fba140c24e_part_0001.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":335,"url":"https:\/\/haghiri75.com\/en\/privacy-focused-ai-is-all-we-need\/","url_meta":{"origin":354,"position":4},"title":"Privacy-focused AI is all we need","author":"prp-e","date":"November 1, 2024","format":false,"excerpt":"I remember in 2020 and 2021, due to Elon Musk's interest in crypto and also\u00a0The Metaverse Hype\u00a0people, specially the ones who had no idea about crypto or blockchain, started investing in the crypto markets. Although it seemed a little bit of a failure, people made profit out of it. It\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":399,"url":"https:\/\/haghiri75.com\/en\/you-only-need-python-to-make-ai-agents\/","url_meta":{"origin":354,"position":5},"title":"You only need Python to make AI agents.","author":"prp-e","date":"December 31, 2024","format":false,"excerpt":"In 2022, ChatGPT released and LLMs becoming the hot topic of pretty much every technology related press, event, YouTube video, etc. It was like finding the secret ingredient to a potion which can make you immortal. But Meta didn't let OpenAI becoming the one and only. They also started the\u2026","rel":"","context":"In &quot;Projects&quot;","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/319996af-289a-4617-8d0c-6580e4793747.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/319996af-289a-4617-8d0c-6580e4793747.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/319996af-289a-4617-8d0c-6580e4793747.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/319996af-289a-4617-8d0c-6580e4793747.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/319996af-289a-4617-8d0c-6580e4793747.png?resize=1050%2C600&ssl=1 3x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/354"}],"collection":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/comments?post=354"}],"version-history":[{"count":7,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/354\/revisions"}],"predecessor-version":[{"id":361,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/354\/revisions\/361"}],"wp:attachment":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/media?parent=354"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/categories?post=354"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/tags?post=354"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}