{"id":391,"date":"2024-11-27T21:48:34","date_gmt":"2024-11-27T18:18:34","guid":{"rendered":"https:\/\/haghiri75.com\/en\/?p=391"},"modified":"2024-11-27T21:48:34","modified_gmt":"2024-11-27T18:18:34","slug":"lets-build-metaverse-with-ai-building-asset-generator","status":"publish","type":"post","link":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-building-asset-generator\/","title":{"rendered":"Let’s build Metaverse with AI: Building asset generator"},"content":{"rendered":"

Look at this:<\/p>\n\r\n

<\/div>\r\n\r\n \n

How do you think this apple has been made? Excellent question. After the previous post<\/a>, I said we should put LLMs out of the picture for now. Also we needed to talk about 3D<\/a>, because it is important in whole metaverse space, right? Today I just did it. I trained a LoRA on FLUX and then tried to make 3D objects from what an AI model is capable of generating.<\/p>\n

The Image Generator<\/h1>\n

In this part, I specifically talk about the image generation procedure. It will be a good experience sharing procedure and the open source models created in this process will be linked in the topic as well.<\/p>\n

For making an image generator model, we need a base model. Since the whole\u00a0Generative Metaverse\u00a0<\/em>project for me was a fun project and not a serious commercial one, I chose FLUX. However, if I try to go to the blockchain\/crypto side of things (probably on TON<\/a> network) I may consider SDXL as base in order to have no problems in terms of commercial use.<\/p>\n

Anyway, everything here is pretty standard. Pretty much every step I took in order to make early versions of Mann-E<\/a>. So I guess it will be worth sharing one more time, right?<\/p>\n

The Dataset<\/h2>\n

AI models are just a bunch of boring mathematical functions and they become amazing when they are fed with good data. So we needed to create a dataset. As always, the best data generator I could use was\u00a0Midjourney\u00a0<\/em>and of course, I headed over to their website<\/a> and recharged my account.<\/p>\n

I played with a good bunch of prompt combinations to find what is the best one fitting what I have in mind. So after tweaking a lot, I got this: <subject>, lowpoly, 3d illustration, dark background, isometric camera angle.\u00a0<\/em><\/p>\n

Here is a sample of what generated with this prompt formula:<\/p>\n

<\/a><\/p>\n

After that, I used ChatGPT<\/a> in order to generate a list of objects we may use or see everyday. After that, I made a prompt list and automated the image generation procedure and got around 800 pictures. Now it was time for training!<\/p>\n

The training<\/h2>\n

First, I was thinking about using Replicate<\/a> or fal.ai<\/a> in order to train the LoRA. Honestly they provide easy and affordable ways of training LoRA on FLUX (and to my knowledge, you also may be able to have SD 1.5 and SDXL LoRA’s trained on replicate) but there is one big problem.<\/p>\n

These websites are usually not suitable for large scale training or if they offer large scale training systems, you should negotiate with them and as I said, this is a fun project. Not a big OpenAI scale commercial product!<\/p>\n

So I was looking for another way. As you may know, Google Colab’s free tier subscription is also no good for FLUX training. So I used\u00a0AI Toolkit\u00a0<\/em>template on RunPod<\/a> in order to train the said LoRA. I used an 80GB A100 and it took around 3 hours on 100 pictures.<\/p>\n

The files<\/h2>\n

If you’re interested in the dataset, I uploaded the whole dataset and pictures here<\/a>. You can see there is a folder called\u00a0minimized images\u00a0<\/em>which is 100 hand picked images from the original dataset.<\/p>\n

And if you’re looking for the LoRA, you can download and even test it here<\/a>.<\/p>\n

<\/a><\/p>\n

The 3D Generation<\/h1>\n

Well, after making the image generator, we needed a way of turning\u00a0single images to 3D files\u00a0<\/em>and of course the 3D format must be something acceptable for all devices.<\/p>\n

OBJ and FBX are great formats when it comes to game development (specially if you’re using Unity game engine) but for WebGL and WebXR, gLTF or GLB formats are usually preferred.<\/p>\n

The best option for this, is fal.ai’s TripoSR API<\/a>. You upload your image, the model is being called and BOOM you have a GLB file which can be used on every WebGL or WebXR project you can think of.<\/p>\n

<\/a><\/p>\n

What’s next?<\/h1>\n

Since I personally am working on another project with Mann-E’s proprietary models, I may stop this particular project right here. I almost did everything I had in mind.<\/p>\n

Although we still have the important topic of\u00a0world generation\u00a0<\/em>using AI, but I guess it needs a more in depth study and will not be this easy at all. Also the commercializing process of the whole thing is also a topic of thought and for now, I just want to keep the project fun.<\/p>\n

Maybe in a few weeks, I return with a more commercial approach and also some ideas about the whole blockchain or crypto space.<\/p>\n","protected":false},"excerpt":{"rendered":"

Look at this: How do you think this apple has been made? Excellent question. After the previous post, I said we should put LLMs out of the picture for now. Also we needed to talk about 3D, because it is important in whole metaverse space, right? Today I just did it. I trained a LoRA … <\/p>\n

Continue reading “Let’s build Metaverse with AI: Building asset generator”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[4],"tags":[36,37,11,22,45,46,24,25,49,50],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8BkKn-6j","jetpack-related-posts":[{"id":344,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-introduction\/","url_meta":{"origin":391,"position":0},"title":"Let’s build Metaverse with AI : Introduction","author":"prp-e","date":"November 21, 2024","format":false,"excerpt":"It was 2021, the whole products under the flag of Facebook, went down for a few hours. I remember that most of my friends just started messaging me on Telegram instead of WhatsApp and also no new post or story was uploaded on Instagram. A few hours passed, everything went\u2026","rel":"","context":"In "Projects"","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/64f5d67a-c1c7-49f2-88a8-6870b20b4cc9.png?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":335,"url":"https:\/\/haghiri75.com\/en\/privacy-focused-ai-is-all-we-need\/","url_meta":{"origin":391,"position":1},"title":"Privacy-focused AI is all we need","author":"prp-e","date":"November 1, 2024","format":false,"excerpt":"I remember in 2020 and 2021, due to Elon Musk's interest in crypto and also\u00a0The Metaverse Hype\u00a0people, specially the ones who had no idea about crypto or blockchain, started investing in the crypto markets. Although it seemed a little bit of a failure, people made profit out of it. It\u2026","rel":"","context":"In "Projects"","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":318,"url":"https:\/\/haghiri75.com\/en\/frontbricks-my-llm-based-weekend-project-which-is-inspired-by-vercels-v0\/","url_meta":{"origin":391,"position":2},"title":"FrontBricks, my LLM-based weekend project which is inspired by Vercel’s V0","author":"prp-e","date":"May 31, 2024","format":false,"excerpt":"Since 2022, there is a hype of\u00a0generative artificial intelligence\u00a0and it resulted in a bunch of cool projects. Although a lot of us may remember that Github's copilot was much older. Those days, I wrote an article about how I was too cheap to pay $10 a month for copilot, so\u2026","rel":"","context":"In "Computer Architecture & Programming"","block_context":{"text":"Computer Architecture & Programming","link":"https:\/\/haghiri75.com\/en\/category\/computer\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/frontbricks.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/frontbricks.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/frontbricks.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/frontbricks.png?resize=700%2C400 2x, https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/frontbricks.png?resize=1050%2C600 3x"},"classes":[]},{"id":192,"url":"https:\/\/haghiri75.com\/en\/i-was-to-cheap-to-pay-10-a-month-for-copilot-so-i-made-my-own\/","url_meta":{"origin":391,"position":3},"title":"I was too cheap to pay $10 a month for copilot, so I made my own","author":"prp-e","date":"September 4, 2022","format":false,"excerpt":"In mid 2021, there was a revolution in coding. As a lazy programmer who always needed a fast and smart assistant, I was really happy to have Github Copilot in my arsenal of coding tools. So I was one of the early adapters of the whole idea of AI pair\u2026","rel":"","context":"In "Computer Architecture & Programming"","block_context":{"text":"Computer Architecture & Programming","link":"https:\/\/haghiri75.com\/en\/category\/computer\/"},"img":{"alt_text":"My own copilot!","src":"https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/Screen-Shot-2022-09-03-at-10.46.46-PM.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/Screen-Shot-2022-09-03-at-10.46.46-PM.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/haghiri75.com\/en\/wp-content\/uploads\/Screen-Shot-2022-09-03-at-10.46.46-PM.png?resize=525%2C300 1.5x"},"classes":[]},{"id":354,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-what-we-have\/","url_meta":{"origin":391,"position":4},"title":"Let’s build Metaverse with AI: What we have?","author":"prp-e","date":"November 22, 2024","format":false,"excerpt":"In the previous post about building metaverse with AI (link), I discussed the generic points of view, what we need and all the stuff like that. In this post, I am going to discuss about AI models we have which can be helpful in order to build the metaverse using\u2026","rel":"","context":"In "Projects"","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/75117f84-c86f-49cc-b41a-4734b8e2db94_part_0001.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":362,"url":"https:\/\/haghiri75.com\/en\/lets-build-metaverse-with-ai-we-need-to-talk-about-3d\/","url_meta":{"origin":391,"position":5},"title":"Let’s build Metaverse with AI: We need to talk about 3D","author":"prp-e","date":"November 24, 2024","format":false,"excerpt":"In the previous post about building metaverse with AI, we discussed different possibilities and AI models we can access in order to make the virtual world. Although I personally am a big fan of 2D worlds, but let's be honest, a 2D world is basically a perfect choice for a\u00a0low\u2026","rel":"","context":"In "Projects"","block_context":{"text":"Projects","link":"https:\/\/haghiri75.com\/en\/category\/projects\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mann-e-images.storage.c2.liara.space\/00624e43-bdaa-412e-9cc2-b486b47cb947_part_0001.png?resize=1400%2C800&ssl=1 4x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/391"}],"collection":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/comments?post=391"}],"version-history":[{"count":6,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/391\/revisions"}],"predecessor-version":[{"id":397,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/posts\/391\/revisions\/397"}],"wp:attachment":[{"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/media?parent=391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/categories?post=391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/haghiri75.com\/en\/wp-json\/wp\/v2\/tags?post=391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}