{"id":1413,"date":"2023-06-26T12:20:09","date_gmt":"2023-06-26T06:20:09","guid":{"rendered":"https:\/\/magiteker.com\/?p=1413"},"modified":"2023-06-27T06:05:49","modified_gmt":"2023-06-27T00:05:49","slug":"experiment-high-accuracy-training","status":"publish","type":"post","link":"https:\/\/magiteker.com\/index.php\/2023\/06\/26\/experiment-high-accuracy-training\/","title":{"rendered":"Experiment: High Accuracy Training"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Utilizing Kohya_ss GUI I&#8217;m experimenting with various settings sourced from different tutorials in the hopes of gathering a basic configuration that can generate accurate renders while using as little time and VRAM as possible during training.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Methodology<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1.) To quickly iterate through configurations I&#8217;m using a small dataset of 5 images, all with captions, to ensure model training happens in approximately and hour or less.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2.) To control the experiment I&#8217;ll be adjusting several training settings:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Number of Repeats<\/li>\n\n\n\n<li>Batch Size<\/li>\n\n\n\n<li>LR Scheduler<\/li>\n\n\n\n<li>Max Resolution<\/li>\n\n\n\n<li>Max Token Length<\/li>\n\n\n\n<li>Regularization Images ( from model vs from dataset )<\/li>\n\n\n\n<li>Learning Rate<\/li>\n\n\n\n<li>Offset Noise<\/li>\n\n\n\n<li>Prior Loss Weight<\/li>\n\n\n\n<li>Memory Efficient Attention<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">3.) Other settings will remain constant through the experiment, these settings include LR Scheduler set to cosine with repeats, optimizer set to Lion, CLIP skip set to 2, and optimization settings such as Gradient Checkpointing and Shuffle Tags enabled. These settings have been set because thus far they&#8217;re the most efficient settings for low VRAM training.<br><br>Note: Xformers has been disabled based on the fact they make the model non-deterministic and would thus reduce accuracy of images rendered.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">4.) In order to gauge success the trainer will generate 10 sample renders using a prompt specific to the dataset. Also at each render interval the training speed ( it\/s ), time lapsed, and loss rate will be recorded for each experiment. These measurements will elucidate the impacts of the various control settings on training.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Initial Settings<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The following will be the base settings assumed to be good based on current research:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Repeats: 10<\/li>\n\n\n\n<li>Batch Size: 1<\/li>\n\n\n\n<li>Epochs: 1<\/li>\n\n\n\n<li>Resolution: 512, 512<\/li>\n\n\n\n<li>Token Length: 75<\/li>\n\n\n\n<li>Regularization: 0<\/li>\n\n\n\n<li>LR: 1e-5<\/li>\n\n\n\n<li>Offset Noise: 0<\/li>\n\n\n\n<li>Prior Loss Weight: 1.0<\/li>\n\n\n\n<li>Memory Efficient Attention: Off<\/li>\n\n\n\n<li>LR Scheduler: Constant<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Equipment<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These experiments are being done on my local machine using an RTX 4090 and Ryzen 5900X with 32Gb of DDR4 RAM. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Each experiment will be a separate post on this blog complete with data and observations. The conclusion will also be here with the optimal training settings discovered.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Utilizing Kohya_ss GUI I&#8217;m experimenting with various settings sourced from different tutorials in the hopes of gathering a basic configuration that can generate accurate renders while using as little time and VRAM as possible during training. Methodology: 1.) To quickly iterate through configurations I&#8217;m using a small dataset of 5 images, all with captions, to [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[67],"tags":[68,71,72,69],"class_list":["post-1413","post","type-post","status-publish","format-standard","hentry","category-stablediffusionai","tag-ai","tag-dreambooth","tag-experiment","tag-stable-diffusion"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p6MIgq-mN","jetpack-related-posts":[{"id":1437,"url":"https:\/\/magiteker.com\/index.php\/2023\/07\/08\/tbd-experiment-multiple-classifications\/","url_meta":{"origin":1413,"position":0},"title":"TBD Experiment: Multiple Classifications","author":"Frank O'Hanlon","date":"July 8, 2023","format":false,"excerpt":"Hypothesis: Dreambooth can utilize multiple classifications in regularization to improve model training while maintaining model style and data. Each classification should correspond to a tag that is common throughout the training data.","rel":"","context":"In &quot;Stable Diffusion&quot;","block_context":{"text":"Stable Diffusion","link":"https:\/\/magiteker.com\/index.php\/category\/tutorials\/stablediffusionai\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1350,"url":"https:\/\/magiteker.com\/index.php\/2023\/06\/21\/stable-diffusion-experiments\/","url_meta":{"origin":1413,"position":1},"title":"Stable Diffusion Experiments","author":"Frank O'Hanlon","date":"June 21, 2023","format":false,"excerpt":"AI module experiments","rel":"","context":"In &quot;Stable Diffusion&quot;","block_context":{"text":"Stable Diffusion","link":"https:\/\/magiteker.com\/index.php\/category\/tutorials\/stablediffusionai\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1382,"url":"https:\/\/magiteker.com\/index.php\/2023\/06\/21\/resource-training-stable-diffusion-with-dreambooth\/","url_meta":{"origin":1413,"position":2},"title":"Resource: Training Stable Diffusion with Dreambooth","author":"Frank O'Hanlon","date":"June 21, 2023","format":false,"excerpt":"-Blog Post This article describes experiments with different Learning Rates in training models using Dreambooth. FTA: Summary of Initial Results To get good results training Stable Diffusion with Dreambooth, it's important to tune the learning rate and training steps for your dataset. High learning rates and too many training steps\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/magiteker.com\/index.php\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1411,"url":"https:\/\/magiteker.com\/index.php\/2023\/06\/24\/dreambooth-kohya-ss-install\/","url_meta":{"origin":1413,"position":3},"title":"Dreambooth Kohya SS Install","author":"Frank O'Hanlon","date":"June 24, 2023","format":false,"excerpt":"- Github Above is the repository for Kohya SS Dreambooth, to install simply clone the repository to a separate folder using Git then run the install.bat file. Note: For better training speed ( iterations per second ) using RTX 40xx GPU's it's advised to use CUDA 11.8 so that Xformers\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/magiteker.com\/index.php\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1386,"url":"https:\/\/magiteker.com\/index.php\/2023\/06\/21\/resource-dreambooth-training-readme\/","url_meta":{"origin":1413,"position":4},"title":"Resource: Dreambooth Training README","author":"Frank O'Hanlon","date":"June 21, 2023","format":false,"excerpt":"- Github README describing proper methods for constructing a dataset.According to the README Regularization images allow for prior preservation of a models existing dataset. Regularization images should be generated before training from the base model. Image classification should be a generic term ( i.e. person, cat, dog, man, woman ).\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/magiteker.com\/index.php\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/magiteker.com\/wp-content\/uploads\/2023\/06\/image-1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/magiteker.com\/wp-content\/uploads\/2023\/06\/image-1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/magiteker.com\/wp-content\/uploads\/2023\/06\/image-1.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/magiteker.com\/wp-content\/uploads\/2023\/06\/image-1.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1441,"url":"https:\/\/magiteker.com\/index.php\/2023\/07\/21\/retrieval-based-voice-conversion\/","url_meta":{"origin":1413,"position":5},"title":"Retrieval Based Voice Conversion","author":"Frank O'Hanlon","date":"July 21, 2023","format":false,"excerpt":"-Github Retrieval Based Voice Conversion is a technique for analyzing speech recordings and training an AI model to emulate vocal patterns. Using this system it is possible to create text to speech models with more natural voices. These models can also be utilized to sing lyrics to songs or recite\u2026","rel":"","context":"In &quot;Stable Diffusion&quot;","block_context":{"text":"Stable Diffusion","link":"https:\/\/magiteker.com\/index.php\/category\/tutorials\/stablediffusionai\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/posts\/1413","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/comments?post=1413"}],"version-history":[{"count":8,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/posts\/1413\/revisions"}],"predecessor-version":[{"id":1426,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/posts\/1413\/revisions\/1426"}],"wp:attachment":[{"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/media?parent=1413"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/categories?post=1413"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/magiteker.com\/index.php\/wp-json\/wp\/v2\/tags?post=1413"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}