Stable Diffusion XL (SD-XL) Stability AI, an advanced algorithm developed by Stability AI, is almost complete. Initially, only ComfyUI was compatible with this AI, but I recently discovered that Vlad also supports it, so I decided to give it a try. For those unfamiliar with Vlad, you can find detailed articles in the Vlad sub-tab of this topic, which includes information about Automatic1111’s evolution: Vlad Diffusion – Fixed Diffusion. It’s important to note that SD-XL and SD-XL Vlad support are still in the experimental phase, so this experience may not cover the basics.
To install SD-XL on Vlad, you need to update Vlad to the latest version. Open the command prompt in the default directory or the installation folder and enter the following command: “git remove https://github.com/vladmandic/automatic”. For a more detailed guide on updating Vlad via webui, refer to the Fixed Diffusion documentation.
Before being able to download the .safetensor model for SD-XL from Vlad, you must apply for permission from HuggingFace. Due to the non-public nature of the SD-XL model, both ComfyUI and Vlad require users to register with HuggingFace and obtain permission to access SD-XL. The registration process for HuggingFace is not mentioned here, but you can request access to SD-XL by visiting the following URL: [https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/tree/main](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/tree/main). When downloading the file “sd_xl_base_0.9.safetensors” for the first time, a user specification page will appear. Complete the required information at the bottom and submit your request. Approval will be granted immediately, allowing you to generate an Access Token.
To download SD-XL in Vlad, you need to generate your own Access Token in HuggingFace. Access the Settings page on the HuggingFace website and click the New Ticket button to generate a new token. Make sure to select the “Read” option for permissions. Name the token as desired and remember where it will be used. After generating the token, copy it by clicking the button with two overlapping squares on the right side of Show.
To install and download SD-XL 0.9 in Vlad, navigate to the Vlad interface and go to the following locations: models > Separators. You need to download and enter two models directly into the Select model field: “stabilityai/stable-diffusion-xl-base-0.9” and “stabilityai/stable-diffusion-xl-driver-0.9”. Paste your own Access Token in the Huggingface token field and click on the Download model button. Please note that the file is quite large, so you might need to wait patiently for the download to complete. If you encounter any issues during the download, you can generate a new Access Token and try again.
Once both models are downloaded, it is recommended to close the Vlad command prompt window and reopen Vlad. The models will appear as folders in the models > Separators directory.
It is important to note that SD-XL is based on the Diffuser model and differs from the commonly used 1.5 model. The sampling methods available are also different, and Vlad has made some customizations for SD-XL. SD-XL is specifically trained with 1024×1024 images, so the Text2Image feature cannot utilize the lower image resolutions available in the 1.5 model. Generating an SD-XL image is a two-step process, requiring both the base and refinement models to be downloaded. SD-XL also uses more VRAM, but you can use the options “-lowvram” and “-medvram” to maintain relatively good performance. Certain features, such as the 1.5 super plugin ControlNet, are not compatible with SD-XL. However, the VAE can be used for Textual Inversion and LoRA, but training is required. Additionally, the LyCORIS feature is unable to be used with SD-XL.
Given these factors, the current adaptability of SD-XL is not as good as the 1.5 model. However, it is possible to change modes in Vlad to use SD-XL. To do so, modify the webui-user.bat file in the same folder as webui.bat, creating your own .bat file with the following content:
Mode 1.5/2.0/2.1:
“`
@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= –original backend
call webui.bat
“`
SD-XL mode:
“`
@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= diffusers –backend
call webui.bat
“`
Please note that the mode switch is permanent, meaning that once you switch to SD-XL mode, it will remain in that mode until you switch back to 1.5/2.0/2.1 mode.
In terms of settings and related notes, it is important to be aware of a memory leak issue in Vlad’s SD-XL mode. This means that VRAM is not fully released after image calculations, and excessive calculations may result in high VRAM usage warnings in the command prompt. If VRAM is insufficient, the generated image may appear incomplete or completely black. To mitigate this issue, you can disable Vlad Reopen.
Furthermore, there are certain settings that can help optimize VRAM usage. You may consider modifying the DPM and related sampling algorithm options to make appropriate comparisons. It is important to note that the DPM solver algorithm is not directly related to SD-XL, but it may need to be adjusted when switching to SD-XL mode to avoid errors.
In terms of impressions, SD-XL presents a different approach compared to the 1.5 model. Some key differences include the need to carefully set negative prompts in the purifier model, as excessive negative prompts can have negative effects. Additionally, popular techniques like Textual Inversion used in 1.5 are not effective in SD-XL. SD-XL 0.9 excels in natural language processing and offers comprehensive style options. However, if the style prompts are not clear enough, it can be challenging to achieve the desired style.
It is worth mentioning that SD-XL has a preference for American styles, which may not be suited to everyone’s preferences. While it can produce realistic images with a good compromise between details and clarity, it is still a model with some limitations.
Overall, SD-XL represents a step towards better emulation, and as the model and algorithms are further developed, it holds promise for creating unique and high-quality artwork.
Recently SD-XL (Stable Diffusion XL) Stability AI is nearing completion. Originally, only ComfyUI could support it, but recently I noticed that Vlad also supports it, so I tried it.
If you don’t know what Vlad is, see the articles in the Vlad sub-tab of this topic. Including Automatic1111’s evolution: Vlad Diffusion – Fixed Diffusion.
At this stage, SD-XL and SD-XL Vlad support is in the experimental stage, thereforeThis experience does not cover the basicssomething.
Installation process | HUGGINGFACE can’t run away
Update Vlad to the latest version first
Open cmd in the default directory (or other installation folder) and enter:
remove git https://github.com/vladmandic/automatic
See Vlad webui update process – Fixed Diffusion
Vlad is updated very often, so it is possible to see a slightly different UI after the update.
Apply for HuggingFace permission
Vlad cannot download the .safetensor model from SD-XL as ComfyUI directly, must be downloaded via the Vlad Model tab. Because the SD-XL model is not public, ComfyUI and Vlad must register HuggingFace and apply for permission to get SD-XL.
I will not mention the HuggingFace registration process, the request for SD-XL access can be made through the following URL:
https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/tree/main
When you try to download the file sd_xl_base_0.9.safetensors for the first time, the user specification page will appear. Complete some basic information at the bottom and submit a request, and the permission will be opened immediately.
Generate an Access ticket
Next, you need to generate your own Access Token in HuggingFace, which will be used to download SD-XL in Vlad. The location is as shown in the screenshot below:
On the HuggingFace website go to Settings
Click the New Ticket button to generate,Please type select Read. Name it at will, let yourself remember where the Token is used:
Remember to copy after a generation, click the button with two overlapping squares on the right side of Show
Install and download SD-XL 0.9
To enter the Vlad interface, go to the following places:
A total of two models need to be downloaded and entered directly into the Select model:
stabilityai/stable-diffusion-xl-bas-0.9
as well as
stabilityai/stable-diffusion-xl-driver-0.9
Then paste your own Access Token in the Huggingface token field, and then click on Download model. The file is very large, so you need to wait patiently for the progress bar in cmd to finish running.
*The Access Ticket may be invalid. If you find that the download fails, you can generate a new ticket and try again.
After both models are downloaded, it is recommended to close the Vlad cmd window and reopen Vlad.
Models are presented as folders in models > Separators
Prerequisites/Installations/Precautions
SD-XL is an algorithm based on the Diffuser model, which is different from the currently commonly used 1.5. The Sampling Methods that can be used are also different (Vlad will customize it on his own). The most important and effective differences are as follows:
It is trained with 1024×1024 images, so Text2Image cannot use the small image resolution of 1.5, the result will be very badGenerating an SD-XL image is a two-step process, so the base and refinement models need to be downloadedIt uses more VRAM, but -lowvram -medvram keeps relatively good performance Can’t use 1.5 super plug-in ControlNet can’t use 1.5 VAE can use Textual Inversion, LoRA, but can’t use 1.5, must be trained’ n extra Unable to use LyCORIS
Due to the above 4~6 reasons, and there are no other derivative models at present, the current adaptability of SD-XL is not as good as 1.5.
How to change modes
In terms of practical use, the most important things are:
Using SD-XL requires changing Vlad mode.
Although there are options at the bottom of the tab Settings > Stable Vlad Diffusion:
original = mode 1.5/2.0/2.1 | diffusers = SD-XL mode
But a simpler way is to make webui-user.bat similar to A1111 yourself, and change directly at startup.. Create your own .txt file and change the extension to .bat, enter the following content and save it in the same folder as webui.bat, and then use the .bat you created to change modes when starting Vlad.
Mode 1.5/2.0/2.1:
@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= –original backend
call webui.bat
SD-XL mode:
@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= diffusers –backend
call webui.bat
Note: the switching mode is a permanent state, that is, after switching to SD-XL mode, it will maintain SD-XL mode until you switch back to 1.5/2.0/2.1 mode.
Settings/Related Notes
First things first:
Vlad’s SD-XL mode seems to have a memory leak problemthat is, the release of VRAM is not complete after the image is calculated, so after a few more calculations, cmd will display a warning of high VRAM usage, and then it will run out of a semi-finished or completely black image when the VRAM is insufficient. Currently, you can only turn off Vlad Reopen.
Related settings | Paste screenshots directly for reference:
Settings to save VRAM usage
The dpm related sampling algorithm may attempt to change the options in the red box to compare differently
About Samplers The DPM solver algorithm is not directly related to SD-XL, but when I switch to SD-XL mode for the first time, I have to apply this option and use DEIS and other related algorithms to avoid errors.
In short, many SD-XL modes are in the experimental stage, including the inability to count half the skip/stop, so you have to be mentally prepared to encounter frequent errors.
Impressions | SD-XL is like facing an artist with completely different habits
Many of the 1.5 routines do not apply to SD-XL. There are three main ones:
The negative impulses of the purifier model need to be set Writing too much will have negative effects, and the Textual Inversion (deep negative, etc.) of 1.5 is invalid. SD-XL 0.9 is more suitable for natural language, and the style is comprehensive, so if the prompt words in terms of style are not clear enough, it is difficult to get the desired style.
The installation location of the purifier model is as follows:
The effect of nothing (that is, not used) can be tried, but it’s usually boring:
Comparison of Negative Tips:
Negative prompt: text, watermark, blur, painting, 3d rendering, smooth, plastic, grainy, low resolution, deep fried, oversaturated,
Negative prompt: bad hands-5, ng_deepnegative_v1_75t, extra fingers, deformed hands, polydactyl, ((low quality, worst quality, monochrome, grayscale, grayscale, watermark, text, blur, jpeg artifacts)), cropping, normal quality , ((signature, username, artist name, logo)), cartoon, canvas frame, ((lowres)), mutilated, bad art, mutilated, extra limbs, b&w, weird colors, duplicate hands, morbid, mutilated, mutated, badly drawn hands , badly drawn face , mutation , ugly , bad proportions , cloned face , out of frame , gross proportions , malformed legs , missing arms , missing legs, extra arms, extra legs, fused fingers, long neck, skin spots, acnes, skin spots, age spot,
Style queue comparison:
Japanese anime style ｜ Japanese anime style – honestly, not like
manga style, full color ｜ Manga Style – Haomei Style
Japanese shounen manga style, full color ｜ Japanese shounen manga style – still good American style
Japanese shoujo anime, full color | Japanese girl manga style – too lazy to complain
After all, it is a model made by Americans, so it can be seen that there is a strong preference for American styles, including the first picture which is more realistic, and the faces of Asians are based on American styles.
It seems that we can only wait for the derivative model after the official launch in the future.
SD-XL understands as far as it goes
The SD-XL has one advantage over the 1.5 model.
1.5 Even if it is a realistic model, the pictures produced often have an inexplicable sense of inauthenticity. The 1.5 model has no “concept of compromise”If you want details, you should get the details together, and if you want to be clear, you should be clear together, but often the actual pictures and characters are not like this.
SD-XL can make better compromises in this regard, as a close-up shot:
In addition to the fact that the background is blurred as in a real photo, only the focus face of the whole character is clear and the details are clear, while the other parts gradually reduce the details and increase the blur. And even if the lighting method is not specifically specified in the stimulus, SD-XL still produces a feeling of light and shadow that is well suited for the outdoors.
Although I will continue to use 1.5 before SD-XL is officially launched and models are modified, I personally look forward to using SD-XL to take a step closer to emulation in the future.
Happy drawing everyone!
#VLAD #SDXL #Early #Access #Fixed #Diffusion
