{"id":2449,"date":"2023-05-21T22:18:02","date_gmt":"2023-05-22T05:18:02","guid":{"rendered":"https:\/\/xyzcreativeworks.com\/?p=2449"},"modified":"2023-05-23T18:48:33","modified_gmt":"2023-05-24T01:48:33","slug":"is-stable-diffusion-inpaint-obsolete-versus-instructpix2pix-add-glasses-to-face","status":"publish","type":"post","link":"https:\/\/xyzcreativeworks.com\/is-stable-diffusion-inpaint-obsolete-versus-instructpix2pix-add-glasses-to-face\/","title":{"rendered":"Is Stable Diffusion Inpaint Obsolete Versus InstructPix2Pix? (Add Glasses to Face)"},"content":{"rendered":"\n

New tech and methods are always exciting, but I have started to think old methods that endure are still the best.<\/p>\n\n\n\n

I recently received a request to find out how to put sunglasses on an image.<\/p>\n\n\n\n

I tried two methods: Inpaint and InstructPix2Pix.<\/p>\n\n\n\n

The goal is to put sunglasses on the portrait. Black sunglasses with minimal reflections. Like these images of Jason Statham:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n

Base Image<\/h2>\n\n\n

When doing something experimental, I like using the least complicated images.<\/p>\n\n\n\n

In the case of sunglasses, we don’t want long hair making it difficult for the AI. <\/p>\n\n\n\n

Having hair would mean the AI has to find a way to hide the frame behind the hair. That makes it a bit more difficult.<\/p>\n\n\n\n

And when you make things difficult for AI, you’ll need to do more Photoshopping.<\/p>\n\n\n\n

Anyway, here’s the base image: a bald guy.<\/p>\n\n\n

\n
\"bald<\/figure><\/div>\n\n

Inpainting In Stable Diffusion<\/h2>\n\n\n

Inpainting in Stable Diffusion refers to a powerful technique used in image editing to restore missing parts of pictures or create entirely new elements within an existing image. \\<\/p>\n\n\n\n

It’s the basis of how you can do a face swap in Stable Diffusion<\/a>.<\/p>\n\n\n\n

In this case, the guy isn’t wearing sunglasses, but we want him to.<\/p>\n\n\n\n

\"putting<\/figure>\n\n\n\n

It’s pretty simple. Just mask the areas where a sunglasses will go.<\/p>\n\n\n\n

I am using Realistic Vision 2.0 as a model and a simple prompt “sunglasses on face”. I tried adding “black” but I still got multi-coloured sunglasses.<\/p>\n\n\n\n

One thing I like to do is to create a CFG-Denoising plot. I’ve got a guide on how to set it up here<\/a>.<\/p>\n\n\n\n

You’ll know why:<\/p>\n\n\n\n

\"sunglasses<\/a>
Click to enlarge<\/figcaption><\/figure>\n\n\n\n

In fact, setting a X\/Y plot up is so important that I’ve already set it up so it appears by default<\/a>.<\/p>\n\n\n\n

As you can see, the middle CFG and middle denoising produces the best images.<\/p>\n\n\n\n

Too little denoising and nothing happens. Too much and it becomes a face in a face.<\/p>\n\n\n\n

\n
\"stable<\/figure>\n\n\n\n
\"stable<\/figure>\n\n\n\n
\"stable<\/figure>\n\n\n\n
\"stable<\/figure>\n<\/figure>\n\n\n\n

These are some hopefuls from this Inpainting method.<\/p>\n\n\n

InstructPix2Pix and Adding Sunglasses<\/h2>\n\n\n

Next, I’ll try the InstructPix2Pix method.<\/p>\n\n\n\n

Today, you don’t even need to install an extension for Automatic1111. You just need to download the checkpoint<\/a>, put it into your Stable Diffusion models folder. I used the .ckpt file, rather than the .safetensors file.<\/p>\n\n\n\n

Steps:<\/p>\n\n\n\n

    \n
  1. Download the checkpoint and put it into your Stable Diffusion models folder<\/li>\n\n\n\n
  2. Go into Stable Diffusion (make sure to refresh UI)<\/li>\n\n\n\n
  3. Select the instructpix2pix checkpoint<\/li>\n\n\n\n
  4. Wait for it to load<\/li>\n\n\n\n
  5. Go to img2img and type in a prompt in conversational English.<\/li>\n<\/ol>\n\n\n\n

    I prompted “put sunglasses on him”<\/p>\n\n\n\n

    \"instructpix2pix<\/figure>\n\n\n

    Run a Image CFG Plot in InstructPix2Pix<\/h2>\n\n\n

    I chose to run a graph plot for Image CFG, a variable that’s specific to InstructPix2Pix.<\/p>\n\n\n\n

    I fixed the seed to “3”, and then ran a Image CFG plot that jumps 0.3 in value from 0.1 to 2.8 (delta = 2.7).<\/p>\n\n\n\n

    Here’s the output:<\/p>\n\n\n\n

    \"\"<\/a>
    Click to enlarge<\/figcaption><\/figure>\n\n\n\n

    Interesting. It seems that at lower Image CFGs, the whole image gets changed.<\/p>\n\n\n

    \n
    \"low<\/figure><\/div>\n\n\n

    Who dat?!<\/p>\n\n\n\n

    At higher Image CFGs, the original image is preserved, as below.<\/p>\n\n\n

    \n
    \"high<\/figure><\/div>\n\n

    But Wait, There Are No Temples on the Glasses<\/h2>\n\n\n

    Unlike the inpainting method, InstructPix2Pix doesn’t seem to connect the glasses to the ears.<\/p>\n\n\n\n

    This is a bit of a problem because I’d have to do further adjustments to the image in order to get to the same outcome as the inpainting method.<\/p>\n\n\n

    Let\u2019s Refine the Inpainting Method<\/h2>\n\n\n

    My goal is to have black sunglasses that look elegant, befitting of the guy in the image.<\/p>\n\n\n\n

    Let’s try drawing a more precise inpainting mask.<\/p>\n\n\n\n

    \"a<\/figure>\n\n\n\n

    Out of a batch of four, I found this to be the best.<\/p>\n\n\n

    \n
    \"yellow<\/figure><\/div>\n\n

    Making the Sunglasses Black in img2img<\/h2>\n\n\n

    I just can’t seem to get the sunglasses black. Perhaps I need to emphasize specific parts of my prompt<\/a>.<\/p>\n\n\n\n

    The old prompt was “black sunglasses”.<\/p>\n\n\n\n

    Let’s try three brackets surrounding “black sunglasses”. <\/p>\n\n\n\n

    The new prompt is (((black sunglasses))).<\/p>\n\n\n\n

    It also turns out that having “B&W” in your negative prompt will affect the black colour of the glasses. Once I removed “B&W”, I got much better chances of getting black sunglasses.<\/p>\n\n\n

    \n
    \"\"<\/figure><\/div>\n\n

    Photoshop as Final Step to the img2img Process<\/h2>\n\n\n

    The final step is to use Photoshop. The image above seems OK, but the sunglasses’ lens looks a bit too pale. <\/p>\n\n\n\n

    Let’s put some darkening curves on the lenses.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    And here’s the result:<\/p>\n\n\n

    \n
    \"photoshop<\/figure><\/div>\n\n

    A Workflow for Adding Sunglasses Onto Image in Stable Diffusion<\/h2>\n\n\n

    My thoughts so far: you’d do well to try both methods and see what you get.<\/p>\n\n\n\n

    Remember, this image is just one of the many images that you might need to put sunglasses onto.<\/p>\n\n\n\n

    You might get a person with hair. You might get a person looking to the side. <\/p>\n\n\n\n

    Basically, the methods above are useful in all scenarios, but the effectiveness of each one might vary depending on your variables.<\/p>\n\n\n\n

    Here’s a general overview of the steps I’d take:<\/p>\n\n\n\n

      \n
    1. Study the input image<\/li>\n\n\n\n
    2. Try instructpix2pix FIRST. Satisfied? Job done.\n