Editing Training Photos

Keep the real cat, generate new surroundings with inpainting and outpainting, and test whether the classifier needs new cats or new backgrounds.

Previous · Course overview · Next

Your Goal

In Meetings 2 and 3 the model drew the whole cat, so it could also draw the wrong breed. Inpainting and outpainting keep every pixel of the real cat and change only what surrounds it. Find out whether new backgrounds and framing help the classifier as much as new cats do.

What the Repository Gives You

Mode What changes Setting
inpaint the background; the cat’s pixels are kept --prompt: the scene; repeat it for several
outpaint the framing: the photo shrinks into the centre and the model paints around it --outpaint-scale: from 1.25 to 2 
uv run synth-generate --name m4-background-one --mode inpaint --prompt "<scene>"
uv run synth-generate --name m4-backgrounds --mode inpaint \
    --prompt "<scene 1>" --prompt "<scene 2>" --prompt "<scene 3>"
uv run synth-generate --name m4-outpaint-1.5 --mode outpaint --outpaint-scale 1.5 \
    --prompt "<your prompt>"

Both modes need --prompt: describe the whole photo, the cat and what surrounds it. With several prompts, the images cycle through them.

As in Meeting 3, both modes start from the training photos themselves and print which ones; --source limits them to the photos you choose.

The mask comes from the Oxford-IIIT Pet annotations, which mark the pet, the background, and the fur edge in every photo. On the contact sheet, the red area is what the model may repaint. In a real project you would have to make such masks yourself, for example with a segmentation model.

Do This

  1. Inpaint with one background prompt and with several, and check on the contact sheets that the cat is unchanged.
  2. Outpaint at two or more scales.
  3. Train every pool with five seeds and compare it with all earlier methods.
  4. Look for shortcuts. If all 55 added images share one background, the classifier may learn the background instead of the cat. Do your results show this?
  5. Choose and test. Choose the best inpainted and the best outpainted run on the validation results, write the choice down, test both with synth-test, and compare their test values with those of Meetings 1–3.

Examples

Two training photos after each edit: the mask (red), the background inpainted with a sofa prompt and with another scene prompt, and the photo outpainted at three scales. The cat’s pixels stay the same in every inpainted image.

Two Egyptian Mau training photos, each with its background mask, inpainted backgrounds, and outpainted versions at three scales

Editing has its own typical failures. Look for them on your contact sheets and describe how often they occur.

Typical failures: the coat pattern spreading into an inpainted background, a second cat, an outpainted photo turned into a picture or put inside a frame, and an image blocked by the safety checker

One of the 15 training photos, Egyptian_Mau_165, has an annotation that marks no cat; the repository skips it for inpainting, so inpainted pools start from 14 photos.

Checkpoint

Before the final meeting, you should be able to show:

  • inpainted pools with one and with several background prompts and at least two outpainted pools, with their contact sheets;
  • their results compared with all earlier methods; and
  • the test results of your best inpainted and outpainted runs.

Useful Materials