Real-Data Baselines

Add Egyptian Mau training examples by oversampling and augmentation, and measure what the classifier gains.

Course overview · Next

Bring Codex or Claude Code if you already use one. Otherwise, install Antigravity CLI before the meeting.

Your Goal

A classifier trained on eight cat breeds sees only 15 photos of Egyptian Mau and 70 of every other breed. Measure how well it recognises Egyptian Mau, and how much two classical fixes help: repeating the 15 photos (oversampling) and adding randomly transformed copies of them (augmentation). These are the baselines that every synthetic-data method in later meetings has to beat.

Dataset protocol: one fixed split, imbalance only in training data, decisions on validation data, and test data for the chosen runs at the end of each meeting

What the Repository Gives You

Create your project from the starter repository. It provides:

  • a fixed split of Oxford-IIIT Pet in data/split.csv, the same for everyone;
  • synth-train, which fine-tunes EfficientNet-B0 pretrained on ImageNet, evaluates it on the validation photos once for each of the seeds 0–4, and saves the metrics of every seed in runs/<name>/results.json; and
  • synth-test, which evaluates the saved models of a run on the test photos and saves their metrics in runs/<name>/test-results.json.

Making tables and plots from these files is part of your work; the repository’s README shows what they contain.

Data role Each other breed Egyptian Mau
Training 70 15
Validation 20 20
Test (chosen runs only) 30 30

Every method adds the same number of Egyptian Mau training examples: 55, so the minority class reaches the 70 photos of the other breeds. Only the way these 55 examples are made changes from method to method.

Do This

  1. Set up the repository and run uv run synth-data, which downloads the photos and checks the split.
  2. Write oversample() in src/synthdata/balance.py. It chooses which of the 15 photos to repeat, so that the 55 extra examples use the photos as evenly as possible.
  3. Write augment(). It returns a randomly transformed copy of one photo, for example a crop, flip, rotation, or colour change from torchvision.transforms. The cat must stay recognisable.

  4. Run the three baselines:

    uv run synth-train --name m1-real --method real
uv run synth-train --name m1-oversample --method oversample
uv run synth-train --name m1-augment --method augment
  5. Compare them in your own table and plot made from the three results.json files, showing every seed, not only the mean.
  6. Change augment() once, for example to stronger or weaker transforms, train it as a new run, and explain the difference.

  7. Test the three baselines. Once you have written down what the validation results show and which version of augment() you keep, test the three runs and report their test values in EXPERIMENTS.md. Every later meeting compares with them.

    uv run synth-test --run m1-real
uv run synth-test --run m1-oversample
uv run synth-test --run m1-augment

How to Read the Results

  • Egyptian Mau recall is the share of the 20 Egyptian Mau validation photos that the classifier recognises. It is the main metric.
  • Macro-F1 is the F1 score averaged over all eight breeds. It shows whether a method helps Egyptian Mau by harming the other breeds.
  • Five seeds train five models that differ only in their random start and data order. Their spread shows how much a result changes by chance; a difference smaller than that spread is not evidence.
  • Validation and test have different jobs. You compare runs and make your choices on the validation photos; the test photos then measure only the runs you chose. A test result summarises the meeting and does not change a choice.

Examples

The top row shows one training photo of each breed with the number of training photos it has: the classifier sees 70 photos of every breed except Egyptian Mau, which has 15. The bottom row shows what the two classical fixes add for one Egyptian Mau photo. Oversampling adds exact copies, so the classifier sees the same pixels again; augmentation adds transformed copies, here cropped, flipped, rotated, and recoloured, so every copy is a little different.

One training photo of each of the eight breeds with its photo count, then one Egyptian Mau photo with three oversampled copies and three augmented copies

Checkpoint

Before the next meeting, you should be able to show:

  • your oversample() and augment() code, and how you checked that it does what you intended;
  • the three baseline runs with five seeds each;
  • the table and plot in EXPERIMENTS.md, with a paragraph on what they show;
  • your augmentation choices, the variant you tried, and its effect; and
  • the test results of the three baselines.

Useful Materials