1. num_workers: what it is

• num_workers = 0: everything runs in the main process (data loading + preprocessing + collation).

• num_workers > 0: multiple worker processes load/decode/CPU-transform data and build batches in parallel, then send ready batches back to the main process.

2. What a worker actually does

1. Get a “task” from an index queue (usually a list of indices, e.g., [42, 17, 88, ...])

2. For each index, run dataset.__getitem__(idx) to fetch one sample

3. Run collate_fn(samples) to combine samples into a batch

4. Put the completed batch into an output queue for the main process to consume

• __getitem__ and collate_fn run inside worker processes

• the main process mainly: dispatch indices → receive batches → move to GPU → forward/backward

3. Who decides “how many indices per task”?

Default behavior (no custom batch_sampler)

• With batch_size = B, DataLoader groups indices into lists of length B

• The last batch:
• drop_last = False (default): may be a smaller batch if not enough samples remain
• drop_last = True: the last incomplete batch is dropped

Custom batch_sampler

• If you provide a batch_sampler, it fully controls how indices are grouped

• Batch sizes can be variable (whatever your sampler emits)

4. prefetch_factor: worker prefetching

• prefetch_factor ≈ how many future batches per worker are kept queued/processing

• Default: prefetch_factor = 2

• In-flight batches ≈ num_workers × prefetch_factor

• num_workers = 8, prefetch_factor = 2 → ~16 batches concurrently being prepared or waiting

Why it helps

• Better pipeline overlap: while the GPU trains on batch k, workers prepare batch k+1, k+2, ...

Why it can hurt

• Higher CPU RAM usage:
• Roughly: batch_memory × num_workers × prefetch_factor (plus Python/object overhead, and optionally pinned memory)

5. CPU memory vs GPU memory (important)

• workers build batches on the host and place them into queues

• batches only consume GPU VRAM after the main process moves them to CUDA (e.g., .to(device))

6. persistent_workers: worker lifetime across epochs

Without persistent_workers=True (default False)

• End of epoch → workers shut down

• Next epoch → workers start again

• Costs: process startup/teardown + re-initialization overhead (often noticeable when epochs are short)

With persistent_workers=True

• Workers are created once and reused across epochs

• Benefits:
• faster epoch transitions
• more stable performance
• avoids repeated worker initialization overhead

• If num_workers > 0 and you train for multiple epochs with a mostly static dataset, persistent_workers=True is usually a win.