A 5xx code is permanent and a 4xx code is temporary, and that single digit decides what happens next. uChecker has a side-by-side breakdown of hard and soft bounces with the common codes on each side.

A permanent failure should leave your list today. A temporary one, like a full mailbox, a busy server or greylisting, deserves a few retries before you give up on it.