How can a system protect against password guessing attacks, and how do banks stop repeated PIN guessing on stolen cards?

Depends on where in the world you are and how the ATM in that specific location is setup, most ATMs will lock you out, 3-tries and thats it. 24-hour lock. In some instances, the ATM will retain the card after say 3 to 5 tries and ask that you inquire within the branch (or nearest branch) of the bank that owns that ATM to have your card retrieved.