What costs do banks incur when handling a deposited check that bounces?

A few pennies really. Almost all the cheque processing is automated. It is nothing more than scanning/processing of physical cheques, and in electronic cheques, simply an authorization, etc.

All in all, I think it would cost an average financial institution less than 30 cents to process it all (even if the cheque gets bounced).