What fees do banks pay to connect to ATM networks or per ATM transaction?

It depends which part of the world you are in and how the interbank network connectivity arrangements are handled. Most banks pay a membership fees to connect to a network (this is a flat fee). Sometimes banks prefer to connect to multiple networks for sake of redundancy.

Almost all the banks pay a per transfer fees. The fees is highly depended on how and where the ATM network is being accessed. Universally the fees is anywhere from 5 cents to 10 cents - and hovers in between. ATM end-points charge typically anywhere from 50 cents to $1 for each transactions. Cross-border or cross-network fees can be as high as $2.50 per transaction.

This fees is paid by the person who is conducting the transaction on the network. The Interbank network operator then settled with the transacted parties on a daily/weekly/monthly basis (again depended on where in the world you are operating).

The largest Interbank network is PLUS (by VISA): Plus (interbank network) followed by Cirrus: Cirrus (interbank network) - they each have over 1+ Million ATMs connected to their network.

Other notable players are: Armed Forces Financial Network (AFFN), NYCE, Euronet, Pulse, Link (UK), Interac (Canada), etc.