What is the accounting system that tracks a country’s international financial transactions called?

FX or Foreign Exchange.

In most banks and the central bank, this is accounting head (or chart of accounts) under which foreign-exchange related transactions are recorded. This could be remittances, business to business payments, letter of credit, loans, current account payments, foreign exchange reserves, etc.

All are recorded under foreign exchange account.