Could a compliant cross-border bill payment service (for example, from the U.S. to Mexico or Indonesia) operate without remittance licenses?

If you're paying local bills in the beneficiary country from the US, this is not possible without registering with FinCEN as well as obtaining a Money Transmitter Licenses in the States from which you are on-boarding customers.

Having said that, there are a couple of companies that could aid you with the license and regulatory aspect if convinced of your business model.