What is the difference between deposit-taking and non-deposit-taking financial institutions with examples?

A good example of a non-deposit taking financial institution would be say MoneyGram or Western Union or your USPS (United States Postal Service). Of all these three institutions can accept cash/money and forward it - i.e. do a delivery of money, but none of them are a deposit-taking institution, i.e. deposit related banking services are offered by neither of these three institutions.

Compare this with any retail bank, that is a deposit-taking institution.