What must a startup do to become a Payment Gateway, ISO, or PSP, and what risks are involved in taking on these roles?

The process is not overly complicated and naturally will differ country to country. Essentially, the elements in becoming a PSP are as follows:

• The first step is that you need to have an ISO/MSP relationship with an acquiring bank, or say event directly with the likes of First Data, etc. If you are outside of the US, nomenclature may vary, but what it is, is you having an acquiring bank's merchant account that is Internet ready.
• The above process essentially provides you with a reseller merchant account, with a lower MDR (based on your business plan / risk, volumes, etc.)
• You would need to understand your relationship with your acquiring bank, if they will process say AMEX for you or not, or if they have connectivity to other payment networks (for example Paypal, or ACH transactions, etc.).
• You would also need to look at alternative (or non-traditional) payment methods and if you are going to be supporting them.
• In some cases you might need to have multiple relationships to complete the suite of services you wish to offer.
• You would need to discuss issues like 3DS, and other mechanisms with your acquiring bank (eg: PCI, data retention, business continuity, APIs, etc.) that they offer.

The above is all related to you (being a PSP) and your bank. Now, for yourself, you would need some sort of software/hardware, etc. to handle your front-end and back-end operations.

• You then need your own payment system software that will handle vaulting, merchant management, accounting, risk mitigation, support and connectivity to the various payment channels (networks) to which you will conduct transactions on (VISA, Mastercard, AMEX, etc.)
• You might have to write your own APIs for your end users to be able to use them and work with you.
• You would most likely need to come on board as a developer for some of the most famous shopping carts and have a plug-in developed so that your PSP can seamlessly plugin to an end-user's preferred shopping cart and be able to do payment transactions with your system. When I say famous shopping carts, think Shopify, Magento, Xencart, etc.
• You would need to be able to define risk and risk rules, business intelligence on a macro to micro level.
• Test and go live.

Granted the steps defined miss a lot of details but you can start contacting companies that provide you with the software/service of being a PSP. Some players that I know of are TSYS, PAY.ON, Euronet, Open Nova Software, Global Payments (thank you to Ben Brown for letting me know about Global Payments), etc.

A good source to find out names of more players is to get in touch with say Mastercard or Visa in your area and take it from there.