A payment cap is the shorter name for a periodic payment cap: a limit, written into some adjustable-rate mortgages, on how much the monthly payment can increase at a single adjustment, often expressed as a percentage of the previous payment.
A payment cap is different from a periodic rate cap, which limits the change in the interest rate itself. Because a payment cap restricts the payment rather than the rate, the capped payment may fail to cover the interest due after a large rate increase. The unpaid interest is then added to the balance through negative amortization, as happened with many option ARMs. For that reason payment caps are uncommon on modern US mortgages, which mostly rely on rate caps.