During the late stages of a star's life, what causes the star to expand into a red giant/supergiant, then explode into a nebula or supernova?
The age of the star.
As the star reaches a certain point in its life cycle, it runs out of hydrogen, which is basically the star's fuel for nuclear fusion.
(Nuclear fusion is the process of combining smaller atoms into larger ones. In this case, hydrogen is fused into helium, which causes the sun to release so much energy. The heat doesn't cause the star to expand because of the pressure on the core)
As it runs out of hydrogen, the pressure on the star's core decreases, and it starts increasing in volume. As a result, the star also cools down and starts glowing less brightly, forming a red giant or supergiant.
If the star is massive enough, it collapses on it's self because the core can no longer support the outer layers and explodes into a nebula. If the star is too massive, however, the collapse causes the matter present to contract into an extremely small volume, with an extremely large mass, and an infinitely large density.
What happens in the late stages of a star's life depends on the mass of the star.
The mass of a star determines its fate. The critical mass is whether it's mass is less than or more than eight solar masses.
Smaller stars, less than eight solar masses, have relatively cool cores. The fuse Hydrogen into helium during their main sequence phase. Once the Hydrogen in the core is depleted the core is mainly Helium. The core isn't hot enough to start helium fusion. At this point fusion stops and the core starts to collapse under gravity.
As the core collapses it heats up. It gets hot enough for Hydrogen fusion to start in the layer surrounding the core. This causes the outer layer of the star to expand into a red giant.
If the core gets hot enough, Helium fusion will start. It can in fact go through a second red giant stage. Ultimately the core is mainly Carbon and Oxygen. No further fusion can take place and the core collapses into a white dwarf. The outer layers expand to form a nebula.
When larger stars run out of Hydrogen; they seamlessly start Helium fusion. After Helium is depleted Carbon fusion starts. this progresses making heavier elements until the core is mainly Iron. No further fusion is possible and the core collapses.
The iron core collapses into a neutron star or even a black hole. The collapse releases a lot of energy which blows away the outer layers of the star in a supernova explosion.