I recently flushed my heater core and noticed that the upper and lower radiator hoses are collapsing.
I'm guessing that's due to air in the system.
So, I have been adding coolant and squeezing all the radiator hoses and have been checking them before I go for a drive.
Is there any other reason why the hoses would collapse other than air in the system?

Either they're old and soft and in need of replacement, or the radiator cap is causing a vacuum as the coolant cools off.

If the radiator cap is causing a vacuum should I get a new radiator cap? Would that fix the issue?

Well they're cheap, so it can't hurt. I'd usually say you need to pressure test it, which can be done at the parts store.

Watch a youtube video on how radiator caps work. Look for one with an animation. It'll explain better than I can.

Get the correct pressure rating for the cap. You will need to check an owner's manual or such for the correct pressure rating The old renix XJs dont have a normal cap, but post 1990 XJs do. While at the parts store, ask to squeeze the Charmin, errrr ......rather squeeze a new radiator hose. Make a mental note how soft it is. If yours are much softer when cold, then they maybe near end of life.

the cap does two things, at high pressure (hot engine) it vents the liquid to the overflow bottle. when at low pressure (cold engine) it opens to allow the vacuum caused by the temperature dropping to suck liquid from the overflow bottle back to the cooling loop. In your case it may not be sucking back the liquid, which causes a vacuum to form in the cooling loop, which can crush hoses, more so with old soft hoses. Often the rubber gaskets in the cap go bad, preventing suck back.

good luck, keep us updated.