When I drive my 1999 Sport up the mountains, the A/C switches from vent to the window defroster on its own. Eventually it will go back to normal and return to the vents. Driving in town, it doesn't do it. It just does it by going in higher elevation. Should I worry about this?

 

You have a vacuum leak. My XJ does the same thing.

 

So will it damage anything else if not fixed? I go up the mountain maybe twice a month to go shooting.

 

It won't damage anything, but things that run off that vacuum may not work correctly and it will likely get progressively worse. The vacuum lines control what vents your heater or ac will come out of and also cruise control if you have it.

When you are going up a steep hill, your engine is working harder and its harder for it to maintain a vacuum probably because of a leak somewhere in the lines.

 

Check the vacuum ball in your front bumper and the two check valves in the vacuum system