For training, wear the goggles as loose as possible without having them leak.  For races, you may want to tighten them more in case of wave action, fists in the face, etc.  I am no doctor, but your problem is quite common, and I don't think it's a big deal.

I don't think you will be doing any damage.

I became allergic to one of my pairs of goggles, I ended up with the sunburnt panda look for a couple of days.

I find its trial and error finding the best googles.  What works for one person may not work for another.

I personally find Zoggs or Speedo work best for me.

I recommend the Barracuda foam goggle. They are expensive (around $35) but worth it, and the package says they were developed especially for distance swimmers. Besides a leak-proof seal, they are comfortable and leave no imprints on your face. My last pair lasted over 10-years. They are available in different lense shapes that fit faces differently, so you should try on the different shapes at the store for the best fit.

One warning though.  I can only use the white foam versions. The black foam gives me severe eyelid dermatitis, and there is a medical paper abstract on teh interweb that says the same thing.

You might try out swedes (or Swedish racing goggles). I swear by them.

They fit right into your eye-socket and  do not need to be tight. There

is no rubber seal.

The good thing about these is that they are very very inexpensive

($4.00) . I have tried $40 baracuda's speedo's... you name it. They all

leaked... or hurt my eyes.. but the swedes never let me down. I swim

60min 4-5 times a week.

If you try them... I recommend using dental floss rather than the string they sell

with them for the nose piece. Run it through a few times. The string

stretches out and causes the goggles to leak after awhile. Floss holds.

They are not for everyone, but at $4.00 it may be worth trying out. They have them online and at swim meets.  I have never seen them in chain sports stores.