One cause of dizziness when swimming is breathing too little or too much. I think you would know if you were breathing too little and were low on oxygen, but some people breathe too much and this can make you dizzy. If you aren't going too hard and are breathing every 2 strokes, you are possibly over breathing. As you speed up you will need more oxygen and every 2 strokes may be okay. When I swim, I try to breathe the exact same amount as I would if I weren't under water.
Since you move your head when you breathe this can also contribute to dizziness. Try to move your head as little as possible, if you are in a pool look at the line and don't move your head. You may also want to try one of those snorkels that go in front of your face that you can swim with because then you don't have to move your head at all!
Of course breathing may not be the cause of it at all for you, but I thought it was worth a mention!
