Swimming with your contact lenses, is possible but very risky. The health risk associated is merely dependent upon the body of water you are in. When swimming in lakes and rivers, there is a huge possibility for a microorganism called acanthamoeba to adhere to your lens, causing infection and inflammation of your cornea (acanthamoeba keratitis).
Swimming in pools and ocean waters on the other hand, decreases your susceptibility to microbial infection, but other problems may evolve. In the pool--eye irritation is possible when chlorine sticks on the surface of your lens; in the ocean--your contact lenses can be dislodged when encountered by large waves.