Yes, you can do that but it is highly advised that you don't as there is the risk to contaminate your eyes with bacteria.
Can I wear contacts while swimming?
Yes but it isn't recommended.
Of course you can. I've been wearing contact lenses while swimming for nearly 6 years now. However, I don't do swimming as a sport, but only during the summer, so it's not an everyday kind of activity or frequent thing. Either way, there are things you need to pay attention to because the risks are high. I have to put them on when I go to the beach because my myopia is quite severe. If yours is not then maybe I would reconsider the whole thing.
If you swim regularly, think about getting daily contact lenses. Mine are monthly but as soon as I get home I clean them and put my glasses on. If you practice daily though, monthlies won't be good for you.
If you got to the beach with friends, I'd also suggest to tell them beforehand. Sometimes when playing they might put you under water unexpectedly or throw water at you while you have your eyes open. I always tell my friends and they pay attention.
So, it isn't recommended for a lot of reasons ( the internet is full of stories of people who got blind or got bad infections) but if you pay a good amount of attention and care you shouldn't have any major issues.