Be true to yourself: If you don’t want to do it, don’t do it.
They may judge you for it, let them. What does it matter if they judge you? If they are your friends, they’ll accept you anyway. If they are rational people, they’ll accept you anyway. If someone likes oranges but not apples, are you going to peer pressure them into eating apples? Why? They are entitled to their own preferences. Whatever you’re being pressured to do is the same thing: if you don’t want to, don’t do it.
However, if everyone there is doing it, and they aren’t welcoming to you, then perhaps you don’t belong there. Don’t mistake this for a lack of self worth (see Common Misconceptions about Self Worth). If you’re on a basketball court with people playing basketball, don’t read a book: get off the court. The time and place sets the activity: come back in an hour when there’s no-one playing basketball, and you are free to read a book on the court.
At the same time, don’t invent rejection: if there is no reason to believe that you aren’t wanted there, then it’s fine for you to stay! The only question is if you want to. Here’s an article on how to avoid Fears Due to Imagination