"Feminist" pro-sex work advocates will tell you that sex work is a question of what a woman wants to do with her body and that as something other than a woman you have nothing to offer to the conversation. They are wrong. Sex work is heinous exploitation, and as a human being, you have the right and obligation to vociferously condemn heinous exploitation.
Let's re-frame: Do you think your mom is more than a breathing sex toy? If you're straight, do you think your female significant other is, in fact, human? Would you say that you generally like women and feel bad about the fact that they are being oppressed? Then you should feel equally bad about the existence of the sex industry, an important aspect of this oppression. Do you advocate, fight for, organize for, philosophize about, and/or otherwise make a mainstay of your core beliefs the proper treatment of poor people, working-class people, and members of historically oppressed races without actually being all three? Then you should feel just fine about doing the same for women without actually being one.
This is not to say that you should rush head-on into anti-sex work circles and try to take a glamorous leading position doing something chivalrous. That would make you part of the patriarchal problem, a paternalistic protector seeking glory through the shielding of the delicate ladies. But there's no reason why you shouldn't have an opinion, share that opinion with other people, and work to make use of sex workers unacceptable among the men you know. There's no reason you shouldn't organize at all. Basically, being supportive of anti-sex work causes without trying to take them over them is pretty much an extension of and exactly like being a feminist-supporter.
(If you don't know what "feminist-supportive" and "pro-feminist" mean, you have some reading to do.)