Is it safe to remove part of the drywall behind plumbing to eliminate noise?

We've had a noise when water is draining from upstairs bathroom since we moved in 13 years ago. Now that we have the wall open on one side for other reasons I can see that the PVC drain isn't parallel to the wall and is rubbing the drywall behind in the section in the photo, specifically the wider joint on the bottom of the Y, possibly the upper part of the Y as well.

I know that the drywall (originally 1950s) is very thick, (probably 3/4 of an inch). I've wondering if I can somehow remove part of the thickness of the drywall to put an air gap. My other option is to cut the drywall out and then replace with thinner drywall but I'm not sure how to attach it, I usually use wood pieces attached to the studs for a nailing surface but there isn't really room.

EDIT: I've been working with a screwdriver on the left side and removing material. I can't get the screwdriver in on the right side well because of the stud. Any other tools I might use?