This is a very hard decision that only you can answer, preferably after much prayer. Bullies are cowards. They strike only when it benefits them and when they're confident they can get away with it.
The real reason any member of a parish is abused is because of weakness, on the part of the pastor.
This might not be totally his fault. He could be new and still finding his way. He could be greatly outnumbered. Bullies are master manipulators, so he's probably operating without all of the facts. However, it's his job to ensure the wolves are kept away from the sheep, especially the most vulnerable ones.
So should you tell your pastor what's happening? Think long and hard before doing this. Your bully (or bullies) have, most likely, cultivated a strong connection with the pastor. So he may not believe these individuals are capable of such despicable acts. He may then doubt your credibility. It's possible the person trying to run you out of church has also made some preemptive moves, by trying to discredit you before you complain.
The difficulty with spiritual bullying is that there aren't a lot of options, as there are a limited number of churches in any given geographical region. Even so, you might want to think about finding another one, so you can focus on God and not on the drama.
If you still want to bring this to your pastor's attention, you'll be on more secure footing if you already have an escape plan. That way, if he's not supportive, you can move on.
Flickr photo by unkreatives