Here's my response to the plea for suggestions:
If the father has the right to see the child, there is nothing to do about that unless he just doesn't want to and will forfeit the weekend and spare you the heartache. The good news is that the child doesn't spend the majority of his time with this bad influence. And unless dad is willing to change, you are stuck just trying to do the best with what you've got.
That sweet little guy will need extra love and guidance to make up for the damage and emotional drain that daddy puts on him. Make sure that when he is with you, he is given absolute consistency and structure, as this will give him security. Model good communication and language for him and do not let him get away with inappropriate language when with you. Even though it's not his fault and he's getting mixed messages, he will eventually learn what is okay at one house versus another. He needs at least one caregiver modeling love and appropriate guidance.
It will be difficult for everyone involved for a long time, as you will just have to keep undoing what he is learning on weekends, but he will eventually grow up and realize who it is that provides him with true love and what he needs. I know that is not the answer you want to hear, but unless visitation can be halted, you are stuck. Children will always crave attention and love from their parents, but if you give him the extra love and guidance that he needs, he WILL wake up one day and figure out who the bad influence is and how much it hurts him. At that time, hopefully he will have enough confidence and love of himself that you can walk through it together and deal with it in the most healthy way possible.
Do your best and hang in there. Remember not to spoil to try and make up for any guilt that he's going through a rough time on his weekends away. Be a rock for him - a loving, consistent, and solid guidance. Remember his tug of war with the mixed messages and be patient with your guidance.