Need help regarding office culture

Whenever any kind of work is assigned to me I hesitate to say no to my senior and manager. This is hampering my personal career graph as I’m java developer but work assigned to me is not related to it.

I keep doing assign work although I don’t want to do.

They keep increasing my responsibilities in different area which is not helpful to me. I think they get to know that I’m easy target to put burden with no hikes.

How to say no and how to get courage to say no .