Is it fair to not work on weekends even if you manager calls you?

Yes

Just take a look at the risk vs reward

It’s the manager’s incompetence if you’ve got to working most weekends.

Fair, yes. Fallout for saying "no" - yes, surely.