My manager is pissed off

Sometime back my manager assigned a task to about 2-3 of us which dint really have a definitive timeline, but was recurring in nature based on the request coming in. Since it was request based my assumption was that when there is a request I will be called and that is when I will need to do the task. Recently my manager came back to me saying that I wasn’t performing well and that he does not see goal oriented motive from my end. The reasoning that he had for this was that there was a request that came up and the other person fulfilled it because they were always available and on top of it. Though I tried to explain that I was waiting on requests to come in at the end I gave in and told him I will be more proactive in pursuing the task. Post a few days when I was doing some work for that task he comes to me and says that I wasn’t even preparing for this task earlier, but now that it got escalated and he has pointed it to me I’m starting to prepare now. I got pissed off and told him that is not the case.

Now I’m contemplating if I should give up ownership of this task altogether for the sake of my dignity & mental peace OR Continue to take it up just to make sure that the point that he is trying to prove (that I have started only now and I’m not good enough) doesn’t come true?