This is a dick manager move from a dick manager.
Why he said this - it's a manipulative way for him to get more work and attention from you. If others are praising you and he isn't seeing your 'seriousness', it just shows that his insecurity and ego needs occasional massaging. But he's saying this to you BECAUSE he thinks you can do something about it, otherwise he wouldn't bother. So it's a douchey way to ask you to put more time and attention to showcase what you're doing to him.
Any point of staying - him telling you this is like an invitation to focus on making him happier. That means there is scope to continue to grow, but it will mean additional effort for stupid outcomes (boss liking you). And if it works, it'll just put him in the headspace where he'll know that passive aggressive whining will make you respond in a way that benefits him.
I'd suggest you start exploring other options while keeping yourself engaged here so you aren't left in a bad place in this job market.
I'm sorry this is happening to you, and I hope you have better luck in your career soon.