I had the exact same thought process when I started out, early in my career. I knew from the beginning that I can work better than some folks from fancy colleges. Not all of them are good as we perceive them to be. But now after few years of experience, I realise this is all about credibility.
Put yourself in shoes of hiring manager. They need someone who can do great work, isn't afraid of long working hours, has the ability to persevere. He has 2 candidates in front of him. 1st is from fancy college, 2nd is from a local college. Who should he hire? Should he hire someone who has already worked hard for entrance exams, got selected in fanciest college by being extra competitive or should he take a chance on some unknown college. Basically, good colleges, company name kind of assures credibility of the skill. People who are from fancy colleges but have no skill, eventually lag behind in careers. That's the reality. Better make peace with it.