Manager thinks my leave starts later but I am travelling tonight. How do I handle this

We are visiting the USA for vacation, but during our initial discussions my manager did not allow leaves for more than twenty days. Later, I made a plan saying I would take only three weeks of leave, for which he said he got me an exception and asked me to use it judiciously. But we have already booked our flights.

The issue is that I will be travelling tonight itself, but according to my manager my leave starts from the 26th. My teammates are aware of this, and they are saying I should admit it to my manager and say I got confused about the dates. I think that would be foolish now. How can I cover this up? Please help. Should I say I have some last minute errands and that my availability will be patchy?