So was tv, the internet, or even electricity a few hundred years ago.
But the components that were assembled to create said things did exist. There has always been potential tv, internet and electricity. Luckily, we have gained the knowledge to invent these things. I'm sure there is potential do invent and do many things we haven't even thought of yet. There is no such potential for time travel.
It's a fairly safe bet that when man was running away from sabre tooth tigers he didn't foresee the potential to invent the internet - even though the components for it existed. The thing that might make time travel possible probably exists now, we just haven't discovered how to do it - tis no different.
Having said that, I highly doubt we'll ever invent time travel but let's face it, whaddo we know!