Its the key to your wallets. I have the same one by the way.
with out that device plugged in to the PC and the 4 digit access code you cannot access the wallets.
Obviously, If you go away without the ledger, you wont be able to access the accounts.
I understand if you lose the ledger, there is a way to get access again with the 24 word recovery sheet, im not sure how easy it is though.