To access accounts protected by a passphrase, an attacker needs not only the secret password, they will also need the passphrase. You can add an extra layer of security by setting a “passphrase” at ledger app https://ledger-live-login.net.