I create a 256 byte secret AES key or such. I encrypt it with one of the four RSA public keys available on the card. Then I can store the resulting secret alongside of the encrypted drive or whatever else needs offline authentication. Only the private key embedded in the smart card can then be used to derive that secret given that a user is able to authenticate with the smart card. I put together a proof of concept that seems to work. Now it's a matter of figuring out of this is at all usable in practice.
SECRET="256byte.dat" # Sample secret data that must be exactly 256 bytes.
echo "== Extracting the public certificate '$KID' from the smart card"
echo "== Extracting the public key from the certificate"
echo "== Encrypting a secret the smart card's public key =="
echo "== Decrypting with the smart card (sign with card's private key)"
echo "== Comparing original secret with a decrypted one"
echo "== Cleaning up =="