If you created your ssh keys with default rsa settings, their format is not compatible in openssl.
To encrypt and decrpt messages in openssl, you need your public key and private key in pem format, and start like this:
-----BEGIN PUBLIC KEY-----
or
-----BEGIN RSA PRIVATE KEY-----
Examples: