Recipe for inter-operating a C# Symmetric Encryption code with OpenSSL

Interoperability is not a difficult thing to achieve. Yet, it is not an easy thing to get either- predominantly because the relevant information are widely fragmented and even Google fails to compact all of them together- leaving you rummaging through a pile of web pages to fish out really useful information from them. I am posting this for my own help- may be in future I will need to come back here to look at this again to see what I did!

Problem # 1: Quirks of Hex key in OpenSSL

I needed to write some C# code which will take in a plain text file and run some Symmetric Encryption routine over them using a given Initialization Vector (IV) and a Secret Key (SK). The output will be some encrypted binary data which should be decoded by the OpenSSL tool. Basically, I wanted following OpenSSL command line to work for me always:

$openssl enc [-aes-128-cbc] -in <CipherData.bin> -out <ClearData.txt> -d -K <SomeSecretKey> -iv <SomeIV>

Here <CipherData.bin> is the encrypted file generated by C#/.NET crypto routine. My aim is to support these algorithms: AES, 3DES, RC2 and DES and at least support following chaining modes: CBC, ECB and CFB.

With -–K (and -iv) option, OpenSSL expects only Hexadecimal secret key (and IV) string whose valid size differ for different algorithms employed. The input hex string is converted into a byte array and used as SK (and IV). The interesting part is that OpenSSL and .NET crypto behave differently if you do not supply a valid length hex stream against these data. While, OpenSSL intelligently (and silently) pads the byte array with trailing 0s to make up for the shortfall in “legal” size or truncate it if the same goes beyond the adequate size, the C#/.NET crypto code dies with an exception, leaving you puzzled- what went wrong? Sometimes, to baffle us more, a secret key which had worked in C#/.Net seems to give you "“bad crypto"” in OpenSSL and vice versa.

The crux is how you encode (and decode) a Hex string into byte[] for SK and IV. Basically the point to remember is that a simple Hex text like "“CAFEBABE"” is 8 ASCII character (8 bytes array) but, becomes 4 bytes array when converted to Hex encoding. I have a sample HexUtil C# class which illustrates these things... wait for next post.

Problem # 2: Line breaks in base64 encoding

By default (-a or -base64 switch) OpenSSL inserts/expects a line break (new line character) after every 64th character printed in the encoded/decoded stream. But C#/.NET Libraries v1.1 does not insert a line break automatically in the generated base64 encoded string. Hence, there is some incompatibility between OpenSSL and C# Base64 converter. I have read that in .NET v2.0, Microsoft has added an additional API to insert line breaks in the generated Base64 strings [add link ?]. Meanwhile, I figured out that there is a command line switch in OpenSSL to disable the default line break behavior in the tool. I used following OpenSSL command to successfully decode the Base64 texts generated by C# code:

$openssl enc -a -A -d -in <B64EncryptedData.txt> -out <B64EncryptedData.txt.out>

Note the extra -A switch in the command line!