Email encryption about xstreader HOT 43 CLOSED

This would be a good feature to have, but unfortunately, the Microsoft documentation of the PST file format does not describe how encrypted emails are stored.

There are commercial tools available that can decrypt at least some encrypted email formats, so I guess they must have reverse engineered something, but I can't find anything on the Internet that reveals the secret.

So, for the moment, I won't be supporting decrypting encrypted emails.

Dijji

from xstreader.

Thanks for getting back to me. I'm going to look into this a bit and see if I can find a way to do it..

from xstreader.

Please reopen if you find anything out

from xstreader.

Sorry to take so long getting back to you on this. I had some life events happen in the meantime. I've come up with some C# POC code for decrypting the P7M files that are included in encrypted emails. This example uses certs already imported to your cert store but I think I can probably get it working with individual certs if that functionality was wanted. What I'm thinking is that you would check if a P7M file is included as an attachment to a message and if so decrypt and display it.

I'm not sure where to start on integrating this into XstReader but if you point me in the right direction I can take a crack at it.

Also, if I wasn't clear, my understanding is that encrypted messages are just attached as P7M files on normal messages thats the behaivor I have experienced in xstreader and other pst/ost readers.

using System;
using System.IO;
using System.Security.Cryptography.Pkcs;
using System.Security.Cryptography.X509Certificates;

namespace p7mdecrypt
{
    class Program
    {
        static void Main(string[] args)
        {
            X509Store store = new X509Store("MY", StoreLocation.CurrentUser);
            store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);

            X509Certificate2Collection collection = (X509Certificate2Collection)store.Certificates;
            X509Certificate2Collection fcollection = (X509Certificate2Collection)collection.Find(X509FindType.FindByTimeValid, DateTime.Now, false);


            EnvelopedCms ec = new EnvelopedCms();
            ec.Decode(File.ReadAllBytes(@"C:\testData\smime.p7m"));
            ec.Decrypt(fcollection);
            byte[] decryptedData = ec.ContentInfo.Content;
            var decryptedMessage = System.Text.Encoding.Default.GetString(decryptedData);
        }
    }
}

from xstreader.

Life events. Tell me about it. Or on second thoughts, don't! Let's just close the door and get on with the fun stuff.

XstReader, happily, already has support for opening email attachments, which means that most of the needed framework is there.

The UI is driven by the IsEmail property on the Attachment class in view.cs. You will need to modify this to return true when the attachment is an email that you can decrypt. This will cause the attachment to present as an email and give the user the option to open it.

The starting point when the attachment is opened is OpenEmailAttachment in MainWindow.xaml.cs, which opens the attachment as a message and pushes the existing message down on a stack (in case the attached message contains attached messages, which is also supported).

The real work begins with OpenAttachedMessage in XstFile.cs, which has the job of creating a Message object, with all its ancillary objects such as Attachments, from Outlook's internal tables.

You would need to modify OpenAttachedMessage to detect the encrypted state and invoke your new unpacking code in, say, OpenAttachedEncryptedMessage, instead. If you present the body of the message as either plain text, HTML or RTF, the viewer will then display it.

I hope that this quick sketch helps. Please don't hesitate to ask any further questions - this would be a very cool extension.

Dijji

from xstreader.

Thanks for tips! I'll hit you up if I get stuck.

from xstreader.

After I have decrypted and decoded the encrypted email I'm left with something that looks like the below(I've trimmed the signature at the end for readability). Is there some functionality you already have built to parse the below into a message object or will I need to parse it myself? I was thinking about trying to copy your creation routine in OpenAttachedMessage and then manually modifying the fields I need to change but I thought I would ask first.

0..	*.H.÷
... .0....1.0	..+......0..	*.H.÷
... .$....lContent-Type: multipart/alternative;
	boundary="----=_NextPart_000_009C_01D40247.3F4447C0"

This is a multipart message in MIME format.

------=_NextPart_000_009C_01D40247.3F4447C0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Yes, I've been thinking the same thing.  The current method is slow.
 

 

From: Dole, Bob 
Sent: Tuesday, June 12, 2018 12:09 PM
To: Jenkins, Jim <[email protected]>
Cc: 
Subject: Example Email

 

Hi Jim, 

 

We're good with this proposal: 


 

Thanks, Bob

 

------=_NextPart_000_009C_01D40247.3F4447C0
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

....<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 =
(filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US =
link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1><p =
class=3DMsoNormal><span style=3D'color:#1F497D'>Yes, I've been thinking =
the same thing.  The current method is slow.<o:p></o:p></span>=
</p><p class=3DMsoNormal><span style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p>=
<p class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><div><div =
style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in =
0in 0in'><p class=3DMsoNormal><b>From:</b> Dole, Bob =
<br><b>Sent:</b> Tuesday, June 12, 2018 12:09 PM<br><b>To:</b> =
Jenkins, Jim &lt;[email protected]&gt;;<br><b>Cc:=
<br><b>Subject:</b> RE: Example =
email<o:p></o:p></p></div></div><p =
class=3DMsoNormal><o:p>&nbsp;</o:p></p><p class=3DMsoNormal><span =
style=3D'color:#1F4E79'>Hi Jim, <o:p></o:p></span></p><p =
class=3DMsoNormal><span =
style=3D'color:#1F4E79'><o:p>&nbsp;</o:p></span></p><p =
class=3DMsoNormal><span style=3D'color:#1F4E79'>We&#8217;re good with =
this proposal: <o:p></o:p></span></p><p =
class=3DMsoNormal><span style=3D'color:#1F4E79'>Thanks, =
Bob<o:p></o:p></span></p><p class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p></div></body></html>
------=_NextPart_000_009C_01D40247.3F4447C0--
<email signature block>

from xstreader.

That's great progress!

I don't have anything myself. Looking around on the Internet, open source support seems surprisingly patchy, but something like this might help: https://www.codeproject.com/Articles/5759/MIME-Message-Composer-Analyser

I haven't looked at it in any detail, but the API looks the right sort of thing for building a Message object from a MIME file. However, it does need somebody with enough C++ skills to make it accessible from C#.

Dijji

from xstreader.

I couldn't resist a quick go. I downloaded the code I sent a link to and wrapped it in a DLL. After a few code tweaks for things that have been deprecated for safety reasons over the years, it compiles quite happily into a 32 bit DLL. Here is the project:
MimeFormat.zip

I haven't tested it, or looked at converting it to Unicode or 64-bit. Have a look yourself, and see if the code offers a promising starting point.

from xstreader.

Thanks for the link. I've been looking at this as well https://sigparser.com/developers/email-parsing/parse-raw-email/ . Can you run me through what properties of a message object ( need to be set to get it to display in the viewer? Or if you could just run me through what they all do. Initially I'm just going to try and pull out the encrypted message's html and throw it onto the screen.
I've tried setting the below but that doesn't seem to be enough to get the message to display.

m.NativeBody = BodyType.HTML;
m.BodyHtml = "<some HTML message>"

One issue I have run into so far is that it appear most of the message metadata (to/from, time, etc) are not encrypted in the attached p7m file they are in the parent message. So I'm having to grab them from there.

Thanks for the help! I'm sure I'm asking some dumb questions but coming at this cold makes it a little harder :) .

from xstreader.

We'll probably want to consider encrypted messages with attachments at some point as well but I'm just trying to get the basics working for now.

from xstreader.

Agreed. Simple stuff first.

You cite the two properties, that I would regard as crucial. The only way I can see that this would miss is if you never arrive at ShowMessage (in the main window code). But since this is called by OpenEmailAttachment right after it has called your revised OpenAttachedMessage, this seems impossible. So I'm afraid this is just a question of stepping through ShowMessage to see where it goes astray.

I'm not sure about taking the other properties from the parent.: Isn't the attached message a full MIME message capable of having it's own? If it doesn't have any, you shouldn't show any.
If, on the other hand, the attachment is just there for its body, you should probably abandon the open attached mail approach and reverse the decrypted body into its parent, if you can assume that the existing body of the parent is not significant.

As for questions, you're doing fine. Blundering around in others people's code is firmly in the geek tradition (it is probably the preferred learning approach). The unspoken condition is that you will have made best efforts before asking a question, and you are passing that test comfortably.

Dijji

from xstreader.

so I decided to do some digging and go take a look at the RFC to see for sure what an S/MIME should have in it. Unfortunately it seems rather non specific

From section 3.1 https://tools.ietf.org/html/rfc3851 :

In order to protect outer, non-content related message headers (for
instance, the "Subject", "To", "From" and "CC" fields), the sending
client MAY wrap a full MIME message in a message/rfc822 wrapper in
order to apply S/MIME security services to these headers. It is up
to the receiving client to decide how to present these "inner"
headers along with the unprotected "outer" headers.

So then I looked to see if I could find any documentation on how exchange handled it and there wasn't a whole lot but I found this. https://docs.microsoft.com/en-us/previous-versions/tn-archive/aa995740(v=exchg.65)?redirectedfrom=MSDN

Message encryption makes the text of a message unreadable by performing an encryption operation on it when it is sent. When the message is received, the text is made readable again by performing a decryption operation when the message is read, as shown in the following figure.

That matches up with the behavior I've seen on all the encrypted emails I have looked at. Message body is encrypted but the headers are not. So maybe we are taking the wrong approach here and should just decrypt the contents when the message is first opened.

One edge case is what happened when an encrypted message has an encrypted message as an attachment. I haven't tested that but it seems like it must just attach like a regular message or it wouldn't have the header info and then it would just have, inside it, a p7m attached.

from xstreader.

Which approach would discard the outer body and the inner headers. This is the right solution if it is the common case.

If you want to try this out, your plug-in point is different. The precedent is EmbedAttachments in the Message class, which takes attached images and munges them into an HTML body, returning a new body.

This gets called from our old friend ShowMessage. You would need to add a new property to Message to test for the presence of an encrypted attachment, and then a new method in Message to replace the body with the encrypted attachment's contents.

Dijji

from xstreader.

Can you help me with getting the bytes of the attachment? I tried looking at the saveattachments method in xstfile.cs but wasn't able to successfully port it into showmessage because of thew private variables etc.

From what I can tell I need to lookup the subnode tree of the attachment and then I'll need to look it up in the node DB and pull the content. Do I have that right? What is the correct way to pull that data when trying to access it in showmessage?

from xstreader.

Also I got showing of content with NativeBody and BodyHtml working. Not sure what i was doing before but after you confirmed I tested again and got it working.

from xstreader.

Just noticed the xstfile property in the "MainWindow : Window" class. Will play with that.

Maybe the right answer would be to write a new method for the xstreader class that returns attachment bytes?

from xstreader.

got this sorted :) working on it further now

from xstreader.

In case you didn't get here, EmbedAttachments is again your precedent. You can get the contents of the attachment in a MemoryStream:
var s = new MemoryStream(); xst.SaveAttachment(s, a); s.Seek(0, SeekOrigin.Begin);

from xstreader.

Hey Dijji,

I have most of this working just one major part left and I'm honestly not exactly sure how to handle it so I thought I would see if you had ideas.

Attachment... so attachments are base64 encoded in the message contents these include both things like inline images or attached pdfs doc etc. From what I can tell in a normal message you would look up their content in the NDB. My content is in this encrypted message though and not directly in the NDB.

The possible ways I could see handling this are:

• Add a property to the attachment class to hold the attachment bytes
• Somehow parse the bytes out of the message and incorporate them into the NDB (I don't really understand all the inner working of this so I wouldn't know where to start)
• Manually updating the message object with the bytes ( this works well for inline images but not so much for other types of attachments).

Thanks for the help!

from xstreader.

Also one other bug I'm running into:

When I set NativeBody to HTML it looks like it gets reset to RTF if I click off and then back on. Any idea why this is happeneing? My changes to the BodyHTML property stay but not the nativebody ones.

from xstreader.

Two quick answers.

Firstly, stay from anything to do with the NDB. There is a lot of apparatus there, and I keep my dealings with it strictly read-only. All it does in the end, anyway, is to deliver byte streams, representing message and attachment content. Use your own mechanisms, and store the results in the message and attachment objects.

On switching to RTF, I have no idea. I would add breakpoint on the property setter just after I had set it to HTML, and then run things on to see who the culprit was.

from xstreader.

Thanks for the reply. Will definitely stay away from NDB and just operate in the attachment class.

The RTF thing turned out to be a Heisenbug for me. I'm not able to replicate it anymore. I cleaned up some of the code I implemented in between so maybe it was an issue with something I was doing and I inadvertently fixed it.

If I update the message object in showmessage should changes to the attachments for it persist between selecting messages in the UI?

At the moment I am trying to remove the p7m attachment since I have extracted the HTML contents of it and am displaying it already. That works fine the first time I select it but the attachments list seems to reset when I select another message.

I tried following execution and watching for it in listMessages but it seems to stay empty all the way up until it gets displayed but then when I return back to it after selecting another message it has the p7m attachment back in it.

from xstreader.

The program deliberately keeps nothing from when a given message is selected to when it is selected again. When a message is selected, its contents are read from the database into memory (Message object, et cetera). When another message is selected, this memory copy is thrown away.

This is the right thing to do for performance. Consider the case of an entire pst file being exported. By the end, every message in the file would be in memory otherwise, causing considerable bloat.

It is also the right thing to do for security. The decrypted copy of an email should be thrown away when a user stops looking at it. Also, one successful decryption does not predicate another: the key may have been removed in the meantime.

So, do everything you want to do when the message is opened, and expect to do everything all over again the next time the message is selected.

Dijji

from xstreader.

Design wise I am on board.

Behaviorally though in my tests changes to BodyHtml (and several other attributes) are persisting across selection/deselection of a given message. Do I need to hook into some deselection code somewhere to clear them or is this unexpected behavior?

It seems to me a new message object isn't being created on reselection or at least somehow it is retaining the decrypted html string in BodyHtml.

from xstreader.

You're right. When it opens a folder, it reads all the messages in it in a minimal state so that they can be listed. However, it does not clean up content after selection when the focus moves away. This is a bug. I propose the following changes:

In message.cs, after line 140, add

   public void ClearContents ()   
    {   
        // Clear out any previous content   
        Body = null;   
        BodyHtml = null;   
        Html = null;   
        Attachments.Clear();   
    }

In view.cs, at the beginning of SetMessage, insert

if (CurrentMessage != null)
CurrentMessage.ClearContents();

Let me know if this does the trick.

from xstreader.

That worked great. Thanks!

from xstreader.

Set up a pull request here #28 let me know if you have any questions or feedback. I learned a lot going through this :)

A couple caveats, I don't do any signature verification. Also, I did run into one email that was sent in winmail.dat format after being unencrypted and I didn't write any support for that other than being able to download it as an attachment.

from xstreader.

The actual function of the code looks fine. I do have some stylistic and structural issues, though. Stylistically, remember that method names always begin with a capital letter.

Structurally, the main problem is your static class. I think I would have put most of what you have in ShowMessage in the Message class, along with all of your static class (which rather takes me back to the era of Visual Basic modules).

For instance, parseMessage clearly belongs on Message, as it spends all its time updating message properties. Also, if it were on Message and called something like ParseMimeBody, it would be ready for later extensions like importing mime from another source.

Similarly, with all the decryption work hanging off ShowMessage, should I choose to decrypt a message when I want to export it (it is an interesting question as to whether I should), I don't have access to that capability within the export method on Message.

It just needs a bit of hacking about with cut-and-paste, and no real functional changes. Would you like to have a go, or would you prefer me to try it myself and check in the results to the branch for your review and approval?

But on the whole, congratulations again on a nice piece of work!

from xstreader.

from xstreader.

Alright think I did a little better job with the pull request this time. #29

from xstreader.

Much better! I've had to turn the picky dial right up, and I'm still not finding a great deal. But here goes.

• ParseMessage should be ParseMimeMessage
• Why is DecryptMessage public. Why would someone outside the class use it?
• Is there a misunderstanding of Encoding.UTF8.GetString(bytes);? For instance, the
  comment on DecodeQuotedPrintable says it returns UTF-8. But it doesn't, it returns a string which is a UTF-16 wide string like everything else on Windows. The UTF-8 actually refers to the encoding it is assuming it will find in the byte array. Despite the comments, you seem to have those correct, but you should go round and check.
• The handling around SaveAttachment and a.AttachmentBytes is rather clunky. SaveAttachment probably wants to ask the attachment if it has its own byte stream, which could have come from any source, not just from mime, and if so, use it. The counterpart to this is that some of the handling in ParseMessage should be moved into the attribute class. The long series of lines all beginning with a. is a strong indicator that encapsulation is being broken.

from xstreader.

• Done
• Changed DecryptMessage and DecodeSignedMessage to private
• HexDecoderEvaluator returns UTF8 encoded characters and is used as the match evaluator for DecodeQuotedPrintable. Maybe I am misunderstanding here but I assumed the regex replaced string would be UTF8 as well.
• I am a bit unclear on what I should do here. Do you think I should replace the Byte[] attachmentBytes with a stream on the attachment class? Would I then do all the parsing of the mime section for the attachment in the attachment class? i.e. move the
  the GetHeaders method over to there. I can see how certain things can just be determined by the attachment bytes like the attachment length or the attachMethod but other things need to be parsed out like the Filename or ContentId.

The handling around SaveAttachment and a.AttachmentBytes is rather clunky. SaveAttachment probably wants to ask the attachment if it has its own byte stream, which could have come from any source, not just from mime, and if so, use it.

this was my intention with the code below but I guess I missed the mark

   public void SaveAttachment(Stream s, Attachment a)
    {
        if (a.AttachmentBytes != null)
        {
            s.Write(a.AttachmentBytes, 0, a.AttachmentBytes.Length);
        }

Thanks for the feedback! Looking forward to learning more.

from xstreader.

What I mean is, it cannot return a UTF-8 encoded string because there is no such thing. See https://docs.microsoft.com/en-us/dotnet/api/system.string?view=netcore-3.1

For attachment, maybe a constructor that takes a byte array and anything else you can't default like maybe ContentId or filename. Then a property to see if it has a byte array value, then if so, a property or method to retrieve it.
Even better, why doesn't that constructor just store the byte array as Content, then we can get rid of the AttachmentBytes property altogether, and remove the special casing needed to handle it?

from xstreader.

Did most of this but ran into some issues around SaveAttachment. I wasn't able to find a way to really simplify it down because of the NDB usage.

It does seem kind of redundant since the same code/checks happen around content but I couldn't think of a way to check both before the NDB is accessed and after without separately checking if content was already set.

from xstreader.

No, I think what you have is about as good as it gets. I'm going to merge your code into master.

Could you give me a couple of lines on what your addition does that I can include in the release notes?

from xstreader.

How about the below:

Added support for viewing encrypted or signed messages and their attachments if matching certificate is in user certificate store. (Does not perform signature verification)

from xstreader.

Looks good. I will build the release shortly, though the usual regression testing will doubtless take me a few days to get through.

from xstreader.

Just a thought and I'm not surew how you would like to handle this but we may want to add some error handling around decryption with a custom error message that says "message failed to decrypt" in mainwindow.xaml.cs (~line 572). Unless we just want it to fall back to the normal error handling.

from xstreader.

This would be in cases, for example, where the correct cert was not in the user cert store.

from xstreader.

This makes a lot of sense. A quiet message on the status bar when you go to open the email would be perfect. At the moment, any thrown exception would result in a message box containing the full stack trace, not nearly so helpful. Would you like to have a go at it, or should I?

from xstreader.

Took a crack at it let me know what you think.

from xstreader.

Released as part of version 1.12.

Sorry for the delay. Life made me an alternative offer which I could not refuse.

from xstreader.