Enhancement: support for filtering emails about cypht HOT 38 CLOSED

Let's get started on this.

An important first step is to pick the best lib. Once we pick and integrate the lib, we can progressively add features and improve the UI. So here is some research:

Used by Roundcube and Kolab, https://github.com/pear/Net_Sieve is a contender but https://pear.php.net/package/Net_Sieve reports that "This package has been superseded, but is still maintained for bugs and security fixes. Package has moved to channel pear.horde.org, package Horde_ManageSieve.", which can be found here: https://github.com/horde/ManageSieve

Some other PHP options:
https://github.com/sergiosgc/Sieve_Parser
https://github.com/ProtonMail/libsieve-php/

Nextcloud mail is working on this: nextcloud/mail#1822
And strangely, they include 2 libs:
https://github.com/nextcloud/mail/pull/1822/files#diff-b5d0ee8c97c7abd7e3fa29b9a27d1780

"Ingo, the "Email Filter Rules Manager", started as a frontend for the Sieve filter language, and is now a generic and complete filter rule frontend that currently is able to create Sieve, procmail, maildrop, and IMAP filter rules."
https://www.horde.org/apps/ingo

And now, some non-PHP projects of interest:

A former Thunderbird add-on which has become an Electron app:
https://github.com/thsmi/sieve/

SOGo (Objective-C) has its own system or permits an external Sieve script:
https://sogo.nu/bugs/view.php?id=4387

Apache James jSieve (Java)
https://github.com/apache/james-jsieve

Do you know of any information that can help our choice?

Thanks!

from cypht.

Lets move "offline access" to the PWA issue and continue the discussion on filtering here. I'm of two minds on this topic:

• Server side filtering is obviously superior. Procmail, Sieve, whatever else exists I don't know about.
• Client side filtering is limited but works everywhere. This is pretty persuasive.

If I were to build this in Cypht right now, I would start with simple regex based client side filtering in the Cypht PHP code via a module set. Not logged in? No filtering. Using another client? No filtering.
Cypht is a lightweight E-mail client, not an always on server side inbox filter.

I would build this with an eye on abstracting the implementation of defined filters such that it might be possible to extend it to write procmail or sieve rules, so for sites that support it you can have nice server side filtering outside of Cypht.

from cypht.

@dumblob Someone is more ambitious than you :-) thsmi/sieve#258

from cypht.

Seeing you adding related links to filtering, I'll add one too 😉.

This is what I did under pressure when the discussion didn't seem to go anywhere: Nextcloud Mail - UI proposal for filtering (it's all based around "filters as views" idea).

from cypht.

Definitely. I built client side filtering in my last mail client and plan to support that eventually here as well.

from cypht.

Awesome. If something should appear, I'd advocate for allowing passing feeds (RSS...) through these filters as well. Yes, matching only headers wouldn't suffice, but extending matching to whole message bodies should be quite easy.

Btw, is there any todo list (not necessarily a firm/fixed one) I could follow (and prevent e.g. duplication of questions)?

from cypht.

As much as possible, I want any feature added to Cypht to apply to all applicable data sources (feeds, E-mail, whatever), so I agree with your proposal.

Insofar as a todo list, not really :) I love any and all feedback though, so feel free to file as many issue items as you like and we can use this as an ad-hoc todo list until we have a more formal roadmap. There are 2 major things I want to complete before starting on something like filters:

1. basic contacts support. This is 2/3rds done
2. outbound attachments. not started yet

Once these are done I think we can call ourselves a basic, but reasonably feature complete, E-mail client. Then we can start on the fun stuff like filters and threading :)

from cypht.

Thanks for the plan overview. I'll happily do some "unofficial" code reviews on the latest commits.

Also, there should be some subtle button to import and export all user filters to a formatted file (either some ad-hoc JSON/... structure or standardized format like Sieve in case the filters will be implemented according to the corresponding standard). The export function is more important (and easier to implement) than the import feature.

In my very own opinion, we don't have to support Sieve filters, because they're quite tricky to implement (especially because to make them enough expressive for real life usage, one must implement many additional RFC standards extending the original Sieve standard) and slow, but rather some dead-easy regex matching performed separately on the email headers section and on the rest of the email. For such regex matching, we should guarantee, that the matching won't get stuck in regex exponential algorithms and thus we should use DFA-based engine, e.g. RE2 engine (maybe https://github.com/arraypad/php-re2).

from cypht.

Really appreciate the support! If you see anything in a commit that looks suspect, I would love your feedback! I agree with all your suggestions WRT a filtering mechanism. I think we could make use of the built in PCRE engine in PHP by white-listing patterns we allow, as well as adding anchors to avoid inadvertent backtracking. I would love to support a standard like Sieve eventually, but a simple regex is a good start in the meantime (and likely good enough for most requirements).

from cypht.

After some consideration and investigation involving RFC 5228 (Sieve), it's implementation in dovecot including extensions, and RFC 5804 (ManageSieve) I came to conclusion, that regexps would be much better (Sieve with all the existing RFC extensions/plugins is still very limited), but their implementation is more or less on the same level of complexity as incorporating some PHP library for the ManageSieve protocol.

This means, that it might make sense to first implement support for ManageSieve and then consider if it's worth implementing direct support for any other filtering (be it Sieve or ad-hoc regexps or both), because e.g. Gmail uses it's own non-standard filters (ManageSieve is not supported).

I hope the PHP libraries mentioned on http://sieve.info/libraries would suffice (in terms of security and velocity).

from cypht.

Agreed to that, Jason: The nice thing with server-site filtering is that it can be centralised for bigger environments, while client-side is always a matter of individual preferences. This is not only about controlling mails, but also, for instance, to have certain mails go into certain folders routinely, by default, for operational reasons, for all staff, etc. Cheers!

from cypht.

For the Cypht integration in Tiki, we plan (medium term) to add more automation. Ex.: filtering mail for a certain criteria, parsing to extract content, and do different actions (ex.: add info to a database, forward a modified email, etc.)

All this should happen server side, but we want a nice client side GUI to build the filters and actions.

from cypht.

@dumblob @jasonmunro: Do you know of any more options or we can start comparing the one I found and proceed by elimination?

Thanks!

from cypht.

@marclaporte no, don't know any other off the head. But as I pointed out above, please focus rather on regexps and discard (or at least postpone) the idea of Sieve as they're extremely limited, slow, etc. Sieve just doesn't fit into the Cyph minimalism at all. Not speaking about the fact that you can achieve everything what Sieve can with regexes.

The cool part is, that support for regex filtering will have the very same or simpler UI than Sieve. So again, please reconsider the priorities. Additionally I claim, that everybody who uses Sieve can write regexes (and those who don't use Sieve, but some very limited subset through some specially crafted UI can be satisfied with the same UI which under the hood uses regexes instead of Sieve).

I find Sieve a complete dead end slowly dying over the last decade or more.

The only thing which could have a marginal chance of convincing me to additionally support Sieve (along regexes) is the ManageSieve protocol. But I'm convinced it's not necessary for Cypht as Cypht is a web mail and not all mail providers offer ManageSieve, so one couldn't even use it everywhere, so it's way easier and safer to implement local matching based on regexes IMHO which'll work always and everywhere.

from cypht.

Ok, but how would we do server-side filters with a mail server we don't control? ( shared hosting)

in Tiki, we can use http://doc.tiki.org/Browser-Automation but what about standalone Cypht?

from cypht.

Ok, but how would we do server-side filters with a mail server we don't control? ( shared hosting)

First of, we don't. The point is there shouldn't be any need for this. Cypht knows which messages are "really new" (i.e. unseen/not_yet_opened) and which are just "set unread" by the user; so the filtering happens at each login or if already logged in each time a new message appears in the inbox.

What did you have in mind with browser automation? Everything will run on the Cypht server, so it has nothing to do with any browser.

from cypht.

Users may access mailbox from another client, so filters should be running in the background.

Browser automation: If some of the actions are client-side.

But the best is to put all this in PHP, on a cron job.

from cypht.

Users may access mailbox from another client,

I intentionally wrote The point is there shouldn't be any need for this. because if someone decides to use Cypht, there is no practical reason to use other email clients. If you'll find such "combined" use cases, I'd be happy to discuss them 😉.

so filters should be running in the background.

That'd require the user to be logged in all the time which will compromise security. But maybe we could make it a user setting in the end.

Browser automation: If some of the actions are client-side.

Very unlikely to be needed as it's definitely not wanted (it'd complicate the client-server communication in both ways and more importantly increase the bandwidth in both ways - and we know that especially upload is usually a scarcity at least on mobile devices).

from cypht.

Some users will want offline access, which could be done via PWA: #395

I want filters to work throughout accounts. Ex.: I want a filter that detects invoices and moves out of my inbox and into the inbox of the bookkeeper. This should happen even if I am offline for a while.

from cypht.

Some users will want offline access, which could be done via PWA:

I'm afraid I don't follow. Why would one need any message filters (don't forget we're talking about newly incoming message filters here) in an offline mode (when by definition you can't get any new messages 😉).

I want filters to work throughout accounts. Ex.: I want a filter that detects invoices and moves out of my inbox and into the inbox of the bookkeeper. This should happen even if I am offline for a while.

This means you'd like the computer to act on your behalf which translates to you'd like to delegate part of your responsibility to a third party (Cypht server and its "affiliates" who usually do explicitly limit their liability to virtually zero).

So I think you'd need to tick the option "stay logged in after log out" to lower security in exchange for more convenient functionality 😉.

from cypht.

"I'm afraid I don't follow." You asked why someone would use something else than Cypht.

from cypht.

"I'm afraid I don't follow." You asked why someone would use something else than Cypht.

Ah, you mean Cypht doesn't offer full offline functionality now, so that could be the reason to use another client. Fair enough.

In that case though I'm afraid PWA has nothing to do with offline access, so we should make a separate issue for offline mode. And I mean a fully "automatic" offline mode - i.e. the client will work as usually even without internet (i.e. no manual intervention/switching will be needed), but will just show some "exclamation mark icon" when composing & sending message which'll disappear once no work is left (i.e. all queued messages to send will be successfully sent out).

from cypht.

"PWA has nothing to do with offline access"

What is a better way to create an offline web app? (Beside Electron)

from cypht.

"PWA has nothing to do with offline access"

What is a better way to create an offline web app? (Beside Electron)

Oh, now I see the misunderstanding. I just meant that for offline access there is no need to implement PWA (because offline access is not equivalent to loading an app offline). And vice versa - if you're PWA-ready you have the app loading offline, but the app itself doesn't need to work offline at all (i.e. doesn't need to do any meaningful work; meaningful from the user's point of view).

I would build this with an eye on abstracting the implementation of defined filters such that it might be possible to extend it to write procmail or sieve rules, so for sites that support it you can have nice server side filtering outside of Cypht.

Awesome. That's the third option - to implement both. I didn't emphasize this option (just initially wrote about ManageSieve as the underlying core) as I anticipated it to be more complicated and difficult to maintain. But I'm glad it's on the todo list 😉. Thanks!

from cypht.

JMAP & Sieve:
https://tools.ietf.org/html/draft-murchison-jmap-sieve-01
https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/jmap_sieve.c

from cypht.

https://www.ietf.org/id/draft-gondwana-sieve-mailboxid-02.txt

from cypht.

Maybe we could finally forget about built-in support for Sieve in Cypht if Cypht will integrate with parts of (or at least refer to) Sieve Editor.

Another interesting thing is that Sieve Editor supports the Regular Expressions Extension of Sieve which could mean that the whole functionality requested in this GitHub issue is covered by Sieve Editor. I'll try to find some time to test it. Feel free to do the same 😉.

from cypht.

@dumblob: Sieve Editor looks powerful and mature. Can you describe how you envisage the integration?

Thanks!

from cypht.

Can you describe how you envisage the integration?

I'd sum my vision as "as loosely as possible while maintaining partially smooth workflow with filters". Maybe just iframe on a few places? In other words, I don't want to integrate Sieve editor into Cypht, but rather integrate Sieve editor with Cypht if that makes sense.

from cypht.

Yes, sure it makes sense. It looks quite advanced and it could provide some quick wins.

My big concern: How would Sieve editor have Cypht's account list and permit to move emails from one account to another, and to forward mail?

Thanks!

from cypht.

Some interesting ideas here:
https://forum.virtualmin.com/t/improvements-to-mail-filtering/108798

from cypht.

We are starting work on this.

Ref: https://gitter.im/cypht-org/community?at=61e1929c2a210c38c1cafb2e

from cypht.

For the record, @henrique-borba has started on this: https://github.com/henrique-borba/php-sieve-manager

from cypht.

Screenshot:
https://gitter.im/cypht-org/community?at=624d9bdbd1b64840dba0efa8

from cypht.

Draft: #569
It uses the lib, which has progressed: https://github.com/henrique-borba/php-sieve-manager/commits/master

from cypht.

Very good progress: #569

from cypht.

Closing as done: #569 (Just need @jasonmunro to approve merge request)

from cypht.

A follow up: #679

FYI @dumblob

from cypht.