mbrt / gmailctl Goto Github PK

The "never send it to spam" action is not supported at the moment. This makes also import fail whenever Gmail decides to modify your filters and add this action. [Not sure why this happens].

The URL with search results of a certain query in Gmail doesn't change between users. We could exploit that by showing URLs that the user can use to test if their filters are correct.

Users should be able to define their tests. Example:

{
  rules: [ /* rules */ ],
  tests: [
    {
      messages: [
        {
          subject: "Some message",
          to: "[email protected]",
          /* more fields */,
        },
        /* other messages */
      ],
      actions: {
        // expected effect of all actions from all matching rules
        labels: ['spammy-alerts'],
        archive: true,
      },
    },
    // more tests
  ],
}

To avoid re-implementing too much of gmail filters logic, we could restrict the checks to well known operators and "ban" the raw queries altogether.

We should also add a new test command in the gmailctl cli.

The problem is that in the temporary directory, the file is not accessible.

See go APIs, and API reference.

It would also be nice to add the possibility to review the changes before committing, by using difflib.

• algorithm implementation
• use Gmail APIs
• command line parameters
• refactor xml export code
• user docs

Instead of adding more and more features to the config file, it's better to rely on a proper configuration language: https://jsonnet.org/

This will require a new version of the config format.

gmailctl requires only one specific scope to function:

https://www.googleapis.com/auth/gmail.settings.basic

It'd be nice to document that, rather than advising to create what amounts to read-write access to everything during project setup. (Also, best practice, of course.)

If you do what I asked in #54 then you would also need

https://www.googleapis.com/auth/gmail.labels

You can find the specific scopes by hitting the reference pages, such as:
https://developers.google.com/gmail/api/v1/reference/users/labels/create#auth

To reproduce:

1. In gmail, create a label "deleteme"
2. In gmail, create a rule that applies label "deleteme" and performs no other actions
3. Delete label "deleteme"
4. Run gmailctl diff

This causes the following stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x8e22a8]

goroutine 1 [running]:
github.com/mbrt/gmailctl/pkg/export/api.defaultImporter.importAction(0x0, 0xb82260, 0xc0005ac450, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/pkg/export/api/api_import.go:65 +0x58
github.com/mbrt/gmailctl/pkg/export/api.defaultImporter.importFilter(0xc0007aea20, 0xb82260, 0xc0005ac450, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/pkg/export/api/api_import.go:48 +0x8e
github.com/mbrt/gmailctl/pkg/export/api.defaultImporter.Import(0xc00058a200, 0x3e, 0x3f, 0xb82260, 0xc0005ac450, 0x0, 0x7fd80111a000, 0xc0001da5a0, 0xc0001f6700, 0xc0001ec0d8)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/pkg/export/api/api_import.go:34 +0xeb
github.com/mbrt/gmailctl/pkg/api.(*gmailAPI).ListFilters(0xc000220760, 0xc000220760, 0x0, 0x0, 0xc00065cec8, 0x8)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/pkg/api/api.go:42 +0x1bc
github.com/mbrt/gmailctl/cmd/gmailctl/cmd.diff(0xc000024440, 0x36, 0xc000024440, 0x36)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/cmd/gmailctl/cmd/diff_cmd.go:54 +0x18e
github.com/mbrt/gmailctl/cmd/gmailctl/cmd.glob..func3(0x11ee1a0, 0x1219318, 0x0, 0x0)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/cmd/gmailctl/cmd/diff_cmd.go:30 +0x42
github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra.(*Command).execute(0x11ee1a0, 0x1219318, 0x0, 0x0, 0x11ee1a0, 0x1219318)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra/command.go:766 +0x2ae
github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x11eeb20, 0x0, 0xac397b, 0x2d)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra/command.go:852 +0x2c0
github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/vendor/github.com/spf13/cobra/command.go:800
github.com/mbrt/gmailctl/cmd/gmailctl/cmd.Execute()
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/cmd/gmailctl/cmd/root_cmd.go:30 +0x32
main.main()
        /usr/local/google/home/lutzky/go/src/github.com/mbrt/gmailctl/cmd/gmailctl/main.go:6 +0x20

Instead, gmailctl should probably just ignore that rule in the importer. The gmail UI attempts to block you from creating rules that have no actions.

Generated filters are sometimes duplicated. Still needs to investigate why.

As I said in a previous post, thank you for this great tool.

I have a set of co-workers, and I have a filter that applies a label to messages received from these co-workers or sent to these co-workers. The list of co-workers includes about twenty people.

To setup this filter in the config file, I created two variables. The first variable looks like this:

local fromCoworkers = {
or: [
{ from: '[email protected]'},
{ from: '[email protected]'},
{ from: '[email protected]'},
...
{ from: '[email protected]'},
],
};

The second variable looks like this:

local toCoworkers = {
or: [
{ to: '[email protected]'},
{ to: '[email protected]'},
{ to: '[email protected]'},
...
{ to: '[email protected]'},
],
};

And the filter rule looks like this:

{
  filter: {
    or: [
      fromCoworkers,
      toCoworkers,          
    ],
  },
  actions: {
    labels: [
      "Coworkers"
    ]
  }
},

This works, but I have to maintain two versions of the same list. I have to list all twenty email addresses in the fromCoworkers variable and then the same set of email addresses in the toCoworkers variable.

Is there a way to do this, so that I only have to maintain one list?

Thanks.

[As with my previous post, I intended to label this as a question. I just read the Github help files about applying labels to issues. It looks like only the owner of a project can apply labels. Anyway, I hope I'm not doing this wrong by not applying a label.]

For example, if I have a test where one of the messages has to: ['some-list'], and I have a rule filtering for to: ['some-list'], the test thinks that this rule is applied to that message. However, if the message has to: ['[email protected]'], the test things that this rule is not applied to that message; in real-gmail-filtering it is.

Is there any way to do offline compilation without connecting to GMail? I would like to try it out quickly.

To enable complex filters we should add an expression operator, that enables custom queries (see
expressions docs).

We could additionally allow variables that refer to global consts to make it even fancier.

I have this as a filter in config.jsonnet:

      filter: {
        and: [
          {
            from: '@paypal.com',
          },
          {
            or: [
              {
                subject: 'Receipt for *',
                isEscaped: true,
              },
              {
                subject: 'authorized a payment',
                isEscaped: true,
              },
            ],
          },
        ],
      },

When I run gmailctl diff, that or branch is shown as:

+    subject: {Receipt for * authorized a payment}

It makes it look like the subject Receipt for * authorized a payment is being matched, rather than Receipt for * or authorized a payment.

I just spent about an hour writing a fairly complicated set of rules, then saved the file and checked the diff and it didn't look quite right. So I typed n and re-ran gmailctl edit and was quite shocked to see all my work had disappeared...

It would be nice if there was another option other than y/n: perhaps e to re-open the file. Also an indication that n will discard your changes.

Using gmailctl export prints the xml without newline at the end. This causes the last line to be missing from the terminal. It works when piped to some other program.

The current documentation explains v1alpha1. We should update it with the new version.

When starting to use gmailctl you get an initial, example template list. It would be really useful if you could download your existing filters so you could tweak them instead of starting afresh.

If a config contains unknown fields no error is reported.

As far as I can tell from Google’s OAuth 2.0 for Mobile & Desktop Apps page and the linked RFC 8252, nobody assumes that the information in the the credentials file is actually secret despite its name.

I recommend double-checking (both in case I’m mistaken, and of course for security reasons), but if I’m right it would greatly simplify installation if you didn’t make users create their own projects.

Gmail supports labels with forward slashes in the name, without mandating the parent label to be there. For example a label could be called "foo/bar" and "foo" doesn't have to be present for this to work.

The current validation is a bit restrictive and demands "foo" to be in the list of labels as well. This was to avoid users' confusion, but removes the use case. Since Gmail doesn't complain about this case, gmailctl shouldn't complain too.

For example: Can I filter an email that looks like this, but only when it's from this specific some-one.

from: 'Some, One' via list-name <[email protected]>
reply-to: "Some, One" <[email protected]>
to: "list-name ([email protected])" <[email protected]>
mailing-list: [email protected]

This is a feature request.

Current:

{
  filter: {
    and: [
      { from: 'ticketsroleaccount <[email protected]>' },
      {
        not: { to: me }
      },
    ],
  },
}

I would like:

{
  filter: {
    from: 'ticketsroleaccount <[email protected]>',
    not: { to: me },
  },
}

This behavior is same as search, and is backward compatible (test whether there are more than one key).

So that a user can write bcc: 'someone@something' instead of query: 'bcc: "someone@something"'.

1. Create a new project if you don't have one
1. Go to 'Enable API and services' and select Gmail
2. Go to credentials and create a new one, by selecting 'Help me choose'
   2a. Select the Gmail API
   2b. Select 'Other UI'
   2c. Access 'User data'.
3. Go to 'OAuth constent screen' and update 'Scopes for Google API', by
   adding:
     * https://www.googleapis.com/auth/gmail.labels
     * https://www.googleapis.com/auth/gmail.metadata
     * https://www.googleapis.com/auth/gmail.settings.basic
4. IMPORTANT: you don't need to submit your changes for verification, as
   you're not creating a public App
5. Download the credentials file into '/Users/tcurdt/.gmailctl/credentials.json'
   and execute the 'init' command again.

This is not covered:

After step 4 it says

and there is no way to "Save" without adding more information.

Filters with lists do not generate any output.

config.jsonnet snippet for filter demonstrating issue:

    {
      filter: { from: "[email protected]" },
      actions: { labels: ['example'] },
    }

gmailctl debug output:

# Search: from:[email protected]
# URL: https://mail.google.com/mail/u/0/#search/from:[email protected]
filter:
  from: [email protected]
actions:
  labels:
  - example

The unescaped + in the address is interpreted as a space in the URL, taking you to a search for from:foo [email protected].

I've got a problem in my rules:

cannot parse config file: error parsing criteria for rule #20: empty filter node

...which is probably accurate, but having to carefully count forward 20 rules by hand, especially in the face of functions, is much more challenging than I'd like.

It'd be great if this dumped out the JSON or equivalent of the rule. That won't give the original jsonnet for it, of course, but I'd expect it has a very high chance of including enough identifying details that would make it fast to track down the error.

This is an "application level" error, incidentally: the JSON generates correctly, but I'm presumably missing something in there.

gmailctl edit fails to pickup the editor if $EDITOR contains arguments. Example EDITOR=nano -w.

See https://support.google.com/mail/answer/7190?hl=en

I downloaded my filters from gmail and the diff command shows no diffs.
However when I look in gmail I have filters like the following:

Matches: list:foo.bar.com -{(to:[email protected])}
Do this: Skip Inbox

Matches: list:foo.bar.com
Do this: Apply label "list"

The downloaded filters look like:

    {
      filter: {
        query: "list:foo.bar.com"
      },
      actions: {
        archive: true
      }
    },
    {
      filter: {
        query: "list:foo.bar.com"
      },
      actions: {
        labels: [
          "list"
        ]
      }
    },

and seem to have lost the part excluding emails sent directly to me.

Am I misunderstanding how the config is supposed to work?

The YAML configuration file is not much used anymore as it's way less useful than the Jsonnet one. We should get rid of it and as we are at it deprecate named filters as well.

Thank you for wonderful project! I have more then 70 filters in gmail and using such tool is really helpful.
My current problem that i have some filters for single "to" address that can fills to multiple labels , does it possible to write something that if filter matches - it labeled and stop processing other filters?
This is really can helps in situations when you need to have big "not" stuff inside many filters. (i'm don't like to see one email in multiple labels)

Repro:

1. gmailctl edit
2. Add a new rule using a label that does not exist
   -> edit fails, user presented with option 'Do you want to continue editing?'
3. Choose 'y'
4. Add the label in gmail (leaving the editor open)
5. Try to go ahead with the edit
   -> Still results in the same error (Label does not exist)

Expected: edit should go through once the label is added

Note: if you choose to abort the editing, the changes you made to the config are lost, or at least there's no obvious way to recover them

"Forward it to: Choose an address.." from Gmail's filter creation is missing as an Action option in gmailctl.

The Gmail API supports it as action.forward. The parameter is a string containing the forwarding email. I confirmed this with the API explorer.

However, forward is different because it's not a pseudo-label like gmailctl's other actions.

user.settings.filters.create returns an error if the forwarding address is not set up on the user's account:

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "failedPrecondition",
    "message": "Unrecognized forwarding address"
   }
  ],
  "code": 400,
  "message": "Unrecognized forwarding address"
 }
}

The same error happens for malformed input, like @ or [][]. The empty string succeeds as if forward was absent.

Should we care about the case where a user wants one filter to forward to multiple email addresses? It's probably rare, and I'm not sure how Gmail would react to the duplicate queries with different forwarding addresses.

I'm blocked from using gmailctl for my personal email because I have 3 filters that forward my email. I forward my award point statements to AwardWallet (so that it doesn't have permissions to read my entire mailbox).

I looked at the code a bit, and I may try implementing this in the next week or so.

The current algorithm is too naive when many changes happen. We should use bipartite matching to minimize the diff.

I just applied my shiny new rule and ...

error adding filters: error exporting filter #0: error in export action: label 'redacted' not found

...ouch. It'd be nice if labels could be auto-created when required.

I suspect that my go version is outdated, but maybe something else went wrong (I am trying to run go program for the first time in my life).

mateusz@grisznak:~/Desktop/tmp$ go install github.com/mbrt/gmailctl/cmd/gmailctl
can't load package: package github.com/mbrt/gmailctl/cmd/gmailctl: cannot find package "github.com/mbrt/gmailctl/cmd/gmailctl" in any of:
	/usr/lib/go-1.6/src/github.com/mbrt/gmailctl/cmd/gmailctl (from $GOROOT)
	($GOPATH not set)
mateusz@grisznak:~/Desktop/tmp$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/lib/go-1.6"
GOTOOLDIR="/usr/lib/go-1.6/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Available filters are listed, unfortunately, I see nowhere clear list of what actions can be applied and their codes

I've imported my current configuration and then tried to edit the rules, but while saving the following message pops up:

Error applying configuration: syntax error in config file: error parsing the config version: yaml: line 3: mapping values are not allowed in this context
Do you want to continue editing? [y/N]: 

Don't know why he thinks it is a yaml.

Config file:

// Auto-imported filters by 'gmailctl download'.
//
// WARNING: This functionality is experimental. Before making any
// changes, check that no diff is detected with the remote filters by
// using the 'diff' command.

// Uncomment if you want to use the standard library.
// local lib = import 'gmailctl.libsonnet';
{
  version: "v1alpha3",
  author: {
    name: "YOUR NAME HERE (auto imported)",
    email: "[email protected]"
  },
  // Note: labels management is optional. If you prefer to use the
  // GMail interface to add and remove labels, you can safely remove
  // this section of the config.
  labels: [
    {
      name: "Belege"
    },
    {
      name: "Reisen"
    },
    {
      name: "[Imap]/Trash"
    },
    {
      name: "Privat"
    },
    {
      name: "Geschäftlich"
    }
  ],
  rules: null
}

Filtering mailing lists can be done by using the has operator with list:(mymaillist-address).

We can currently workaround that by using:

  - filters:
      has:
        - list:(fd84c1c757e02889a9b08d289.122677.list-id.mcsv.net)

Ideally you would like to use instead:

  - filters:
      list:
        - fd84c1c757e02889a9b08d289.122677.list-id.mcsv.net

Quotes inside subject produce unexpected result, they basically remove quoting. Escaping is not really supported in GMail so it would be nice to print an error message if this happens to not confuse users.

    {   
      filter: {
        subject: '"hello world"',
      },  
      actions: {
        labels: ["testing"],
      },  
    },  

--- Current
+++ TO BE APPLIED
@@ -1 +1,5 @@
+* Criteria:
+    subject: ""hello world""
+  Actions:
+    apply label: testing
 

Sample error message:

config tests failed: test 'stuff' failed: message #0 is going to get unexpected actions: {"archive":true,"labels":["stuff"]}

The test was configured to just check for labels: ["stuff"]. A test failure would look nicer like so:

config tests failed: test 'stuff' failed: message #0 is going to get unexpected actions, -want +got:
 {
+  "archive": true,
   "labels":["stuff"]
 }

This should be easy using https://github.com/google/go-cmp.

When the user runs gmailctl init the gmailctl.libsonnet is created. Unfortunately the file it never updated automatically again.
I propose gmailctl init --reset-lib or similar flag to force update all library files (current and future).
It is understandable that gmailctl will not rewrite config.jsonnet as it may contain user configuration, however library files are not expected to be touched by the user.

When (for example) you run gmailctl apply you are prompted with

Do you want to apply them? [y/N]:

The Linux style of [y/N] suggests that the default behaviour when just hitting the Return key will be 'N' and nothing will happen. But currently, this flow will get you stuck.

To reproduce, make a change, run gmailctl apply and when prompted just hit return.

Tested on Mac with iTerm

I suspect this originates in the askYN method that uses fmt.Scanln that gives an error if you just hit return.

I will submit a PR with an alternative for fmt.Scanln using bufio ReadString

A simple lint command would be nice for just checking to see if the file is properly formed. Currently I use debug, but it would be nice to check the file without all that output in the terminal.

I started testing Gmail API (with simplest possible Python script based on their tutorial) and discovered that sequentially added filters are appearing on filter list in a random order.

Is this tool using API in way that also results in the same situation?

I have some filters that relied on import to Gmail keeping order and now I am looking for a solution. I starting to worry that there is no one but maybe gmailctl has some miraculous way of ensuring stable order.

gmailctl is a wonderful tool! I've been searching for something like this for a long time. Thank you for creating it.

I created a config file with about 20 or so filters. When I apply the config fild to my Gmail account, the filters do not remain in the order I have them in the config file. The order in the Gmail UI seems to be random.

I don't need the filters to be in order. My concern is not at all related to the sequence in which the filters are applied. I would like the filters in the Gmail UI to reflect the order of the filters in my config file, because that would help me manage them better. For example, if something is going wrong with a filter, I would be able to more quickly and easily find it in the Gmail UI, if the filters were in my preferred order.

Thanks, again, for creating this great tool.

[P.S. I apologize for not adding a tag to this post. I intended to label it as a question. I see on the right side it says, 'Labels: None yet', but I don't see how to give the post a label.]

I've been thinking about adding some helper functions to the gmailctl.libsonnet library.

In particular, I'm looking on a way to create valid structured filters using the logic operators. May be something that could be used like this,

local gmailctl = import 'gmailctl.libsonnet';
local Filter = gmailctl.Filter;

Filter.new()
=>
{}

Filter.new().and({to: "[email protected]"}).values()
=>
{
  and: [{to: "[email protected]"}]
}

The idea would be to be able to use method chaining to build and reuse filters. You could do something like

local to_me = Filter
				.new()
				.and({to: "[email protected]"});

local chained_filter = Filter
						.new()
						.and({subject: "special email"})
						.or(to_me);

chained_filter.values();
=>
{
	and: [{subject: "special email"],
	or: [{ and: [{to: "[email protected]"}]]
}

I'm a noob in jsonnet, I've been trying to implement something like this with some success. I'd love to know your thoughts on the idea, implementation, and d actual syntaxis.

Filters of the form:

{A B C D} -{E F G H}

Could be used as a catch all archive rule (e.g. either of these mail lists but not directed to me, or with this subject line).

This filter can become pretty big and so ignored by Gmail. Rewriting it into a set of filters like this:

A -{E F G H}
B -{E F G H}
...

Would solve the issue (if the exclusion list is short enough).

We should automatically apply this transformation in the SimplifyCriteria step.