README.md 10.1 KB
Newer Older
1
# Gmailieer
Gaute Hope's avatar
Gaute Hope committed
2

Gaute Hope's avatar
Gaute Hope committed
3 4
<img src="doc/demo.png">

Gaute Hope's avatar
Gaute Hope committed
5 6
This program can pull email and labels (and changes to labels) from your GMail
account and store them locally in a maildir with the labels synchronized with a
7 8
[notmuch](https://notmuchmail.org/) database. The changes to tags in the
notmuch database may be pushed back remotely to your GMail account.
Gaute Hope's avatar
Gaute Hope committed
9

10
## Disclaimer
Gaute Hope's avatar
Gaute Hope committed
11

12 13
Gmailieer will not and can not:

14
* Add or delete messages on your remote account (except syncing the `trash` or `spam` label to messages, and those messages will eventually be [deleted](https://support.google.com/mail/answer/7401?co=GENIE.Platform%3DDesktop&hl=en))
15 16
* Modify messages other than their labels

17
While Gmailieer has been used to successfully synchronize millions of messages and tags by now, it comes with **NO WARRANTIES**.
Gaute Hope's avatar
Gaute Hope committed
18

19
## Requirements
20 21

* Python 3
Gaute Hope's avatar
Gaute Hope committed
22
* `notmuch >= 0.25` python bindings
23 24
* `google_api_python_client` (sometimes `google-api-python-client`)
* `oauth2client`
Gaute Hope's avatar
Gaute Hope committed
25
* `tqdm` (optional - for progress bar)
26

27
## Installation
28 29 30

After cloning this repository, symlink `gmi` to somewhere on your path, or use `python setup.py`.

31
# Usage
Gaute Hope's avatar
Gaute Hope committed
32

33
This assumes your root mail folder is in `~/.mail` and that this folder is _already_ set up with notmuch.
Gaute Hope's avatar
Gaute Hope committed
34

35
1. Make a directory for the gmailieer storage and state files (local repository).
Gaute Hope's avatar
Gaute Hope committed
36 37

```sh
Gaute Hope's avatar
Gaute Hope committed
38 39 40
$ cd    ~/.mail
$ mkdir account.gmail
$ cd    account.gmail/
Gaute Hope's avatar
Gaute Hope committed
41 42
```

43 44 45
All commands should be run from the local mail repository unless otherwise specified.


46
2. Ignore the `.json` files in notmuch. Any tags listed in `new.tags` will be added to newly pulled messages. Process tags on new messages directly after running gmi, or run `notmuch new` to trigger the `post-new` hook for [initial tagging](https://notmuchmail.org/initial_tagging/). The `new.tags` are not ignored by default if you do not remove them, but you can prevent custom tags from being pushed to the remote by using e.g. `gmi set --ignore-tags-local new`.
Gaute Hope's avatar
Gaute Hope committed
47 48 49

```
[new]
50
tags=new
Gaute Hope's avatar
Gaute Hope committed
51
ignore=*.json;
Gaute Hope's avatar
Gaute Hope committed
52 53
```

54
3. Initialize the mail storage:
Gaute Hope's avatar
Gaute Hope committed
55 56

```sh
57
$ gmi init your.email@gmail.com
Gaute Hope's avatar
Gaute Hope committed
58 59
```

60
`gmi init` will now open your browser and request limited access to your e-mail.
Gaute Hope's avatar
Gaute Hope committed
61

62 63 64
> The access token is stored in `.credentials.gmailieer.json` in the local mail repository. If you wish, you can specify [your own api key](#using-your-own-api-key) that should be used.

4. You're now set up, and you can do the initial pull.
Gaute Hope's avatar
Gaute Hope committed
65

Gaute Hope's avatar
Gaute Hope committed
66 67
> Use `gmi -h` or `gmi command -h` to get more usage information.

68
## Pull
Gaute Hope's avatar
Gaute Hope committed
69 70 71 72

will pull down all remote changes since last time, overwriting any local tag
changes of the affected messages.

Gaute Hope's avatar
Gaute Hope committed
73
```sh
Gaute Hope's avatar
Gaute Hope committed
74
$ gmi pull
Gaute Hope's avatar
Gaute Hope committed
75 76
```

Gaute Hope's avatar
Gaute Hope committed
77 78
the first time you do this, or if a full synchronization is needed it will take longer.

79
## Push
Gaute Hope's avatar
Gaute Hope committed
80

Gaute Hope's avatar
Gaute Hope committed
81 82 83
will push up all changes since last push, conflicting changes will be ignored
unless `-f` is specified. these will be overwritten with the remote changes at
the next `pull`.
Gaute Hope's avatar
Gaute Hope committed
84

Gaute Hope's avatar
Gaute Hope committed
85
```sh
Gaute Hope's avatar
Gaute Hope committed
86
$ gmi push
Gaute Hope's avatar
Gaute Hope committed
87 88
```

89
## Normal synchronization routine
Gaute Hope's avatar
Gaute Hope committed
90 91 92

```sh
$ cd ~/.mail/account.gmail
93
$ gmi sync
Gaute Hope's avatar
Gaute Hope committed
94 95
```

96 97 98 99 100
This effectively does a `push` followed by a `pull`. Any conflicts detected
with the remote in `push` will not be pushed. After the next `pull` has been
run the conflicts should be resolved, overwriting the local changes with the
remote changes. You can force the local changes to overwrite the remote changes
by using `push -f`.
Gaute Hope's avatar
Gaute Hope committed
101

Gaute Hope's avatar
Gaute Hope committed
102
> Note: If changes are being made on the remote, on a message that is currently being synced with `gmailieer`, the changes may be overwritten or merged in weird ways.
103 104 105

See below for more [caveats](#caveats).

106
# Settings
107

108
Gmailieer can be configured using `gmi set`. Use without any options to get a list of the current settings as well as the current history ID and notmuch revision.
109

110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133
**`Account`** is the GMail account the repository is synced with. Configured during setup with [`gmi init`](#usage).

**`historyId`** is the latest synced GMail revision. Anything since this ID will be fetched on the next [`gmi pull`](#pull) (partial).

**`lastmod`** is the latest synced Notmuch database revision. Anything changed after this revision will be pushed on [`gmi push`](#ush).

**`Timeout`** is the timeout in seconds used for the HTTP connection to GMail. `0` means the forever or system error/timeout, [whichever occurs first](https://github.com/gauteh/gmailieer/issues/83#issuecomment-396487919).

**`Drop non existing labels`** can be used to silently ignore errors where GMail gives us a label identifier which is not associated with a label. See [Caveats](#caveats).

**`Replace slash with dot`** is used to replace the sub-label separator (`/`) with a dot (`.`). I think this is easier to work with. *Important*: See note below on [changing this setting after initial sync](#changing-ignored-tags-and-translation-after-initial-sync).

**`Ignore tags (local)`** can be used to specify a list of tags which should not be synced from local to remote (e.g. [`new`](#usage)). In addition to the user-configured tags these tags are ignored: `'attachment', 'encrypted', 'signed', 'passed', 'replied', 'muted', 'mute', 'todo', 'Trash', 'voicemail'`. Some are special tags in notmuch and some are unsupported by GMail. See [Caveats](#caveats) below for more explanations. *Note:* This setting expects [_translated_ tags](#translation-between-labels-and-tags).
  
  *Important*: See note below on [changing this setting after initial sync](#changing-ignored-tags-and-translation-after-initial-sync).

**`Ignore tags (remote)`** can be used to specify a list of tags (labels) which should not be synced from remote (GMail) to local. By default the [`CATEGORY_*` type](https://developers.google.com/gmail/api/guides/labels) labels which are mapped to the Personal/Promotions/etc tabs in the GMail interface are ignored. You can specify that no label should ignored by doing: `gmi set --ignore-tags-remote ""`. *Note:* This setting expects [_*un*translated_ tags](#translation-between-labels-and-tags).
  
  *Important*: See note below on [changing this setting after initial sync](#changing-ignored-tags-and-translation-after-initial-sync).


## Changing ignored tags and translation after initial sync

If you change the [ignored tags](#settings) after the initial sync this will not update already synced messages. This means that if a change is made locally on an already synced message the previously ignored remote labels may be deleted. Conversely, if a change occurs remotely on a message which previously which has local tags that were ignored before, these ignored tags may be deleted.
134

135
The best way to deal with this is to do a full push or pull after having changed one of the settings. **Do not change both `--ignore-tags-locally` and `--ignore-tags-remote` at the same time.**
136

137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169
Before changing either setting make sure you are fully synchronized. After changing e.g. `--ignore-tags-remote` do first a dry-run and then a real run of full `gmi pull -f --dry-run`. This will fetch the full tag list for all messages and overwrite the local tags of all your messages with the remote labels.

When changing the opposite setting: `--ignore-tags-local`, do a full push (dry-run first): `gmi push -f --dry-run`.

The same goes for the option `--replace-slash-with-dot`. I prefer to do `gmi pull -f --dry-run` after changing this option. This will overwrite the local tags with the remote labels.

# Translation between labels and tags

We translate some of the GMail labels to other tags. The map of labels to tags are:

```py
  'INBOX'     : 'inbox',
  'SPAM'      : 'spam',
  'TRASH'     : 'trash',
  'UNREAD'    : 'unread',
  'STARRED'   : 'flagged',
  'IMPORTANT' : 'important',
  'SENT'      : 'sent',
  'DRAFT'     : 'draft',
  'CHAT'      : 'chat',

  'CATEGORY_PERSONAL'     : 'personal',
  'CATEGORY_SOCIAL'       : 'social',
  'CATEGORY_PROMOTIONS'   : 'promotions',
  'CATEGORY_UPDATES'      : 'updates',
  'CATEGORY_FORUMS'       : 'forums',
```

# Using your own API key

Gmailieer ships with an API key that is shared openly, this key shares API quota, but [cannot be used to access data](https://github.com/gauteh/gmailieer/pull/9) unless access is gained to your private `access_token` or `refresh_token`.

You can get an [api key](https://console.developers.google.com/flows/enableapi?apiid=gmail) for a CLI application to use for yourself. Store the `client_secret.json` file somewhere safe and specify it to `gmi auth -c`. You can do this on a repository that is already initialized.
170 171


172
# Caveats
173

174
* The GMail API does not let you sync `muted` messages. Until [this Google
175 176
bug](https://issuetracker.google.com/issues/36759067) is fixed, the `mute` and `muted` tags are not synchronized with the remote.

177
* The [`todo`](https://github.com/gauteh/gmailieer/issues/52) and [`voicemail`](https://github.com/gauteh/gmailieer/issues/74) labels seem to be reserved and will be ignored.
178

179 180
* The `draft` and `sent` labels are read only: They are synced from GMail to local notmuch tags, but not back (if you change them via notmuch).

181
* [Only one of the tags](https://github.com/gauteh/gmailieer/issues/26) `inbox`, `spam`, and `trash` may be added to an email. For
182
the time being, `trash` will be prefered over `spam`, and `spam` over inbox.
183

184 185
* `Trash` (capital `T`) is reserved and not allowed, use `trash` (lowercase, see above) to bin messages remotely.

186 187
* Sometimes GMail provides a label identifier on a message for a label that does not exist. If you encounter this [issue](https://github.com/gauteh/gmailieer/issues/48) you can get around it by using `gmi set --drop-non-existing-labels` and re-try to pull. The labels will now be ignored, and if this message is ever synced back up the unmapped label ID will be removed. You can list labels with `gmi pull -t`.

188 189
* You [cannot add any new files](https://github.com/gauteh/gmailieer/issues/54) (files starting with `.` will be ignored) to the gmailieer repository. Gmailieer uses the directory content an index of local files. Gmailieer does not push new messages to your account (note that if you send messages with GMail, GMail automatically adds the message to your mailbox).

190 191
* Make sure that you use the same domain for you GMail account as you initially created your account with: usually `@gmail.com`, but sometimes `@googlemail.com`. Otherwise you might get a [`Delegation denied` error](https://github.com/gauteh/gmailieer/issues/88).