Planet Tor

@blog December 8, 2021 - 00:00 • a day ago
New Release: Tor Browser 11.0.2

Tor Browser 11.0.2 is now available from the Tor Browser download page and also from our distribution directory

This version updates Firefox on Windows, macOS, and Linux to 91.4.0esr. This version includes important security updates to Firefox.

Full changelog

The full changelog since Tor Browser 11.0.1 is:

  • Windows, MacOS & Linux:
    • Update Firefox to 91.4.0esr
    • Bug 40318: Remove check for DISPLAY env var in start-tor-browser
    • Bug 40386: Add new default obfs4 bridge "deusexmachina"
    • Bug 40682: Disable network.proxy.allow_bypass
  • Linux

Known issues

Tor Browser 11.0.2 comes with a number of known issues (please check the following list before submitting a new bug report):

  • Bug 40668: DocumentFreezer & file scheme
  • Bug 40382: Fonts don’t render
  • Bug 40679: Missing features on first-time launch in esr91 on MacOS
  • Bug 40667: AV1 videos shows as corrupt files in Windows 8.1
  • Bug 40666: Switching svg.disable affects NoScript settings
  • Bug 40693: Potential Wayland dependency
  • Bug 40705: “visit our website” link on about:tbupdate pointing to different locations
  • Bug 40706: Fix issue in https-e wasm
...
@blog December 7, 2021 - 00:00 • 2 days ago
Responding to Tor censorship in Russia

Update: Right after we published this article, the Russian government has officially blocked our main website in Russia. Users can circumvent this block by visiting our website mirror.

Since December 1st, some Internet providers in Russia have started to block access to Tor. Today, we've learned that the Federal Service for Supervision of Communications, Information Technology and Mass Media (Roskomnadzor), a Russian government bureaucratic entity, is threatening to censor our main website (torproject.org). Russia is the country with the second largest number of Tor users, with more than 300,000 daily users or 15% of all Tor users. As it seems this situation could quickly escalate to a country-wide Tor block, it's urgent that we respond to this censorship! We need your help NOW to keep Russians connected to Tor!

Run a Tor bridge

Last month we launched the campaign Help Censored Users, Run a Tor Bridge to motivate more volunteers to spin up more bridges. The campaign has been a great success, and we've already achieved our goal of 200 new obfs4 bridges. Today, we have more than 400 new bridges.

But now, if the censorship pattern that we're analyzing in some Russian internet providers is to be deployed country-wide, we will need many more bridges to keep Russians online. Thanks to researchers, we've learned that the default bridges available in Tor Browser aren't working in some places in Russia - this includes Snowflake bridges and obfs4 bridges obtained dynamically using Moat. Russian users need to follow our guide to use bridges that are not blocked.

We are calling on everyone to spin up a Tor bridge! If you've ever considered running a bridge, now is an excellent time to get started, as your help is urgently needed. You can find the requirements and instructions for starting a bridge in the Help Censored Users, Run a Tor Bridge blog post.

We need the support of the Internet Freedom community

Teach users about Tor bridges

Digital security trainers and internet freedom advocates, your help is needed! As this instance of censorship limits direct access to our website, malicious actors could start phishing users with fake Tor Browsers or spreading disinformation about Tor. Teaching users how to bypass censorship and how to get the official Tor Browser version using GetTor or a mirror will be crucial. We need you help spread accurate information about Tor and Tor bridges, particularly among Russian audiences.

Localize Tor

We have an extremely helpful and responsive Russian translator community, but we urgently need more volunteers. Learn how to become a Tor translator and join Tor's localization IRC channel or use Element to connect to (#tor-l10n:matrix.org).

Document internet censorship

Russian users can help us see how the Russian government is censoring the internet by running the OONI probe app on their mobile or desktop devices. OONI, the Open Observatory of Network Interference, will test if and how Tor is being blocked by your internet provider. After installing, please run the "Circumvention test", which will check if circumvention tools like Tor are blocked. Internet measurements are important for detection of anomalous activities; a volunteer running the OONI probe and discussing results with the Tor community was how we discovered the current censorship in Russia.

Apply pressure

International digital rights and human rights organizations must pressure Russia's government to immediately revert this censorship.

We will update this post if the situation changes. To receive a notification for updates, you can subscribe to our new Forum and click on the bell icon.

...
@ooni December 1, 2021 - 00:00 • 8 days ago
[Event Report] India, Let's Build the List
This is a guest post by The Bachchao Project, originally published here. The Bachchao Project in partnership with OONI hosted an online event on 9th and 10th October 2021 to update the Citizen Lab test list for India. The event, which was called “India, Lets build the list”, was organised to help strengthen community based monitoring of internet censorship in India. The event allowed experts from different fields to contribute to a curated list of websites that are relevant to India and which are regularly tested for censorship by volunteers in India. ...
@blog November 30, 2021 - 00:00 • 9 days ago
Privacy-Preserving and Incrementally-Deployable Support for Certificate Transparency in Tor
This is a guest post by Rasmus Dahlberg, Tobias Pulls, Tom Ritter, and Paul Syverson.

The Why of Certificate Transparency

There are many things that the web could be better at. One part relates to transparent management of TLS certificates. In case you are not familiar with certificates, websites present them to visitors in an attempt to prove their identities. For example, "I'm www.torproject.org and not some imposter".

The problem is that certificates can be issued by many different central authorities. If one of these authorities gets the issuance process wrong, e.g. due to mistakes, coercion, or compromise, there may be a mis-issued certificate for some domain name. A mis-issued certificate can be used by attackers to impersonate websites. This is obviously not great. In the context of Tor it is also easy for an attacker to run an exit to be in a perfect position to perform such attacks. Tor has been shipping the HTTPS Everywhere extension with Tor Browser for some time to stop any attackers at exit relays who try to hijack unencrypted connections. Although this is a valuable protection, it will not stop an attacker with access to a mis-issued certificate.

The goal of Certificate Transparency is to ensure that certificate mis-issuance does not go unnoticed. The idea is that before a browser accepts a certificate as valid, it must be visible in a public Certificate Transparency log. This is excellent, because now anyone can inspect the logs to see for themselves whether there are any mis-issued certificates. If something bad shows up, one can act on that accordingly.

Although the basic idea of Certificate Transparency is simple, the exact instantiation turns out to be less straightforward in the real-world. Should the public logs be parties that you trust blindly? Are there any privacy concerns? What about backwards-compatibility? These are all valid questions that we considered for Tor in particular.

The How in Tor

Like other browsers that already enforce Certificate Transparency partially, we used trusted logs as a starting point for our addition to Tor Browser. It is not ideal, but much better than what existed before. This reduces the attack surface from hundreds of trusted central authorities that issue certificates to a handful of Certificate Transparency logs--- a significant win for the web!

Next, we showed how to relax these trust assumptions gradually while taking advantage of Tor Browser and the Tor network to preserve privacy. This would be a larger challenge for any browser that cannot leverage an anonymity network with thousands of relays across the globe. For example, to do better than simply having trusted logs available, you need to verify that public logging actually took place. That verification currently requires external interaction with other parties, effectively leaking browsing history to whomever you interact with. In contrast, our incremental designs let Tor relays do that verification for you. Your privacy is preserved by Tor, and aggregate leakages from Tor are reduced via caching.

For more detail we refer the interested reader to our paper and presentation.

Next steps include the following:

  • Complete partial Certificate Transparency enforcement in Firefox. This would bring Certificate Transparency with trusted logs to both Firefox and Tor Browser.
  • Create and implement a torspec proposal that uses Tor relays to increase your confidence that public logging actually happened when using Tor Browser.

Conclusion

It is exciting to see that Certificate Transparency can be deployed safely in Tor without being restricted to a weak attacker. Our incremental design considers an attacker that controls a fraction of Tor relays, at least one central authority that issues certificates, and all but one Certificate Transparency log. Our full design relaxes these trust assumptions further by allowing the attacker to control all Certificate Transparency logs.

...
@ooni November 30, 2021 - 00:00 • 9 days ago
Why Collaboration and Transparency is Key to Internet Measurement
This post was originally published on the Internet Society Pulse blog. With Internet shutdowns, disruptions and censorship events on the increase around the world, tracking where such events are happening and gathering evidence to help in the fight against them is becoming more and more important. Tracking these events is crucial because of the impact they have on society and the economy. When social media apps are blocked, for example, freedom of speech, access to information, and movement-building is hampered. ...
@anarcat November 21, 2021 - 16:04 • 18 days ago
The last syncmaildir crash

My syncmaildir (SMD) setup failed me one too many times (previously, previously). In an attempt to migrate to an alternative mail synchronization tool, I looked into using my IMAP server again, and found out my mail spool was in a pretty bad shape. I'm comparing mbsync and offlineimap in the next post but this post talks about how I recovered the mail spool so that tools like those could correctly synchronise the mail spool again.

The latest crash

On Monday, SMD just started failing with this error:

nov 15 16:12:19 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:12:22 angela systemd[2305]: smd-pull.service: Succeeded.
nov 15 16:12:22 angela systemd[2305]: Finished pull emails with syncmaildir.
nov 15 16:14:08 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:14:11 angela systemd[2305]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
nov 15 16:14:11 angela systemd[2305]: smd-pull.service: Failed with result 'exit-code'.
nov 15 16:14:11 angela systemd[2305]: Failed to start pull emails with syncmaildir.
nov 15 16:16:14 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Network error.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Unable to get any data from the other endpoint.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: This problem may be transient, please retry.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Hint: did you correctly setup the SERVERNAME variable
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: on your client? Did you add an entry for it in your ssh
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: configuration file?
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Network error
nov 15 16:16:17 angela smd-pull[27188]: register: smd-client@localhost: TAGS: error::context(handshake) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
nov 15 16:16:17 angela systemd[2305]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
nov 15 16:16:17 angela systemd[2305]: smd-pull.service: Failed with result 'exit-code'.
nov 15 16:16:17 angela systemd[2305]: Failed to start pull emails with syncmaildir.

What is frustrating is that there's actually no network error here. Running the command by hand I did see a different message, but now I have lost it in my backlog. It had something to do with a filename being too long, and I gave up debugging after a while. This happened suddenly too, which added to the confusion.

In a fit of rage I started this blog post and experimenting with alternatives, which led me down a lot of rabbit holes.

Reviewing my previous mail crash documentation, it seems most solutions involve talking to an IMAP server, so I figured I would just do that. Wanting to try something new, i gave isync (AKA mbsync) a try. Oh dear, I did not expect how much trouble just talking to my IMAP server would be, which wasn't not isync's fault, for what that's worth. It was the primary tool I used to debug things, and served me well in that regard.

Mailbox corruption

The first thing I found out is that certain messages in the IMAP spool were corrupted. mbsync would stop on a FETCH command and Dovecot would give me those errors on the server side.

"wrong W value"

nov 16 15:31:27 marcos dovecot[3621800]: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Maildir filename has wrong W value, renamed the file from /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S to /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495:2,S
nov 16 15:31:27 marcos dovecot[3621800]: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Deleting corrupted cache record uid=1582: UID 1582: Broken virtual size in mailbox junk: read(/home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S): FETCH BODY[] got too little data: 2540 vs 2578

At least this first error was automatically healed by Dovecot (by renaming the file without the W= flag). The problem is that the FETCH command fails and mbsync exits noisily. So you need to constantly restart mbsync with a silly command like:

while ! mbsync -a; do sleep 1; done

"cached message size larger than expected"

nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=mail stream)
nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: Deleting corrupted cache record uid=19288: UID 19288: Broken physical size in mailbox Sent: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288)
nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=)
nov 16 13:53:08 marcos dovecot[3520770]: imap-login: Panic: epoll_ctl(del, 7) failed: Bad file descriptor

This second problem is much harder to fix, because dovecot does not recover automatically. This is Dovecot complaining that the cached size (the S= field, but also present in Dovecot's metadata files) doesn't match the file size.

I wonder if at least some of those messages were corrupted in the OfflineIMAP to syncmaildir migration because part of that procedure is to run the strip_header script to remove content from the emails. That could easily have broken things since the files do not also get renamed.

Workaround

So I read a lot of the Dovecot documentation on the maildir format, and wrote an extensive fix script for those two errors. The script worked and mbsync was able to sync the entire mail spool.

And no, rebuilding the index files didn't work. Also tried doveadm force-resync -u anarcat which didn't do anything.

In the end I also had to do this, because the wrong cache values were also stored elsewhere.

service dovecot stop ; find -name 'dovecot*' -delete; service dovecot start

This would have totally broken any existing clients, but thankfully I'm starting from scratch (except maybe webmail, but I'm hoping it will self-heal as well, assuming it only has a cache and not a full replica of the mail spool).

Incoherence between Maildir and IMAP

Unfortunately, the first mbsync was incomplete as it was missing about 15,000 mails:

anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' | wc -l 
384836
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' | wc -l 
369221

As it turns out, mbsync was not at fault here either: this was yet more mail spool corruption.

It's actually 26 folders (out of 205) with inconsistent sizes, which can be found with:

for folder in * .[^.]* ; do 
  printf "%s\t%d\n" $folder $(find "$folder" -type f -a \! -name '.*' | wc -l );
done

The special \! -name '.*' bit is to ignore the mbsync metadata, which creates .uidvalidity and .mbsyncstate in every folder. That ignores about 200 files but since they are spread around all folders, which was making it impossible to review where the problem was.

Here is what the diff looks like:

--- Maildir-list    2021-11-17 20:42:36.504246752 -0500
+++ Maildir-mbsync-list 2021-11-17 20:18:07.731806601 -0500
@@ -6,16 +6,15 @@
[...]
 .Archives  1
 .Archives.2010 3553
-.Archives.2011 3583
-.Archives.2012 12593
+.Archives.2011 3582
+.Archives.2012 620
 .Archives.2013 8576
 .Archives.2014 11057
-.Archives.2015 8173
+.Archives.2015 8165
 .Archives.2016 54
 .band  34
 .bitbuck   1
@@ -38,13 +37,12 @@
 .couchsurfers  2
-cur    11285
+cur    11280
 .current   130
 .cv    2
 .debbug    262
-.debian    37544
-drafts 1
-.Drafts    4
+.debian    37533
+.Drafts    2
 .drone 241
 .drupal    188
 .drupal-devel  303
[...]

Misfiled messages

It's a bit all over the place, but we can already notice some huge differences between mailboxes, for example in the Archives folders. As it turns out, at least 12,000 of those missing mails were actually misfiled: instead of being in the Maildir/.Archives.2012/cur/ folder, they were directly in Maildir/.Archives.2012/. This is something that doesn't matter for SMD (and possibly for notmuch? it does matter, notmuch suddenly found 12,000 new mails) but that definitely matters to Dovecot and therefore mbsync...

After moving those files around, we still have 4,000 message missing:

anarcat@angela:~(main)$ find Maildir-mbsync/  -type f -a \! -name '.*' | wc -l 
381196
anarcat@angela:~(main)$ find Maildir/  -type f -a \! -name '.*' | wc -l 
385053

The problem is that those 4,000 missing mails are harder to track. Take, for example, .Archives.2011, which has a single message missing, out of 3,582. And the files are not identical: the checksums don't match after going through the IMAP transport, so we can't use a tool like hashdeep to compare the trees and find why any single file is missing.

"register" folder

One big chunk of the 4,000, however, is a special folder called register in my spool, which I am syncing separately (see Securing registration email for details on that setup). That actually covers 3,700 of those messages, so I actually have a more modest 300 messages to figure out, after (easily!) configuring mbsync to sync that folder separately:

 @@ -30,9 +33,29 @@ Slave :anarcat-local:
  # Exclude everything under the internal [Gmail] folder, except the interesting folders
  #Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
  # Or include everything
 -Patterns *
 +#Patterns *
 +Patterns * !register  !.register
  # Automatically create missing mailboxes, both locally and on the server
  #Create Both
  Create slave
  # Sync the movement of messages between folders and deletions, add after making sure the sync works
  #Expunge Both
 +
 +IMAPAccount anarcat-register
 +Host imap.anarc.at
 +User register
 +PassCmd "pass imap.anarc.at-register"
 +SSLType IMAPS
 +CertificateFile /etc/ssl/certs/ca-certificates.crt
 +
 +IMAPStore anarcat-register-remote
 +Account anarcat-register
 +
 +MaildirStore anarcat-register-local
 +SubFolders Maildir++
 +Inbox ~/Maildir-mbsync/.register/
 +
 +Channel anarcat-register
 +Master :anarcat-register-remote:
 +Slave :anarcat-register-local:
 +Create slave

"tmp" folders and empty messages

After syncing the "register" messages, I end up with the measly little 160 emails out of sync:

anarcat@angela:~(main)$ find Maildir-mbsync/  -type f -a \! -name '.*' | wc -l 
384900
anarcat@angela:~(main)$ find Maildir/  -type f -a \! -name '.*' | wc -l 
385059

Argh. After more digging, I have found 131 mails in the tmp/ directories of the client's mail spool. Mysterious! On the server side, it's even more files, and not the same ones. Possible that those were mails that were left there during a failed delivery of some sort, during a power failure or some sort of crash? Who knows. It could be another race condition in SMD if it runs while mail is being delivered in tmp/...

The first thing to do with those is to cleanup a bunch of empty files (21 on angela):

find .[^.]*/tmp -type f -empty -delete

As it turns out, they are all duplicates, in the sense that notmuch can easily find a copy of files with the same message ID in its database. In other words, this hairy command returns nothing

find .[^.]*/tmp -type f | while read path; do
  msgid=$(grep -m 1  -i ^message-id "$path" | sed 's/Message-ID: //i;s/[<>]//g');
  if notmuch count --exclude=false  "id:$msgid" | grep -q 0; then
    echo "$path <$msgid> not in notmuch" ;
  fi;
done

... which is good. Or, to put it another way, this is safe:

find .[^.]*/tmp -type f -delete

Poof! 314 mails cleaned on the server side. Interestingly, SMD doesn't pick up on those changes at all and still sees files in tmp/ directories on the client side, so we need to operate the same twisted logic there.

notmuch to the rescue again

After cleaning that on the client, we get:

anarcat@angela:~(main)$ find Maildir/  -type f -a \! -name '.*' | wc -l 
384928
anarcat@angela:~(main)$ find Maildir-mbsync/  -type f -a \! -name '.*' | wc -l 
384901

Ha! 27 mails difference. Those are the really sticky, unclear ones. I was hoping a full sync might clear that up, but after deleting the entire directory and starting from scratch, I end up with:

anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' | wc -l 
385034
anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' | wc -l 
384993

That is: even more messages missing (now 37). Sigh.

Thankfully, this is something notmuch can help with: it can index all files by Message-ID (which I learned is case-insensitive, yay) and tell us which messages don't make it through.

Considering the corruption I found in the mail spool, I wouldn't be the least surprised those messages are just skipped by the IMAP server. Unfortunately, there's nothing on the Dovecot server logs that would explain the discrepancy.

Here again, notmuch comes to the rescue. We can list all message IDs to figure out that discrepancy:

notmuch search --exclude=false --output=messages '*' | pv -s 18M | sort > Maildir-msgids
notmuch --config=.notmuch-config-mbsync search --exclude=false --output=messages '*' | pv -s 18M | sort > Maildir-mbsync-msgids

And then we can see how many messages notmuch thinks are missing:

$ wc -l *msgids
372723 Maildir-mbsync-msgids
372752 Maildir-msgids

That's 29 messages. Oddly, it doesn't exactly match the find output:

anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' | wc -l 
385204
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' | wc -l 
385241

That is 10 more messages. Ugh. But actually, I know what those are: more misfiled messages (in a .folder/draft/ directory, bizarrely, so the totals actually match.

In the notmuch output, there's a lot of stuff like this:

id:notmuch-sha1-fb880d673e24f5dae71b6b4d825d4a0d5d01cde4

Those are messages without a valid Message-ID. Notmuch (presumably) constructs one based on the file's checksum. Because the files differ between the IMAP server and the local mail spool (which is unfortunate, but possibly inevitable), those do not match. There are exactly the same number of those on both sides, so I'll go ahead and assume those are all accounted for.

What remains is:

anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids  | grep '^\-[^-]' | grep -v sha1 | wc -l 
2
anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids  | grep '^\+[^+]' | grep -v sha1 | wc -l 
21
anarcat@angela:~(main)$ 

ie. 21 missing from mbsync, and, surprisingly, 2 missing from the original mail spool.

Further inspection also showed they were all messages with some sort of "corruption": no body and only headers. I am not sure that is a legal email format in the first place. Since they were mostly spam or administrative emails ("You have been unsubscribed from mailing list..."), it seems fairly harmless to ignore those.

Conclusion

As we'll see in the next article, SMD has stellar performance. But that comes at a huge cost: it accesses the mail storage directly. This can (and has) created significant problems on the mail server. It's unclear exactly why those things happen, but Dovecot expects a particular storage format on its file, and it seems unwise to bypass that.

In the future, I'll try to remember to avoid that, especially since mechanisms like SMD require special server access (SSH) which, in the long term, I am not sure I want to maintain or expect.

In other words, just talking with an IMAP server opens up a lot more possibilities of hosting than setting up a custom synchronisation protocol over SSH. It's also safer and more reliable, as we have seen. Thankfully, I've been able to recover from all the errors I could find, but it could have gone differently and it would have been possible for SMD to permanently corrupt significant part of my mail archives.

In the end, however, the last drop was just another weird bug which, ironically, SMD mysteriously recovered from on its own while I was writing this documentation and migrating away from it.

In any case, I recommend SMD users start looking for alternatives. The project has been archived upstream, and the Debian package has been orphaned. I have seen significant mail box corruption, including entire mail spool destruction, mostly due to incorrect locking code. I have filed a release-critical bug in Debian to make sure it doesn't ship with Debian bookworm.

Alternatives like mbsync provide fast and reliable transport, including over SSH. See the next article for further discussion of the alternatives.

...
@anarcat November 21, 2021 - 16:04 • 18 days ago
mbsync vs OfflineIMAP

After recovering from my latest email crash (previously, previously), I had to figure out which tool I should be using. I had many options but I figured I would start with a popular one (mbsync).

But I also evaluated OfflineIMAP which was resurrected from the Python 2 apocalypse, and because I had used it before, for a long time.

Read on for the details.

Benchmark setup

All programs were tested against a Dovecot 1:2.3.13+dfsg1-2 server, running Debian bullseye.

The client is a Purism 13v4 laptop with a Samsung SSD 970 EVO 1TB NVMe drive.

The server is a custom build with a AMD Ryzen 5 2600 CPU, and a RAID-1 array made of two NVMe drives (Intel SSDPEKNW010T8 and WDC WDS100T2B0C).

The mail spool I am testing against has almost 400k messages and takes 13GB of disk space:

$ notmuch count --exclude=false
372758
$ du -sh --exclude xapian Maildir
13G Maildir

The baseline we are comparing against is SMD (syncmaildir) which performs the sync in about 7-8 seconds locally (3.5 seconds for each push/pull command) and about 10-12 seconds remotely.

Anything close to that or better is good enough. I do not have recent numbers for a SMD full sync baseline, but the setup documentation mentions 20 minutes for a full sync. That was a few years ago, and the spool has obviously grown since then, so that is not a reliable baseline.

A baseline for a full sync might be also set with rsync, which copies files at nearly 40MB/s, or 317Mb/s!

anarcat@angela:tmp(main)$ time rsync -a --info=progress2 --exclude xapian  shell.anarc.at:Maildir/ Maildir/
 12,647,814,731 100%   37.85MB/s    0:05:18 (xfr#394981, to-chk=0/395815)    
72.38user 106.10system 5:19.59elapsed 55%CPU (0avgtext+0avgdata 15988maxresident)k
8816inputs+26305112outputs (0major+50953minor)pagefaults 0swaps

That is 5 minutes to transfer the entire spool. Incremental syncs are obviously pretty fast too:

anarcat@angela:tmp(main)$ time rsync -a --info=progress2 --exclude xapian  shell.anarc.at:Maildir/ Maildir/
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/395815)    
1.42user 0.81system 0:03.31elapsed 67%CPU (0avgtext+0avgdata 14100maxresident)k
120inputs+0outputs (3major+12709minor)pagefaults 0swaps

As an extra curiosity, here's the performance with tar, pretty similar with rsync, minus incremental which I cannot be bothered to figure out right now:

anarcat@angela:tmp(main)$ time ssh shell.anarc.at tar --exclude xapian -cf - Maildir/ | pv -s 13G | tar xf - 
56.68user 58.86system 5:17.08elapsed 36%CPU (0avgtext+0avgdata 8764maxresident)k
0inputs+0outputs (0major+7266minor)pagefaults 0swaps
12,1GiO 0:05:17 [39,0MiB/s] [===================================================================> ] 92%

Interesting that rsync manages to almost beat a plain tar on file transfer, I'm actually surprised by how well it performs here, considering there are many little files to transfer.

(But then again, this maybe is exactly where rsync shines: while tar needs to glue all those little files together, rsync can just directly talk to the other side and tell it to do live changes. Something to look at in another article maybe?)

Since both ends are NVMe drives, those should easily saturate a gigabit link. And in fact, a backup of the server mail spool achieves much faster transfer rate on disks:

anarcat@marcos:~$ tar fc - Maildir | pv -s 13G > Maildir.tar
15,0GiO 0:01:57 [ 131MiB/s] [===================================] 115%

That's 131Mibyyte per second, vastly faster than the gigabit link. The client has similar performance:

anarcat@angela:~(main)$ tar fc - Maildir | pv -s 17G > Maildir.tar
16,2GiO 0:02:22 [ 116MiB/s] [==================================] 95%

So those disks should be able to saturate a gigabit link, and they are not the bottleneck on fast links. Which begs the question of what is blocking performance of a similar transfer over the gigabit link, but that's another question altogether, because no sync program ever reaches the above performance anyways.

Finally, note that when I migrated to SMD, I wrote a small performance comparison that could be interesting here. It show SMD to be faster than OfflineIMAP, but not as much as we see here. In fact, it looks like OfflineIMAP slowed down significantly since then (May 2018), but this could be due to my larger mail spool as well.

mbsync

The isync (AKA mbsync) project is written in C and supports syncing Maildir and IMAP folders, with possibly multiple replicas. I haven't tested this but I suspect it might be possible to sync between two IMAP servers as well. It supports partial mirorrs, message flags, full folder support, and "trash" functionality.

Complex configuration file

I started with this .mbsyncrc configuration file:

SyncState *
Sync New ReNew Flags

IMAPAccount anarcat
Host imap.anarc.at
User anarcat
PassCmd "pass imap.anarc.at"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore anarcat-remote
Account anarcat

MaildirStore anarcat-local
# Maildir/top/sub/sub
#SubFolders Verbatim
# Maildir/.top.sub.sub
SubFolders Maildir++
# Maildir/top/.sub/.sub
# SubFolders legacy
# The trailing "/" is important
#Path ~/Maildir-mbsync/
Inbox ~/Maildir-mbsync/

Channel anarcat
# AKA Far, convert when all clients are 1.4+
Master :anarcat-remote:
# AKA Near
Slave :anarcat-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
Patterns *
# Automatically create missing mailboxes, both locally and on the server
#Create Both
Create slave
# Sync the movement of messages between folders and deletions, add after making sure the sync works
#Expunge Both

Long gone are the days where I would spend a long time reading a manual page to figure out the meaning of every option. If that's your thing, you might like this one. But I'm more of a "EXAMPLES section" kind of person now, and I somehow couldn't find a sample file on the website. I started from the Arch wiki one but it's actually not great because it's made for Gmail (which is not a usual Dovecot server). So a sample config file in the manpage would be a great addition. Thankfully, the Debian packages ships one in /usr/share/doc/isync/examples/mbsyncrc.sample but I only found that after I wrote my configuration. It was still useful and I recommend people take a look if they want to understand the syntax.

Also, that syntax is a little overly complicated. For example, Far needs colons, like:

Far :anarcat-remote:

Why? That seems just too complicated. I also found that sections are not clearly identified: IMAPAccount and Channel mark section beginnings, for example, which is not at all obvious until you learn about mbsync's internals. There are also weird ordering issues: the SyncState option needs to be before IMAPAccount, presumably because it's global.

Using a more standard format like .INI or TOML could improve that situation.

Stellar performance

A transfer of the entire mail spool takes 56 minutes and 6 seconds, which is impressive.

It's not quite "line rate": the resulting mail spool was 12GB (which is a problem, see below), which turns out to be about 29Mbit/s and therefore not maxing the gigabit link, and an order of magnitude slower than rsync.

The incremental runs are roughly 2 seconds, which is even more impressive, as that's actually faster than rsync:

===> multitime results
1: mbsync -a
            Mean        Std.Dev.    Min         Median      Max
real        2.015       0.052       1.930       2.029       2.105       
user        0.660       0.040       0.592       0.661       0.722       
sys         0.338       0.033       0.268       0.341       0.387    

Those tests were performed with isync 1.3.0-2.2 on Debian bullseye. Tests with a newer isync release originally failed because of a corrupted message that triggered bug 999804 (see below). Running 1.4.3 under valgrind works around the bug, but adds a 50% performance cost, the full sync running in 1h35m.

Once the upstream patch is applied, performance with 1.4.3 is fairly similar, considering that the new sync included the register folder with 4000 messages:

120.74user 213.19system 59:47.69elapsed 9%CPU (0avgtext+0avgdata 105420maxresident)k
29128inputs+28284376outputs (0major+45711minor)pagefaults 0swaps

That is ~13GB in ~60 minutes, which gives us 28.3Mbps. Incrementals are also pretty similar to 1.3.x, again considering the double-connect cost:

===> multitime results
1: mbsync -a
            Mean        Std.Dev.    Min         Median      Max
real        2.500       0.087       2.340       2.491       2.629       
user        0.718       0.037       0.679       0.711       0.793       
sys         0.322       0.024       0.284       0.320       0.365

Those tests were all done on a Gigabit link, but what happens on a slower link? My server uplink is slow: 25 Mbps down, 6 Mbps up. There mbsync is worse than the SMD baseline:

===> multitime results
1: mbsync -a
Mean        Std.Dev.    Min         Median      Max
real        31.531      0.724       30.764      31.271      33.100      
user        1.858       0.125       1.721       1.818       2.131       
sys         0.610       0.063       0.506       0.600       0.695       

That's 30 seconds for a sync, which is an order of magnitude slower than SMD.

Great user interface

Compared to OfflineIMAP and (ahem) SMD, the mbsync UI is kind of neat:

anarcat@angela:~(main)$ mbsync -a
Notice: Master/Slave are deprecated; use Far/Near instead.
C: 1/2  B: 204/205  F: +0/0 *0/0 #0/0  N: +1/200 *0/0 #0/0

(Note that nice switch away from slavery-related terms too.)

The display is minimal, and yet informative. It's not obvious what does mean at first glance, but the manpage is useful at least for clarifying that:

This represents the cumulative progress over channels, boxes, and messages affected on the far and near side, respectively. The message counts represent added messages, messages with updated flags, and trashed messages, respectively. No attempt is made to calculate the totals in advance, so they grow over time as more information is gathered. (Emphasis mine).

In other words:

  • C 2/2: channels done/total (2 done out of 2)
  • B 204/205: mailboxes done/total (204 out of 205)
  • F: changes on the far side
  • N: +10/200 *0/0 #0/0: changes on the "near" side:
    • +10/200: 10 out of 200 messages downloaded
    • *0/0: no flag changed
    • #0/0: no message deleted

You get used to it, in a good way. It does not, unfortunately, show up when you run it in systemd, which is a bit annoying as I like to see a summary mail traffic in the logs.

Interoperability issue

In my notmuch setup, I have bound key S to "mark spam", which basically assigns the tag spam to the message and removes a bunch of others. Then I have a notmuch-purge script which moves that message to the spam folder, for training purposes. It basically does this:

notmuch search --output=files --format=text0 "$search_spam" \
    | xargs -r -0 mv -t "$HOME/Maildir/${PREFIX}junk/cur/"

This method, which worked fine in SMD (and also OfflineIMAP) created this error on sync:

Maildir error: duplicate UID 37578.

And indeed, there are now two messages with that UID in the mailbox:

anarcat@angela:~(main)$ find Maildir/.junk/ -name '*U=37578*'
Maildir/.junk/cur/1637427889.134334_2.angela,U=37578:2,S
Maildir/.junk/cur/1637348602.2492889_221804.angela,U=37578:2,S

This is actually a known limitation or, as mbsync(1) calls it, a "RECOMMENDATION":

When using the more efficient default UID mapping scheme, it is important that the MUA renames files when moving them between Maildir fold ers. Mutt always does that, while mu4e needs to be configured to do it:

(setq mu4e-change-filenames-when-moving t)

So it seems I would need to fix my script. It's unclear how the paths should be renamed, which is unfortunate, because I would need to change my script to adapt to mbsync, but I can't tell how just from reading the above.

(A manual fix is actually to rename the file to remove the U= field: mbsync will generate a new one and then sync correctly.)

Fortunately, someone else already fixed that issue: afew, a notmuch tagging script (much puns, such hurt), has a move mode that can rename files correctly, specifically designed to deal with mbsync. I had already been told about afew, but it's one more reason to standardize my notmuch hooks on that project, it looks like.

Update: I have tried to use afew and found it has significant performance issues. It also has a completely different paradigm to what I am used to: it assumes all incoming mail has a new and lays its own tags on top of that (inbox, sent, etc). It can only move files from one folder at a time (see this bug) which breaks my spam training workflow. In general, I sync my tags into folders (e.g. ham, spam, sent) and message flags (e.g. inbox is F, unread is "not S", etc), and afew is not well suited for this (although there are hacks that try to fix this). I have worked hard to make my tagging scripts idempotent, and it's something afew doesn't currently have. Still, it would be better to have that code in Python than bash, so maybe I should consider my options here.

Stability issues

The newer release in Debian bookworm (currently at 1.4.3) has stability issues on full sync. I filed bug 999804 in Debian about this, which lead to a thread on the upstream mailing list. I have found at least three distinct crashes that could be double-free bugs "which might be exploitable in the worst case", not a reassuring prospect.

The thing is: mbsync is really fast, but the downside of that is that it's written in C, and with that comes a whole set of security issues. The Debian security tracker has only three CVEs on isync, but the above issues show there could be many more.

Reading the source code certainly did not make me very comfortable with trusting it with untrusted data. I considered sandboxing it with systemd (below) but having systemd run as a --user process makes that difficult. I also considered using an apparmor profile but that is not trivial because we need to allow SSH and only some parts of it...

Thankfully, upstream has been diligent at addressing the issues I have found. They provided a patch within a few days which did fix the sync issues.

Update: upstream actually took the issue very seriously. They not only got CVE-2021-44143 assigned for my bug report, they also audited the code and found several more issues collectively identified as CVE-2021-3657, which actually also affect 1.3 (ie. Debian 11/bullseye/stable). Somehow my corpus doesn't trigger that issue, but it was still considered serious enough to warrant a CVE. So one the one hand: excellent response from upstream; but on the other hand: how many more of those could there be in there?

Automation with systemd

The Arch wiki has instructions on how to setup mbsync as a systemd service. It suggests using the --verbose (-V) flag which is a little intense here, as it outputs 1444 lines of messages.

I have used the following .service file:

[Unit]
Description=Mailbox synchronization service
ConditionHost=!marcos
Wants=network-online.target
After=network-online.target
Before=notmuch-new.service

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync -a
Nice=10
IOSchedulingClass=idle
NoNewPrivileges=true

[Install]
WantedBy=default.target

And the following .timer:

[Unit]
Description=Mailbox synchronization timer
ConditionHost=!marcos

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service

[Install]
WantedBy=timers.target

Note that we trigger notmuch through systemd, with the Before and also by adding mbsync.service to the notmuch-new.service file:

[Unit]
Description=notmuch new
After=mbsync.service

[Service]
Type=oneshot
Nice=10
ExecStart=/usr/bin/notmuch new

[Install]
WantedBy=mbsync.service

An improvement over polling repeatedly with a .timer would be to wake up only on IMAP notify, but neither imapnotify nor goimapnotify seem to be packaged in Debian. It would also not cover for the "sent folder" use case, where we need to wake up on local changes.

Password-less setup

The sample file suggests this should work:

IMAPStore remote
Tunnel "ssh -q host.remote.com /usr/sbin/imapd"

Add BatchMode, restrict to IdentitiesOnly, provide a password-less key just for this, add compression (-C), find the Dovecot imap binary, and you get this:

IMAPAccount anarcat-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C anarcat@imap.anarc.at /usr/lib/dovecot/imap"

And it actually seems to work:

$ mbsync -a
Notice: Master/Slave are deprecated; use Far/Near instead.
C: 0/2  B: 0/1  F: +0/0 *0/0 #0/0  N: +0/0 *0/0 #0/0imap(anarcat): Error: net_connect_unix(/run/dovecot/stats-writer) failed: Permission denied
C: 2/2  B: 205/205  F: +0/0 *0/0 #0/0  N: +1/1 *3/3 #0/0imap(anarcat)<1611280><90uUOuyElmEQlhgAFjQyWQ>: Info: Logged out in=10808 out=15396642 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=1 body_bytes=8087

It's a bit noisy, however. dovecot/imap doesn't have a "usage" to speak of, but even the source code doesn't hint at a way to disable that Error message, so that's unfortunate. That socket is owned by root:dovecot so presumably Dovecot runs the imap process as $user:dovecot, which we can't do here. Oh well?

Interestingly, the SSH setup is not faster than IMAP.

With IMAP:

===> multitime results
1: mbsync -a
            Mean        Std.Dev.    Min         Median      Max
real        2.367       0.065       2.220       2.376       2.458       
user        0.793       0.047       0.731       0.776       0.871       
sys         0.426       0.040       0.364       0.434       0.476

With SSH:

===> multitime results
1: mbsync -a
            Mean        Std.Dev.    Min         Median      Max
real        2.515       0.088       2.274       2.532       2.594       
user        0.753       0.043       0.645       0.766       0.804       
sys         0.328       0.045       0.212       0.340       0.393

Basically: 200ms slower. Tolerable.

Migrating from SMD

The above was how I migrated to mbsync on my first workstation. The work on the second one was more streamlined, especially since the corruption on mailboxes was fixed:

  1. install isync, with the patch:

    dpkg -i isync_1.4.3-1.1~_amd64.deb
    
  2. copy all files over from previous workstation to avoid a full resync (optional):

    rsync -a --info=progress2 angela:Maildir/ Maildir-mbsync/
    
  3. rename all files to match new hostname (optional):

    find Maildir-mbsync/ -type f -name '*.angela,*' -print0 |  rename -0 's/\.angela,/\.curie,/'
    
  4. trash the notmuch database (optional):

    rm -rf Maildir-mbsync/.notmuch/xapian/
    
  5. disable all smd and notmuch services:

    systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer
    
  6. do one last sync with smd:

    smd-pull --show-tags ; smd-push --show-tags ; notmuch new ; notmuch-sync-flagged -v
    
  7. backup notmuch on the client and server:

    notmuch dump | pv > notmuch.dump
    
  8. backup the maildir on the client and server:

    cp -al Maildir Maildir-bak
    
  9. create the SSH key:

    ssh-keygen -t ed25519 -f .ssh/id_ed25519_mbsync
    cat .ssh/id_ed25519_mbsync.pub
    
  10. add to .ssh/authorized_keys on the server, like this:

    command="/usr/lib/dovecot/imap",restrict ssh-ed25519 AAAAC...

  11. move old files aside, if present:

    mv Maildir Maildir-smd
    
  12. move new files in place (CRITICAL SECTION BEGINS!):

    mv Maildir-mbsync Maildir
    
  13. run a test sync, only pulling changes:

    mbsync --create-near --remove-none --expunge-none --noop anarcat-register

  14. if that works well, try with all mailboxes:

    mbsync --create-near --remove-none --expunge-none --noop -a

  15. if that works well, try again with a full sync:

    mbsync register mbsync -a

  16. reindex and restore the notmuch database, this should take ~25 minutes:

    notmuch new
    pv notmuch.dump | notmuch restore
    
  17. enable the systemd services and retire the smd-* services:

    systemctl --user enable mbsync.timer notmuch-new.service systemctl --user start mbsync.timer rm ~/.config/systemd/user/smd* systemctl daemon-reload

During the migration, notmuch helpfully told me the full list of those lost messages:

[...]
Warning: cannot apply tags to missing message: CAN6gO7_QgCaiDFvpG3AXHi6fW12qaN286+2a7ERQ2CQtzjSEPw@mail.gmail.com
Warning: cannot apply tags to missing message: CAPTU9Wmp0yAmaxO+qo8CegzRQZhCP853TWQ_Ne-YF94MDUZ+Dw@mail.gmail.com
Warning: cannot apply tags to missing message: F5086003-2917-4659-B7D2-66C62FCD4128@gmail.com
[...]
Warning: cannot apply tags to missing message: mailman.2.1316793601.53477.sage-members@mailman.sage.org
Warning: cannot apply tags to missing message: mailman.7.1317646801.26891.outages-discussion@outages.org
Warning: cannot apply tags to missing message: notmuch-sha1-000458df6e48d4857187a000d643ac971deeef47
Warning: cannot apply tags to missing message: notmuch-sha1-0079d8e0c3340e6f88c66f4c49fca758ea71d06d
Warning: cannot apply tags to missing message: notmuch-sha1-0194baa4cfb6d39bc9e4d8c049adaccaa777467d
Warning: cannot apply tags to missing message: notmuch-sha1-02aede494fc3f9e9f060cfd7c044d6d724ad287c
Warning: cannot apply tags to missing message: notmuch-sha1-06606c625d3b3445420e737afd9a245ae66e5562
Warning: cannot apply tags to missing message: notmuch-sha1-0747b020f7551415b9bf5059c58e0a637ba53b13
[...]

As detailed in the crash report, all of those were actually innocuous and could be ignored.

Also note that we completely trash the notmuch database because it's actually faster to reindex from scratch than let notmuch slowly figure out that all mails are new and all the old mails are gone. The fresh indexing took:

nov 19 15:08:54 angela notmuch[2521117]: Processed 384679 total files in 23m 41s (270 files/sec.).
nov 19 15:08:54 angela notmuch[2521117]: Added 372610 new messages to the database.

While a reindexing on top of an existing database was going twice as slow, at about 120 files/sec.

Current config file

Putting it all together, I ended up with the following configuration file:

SyncState *
Sync All

# IMAP side, AKA "Far"
IMAPAccount anarcat-imap
Host imap.anarc.at
User anarcat
PassCmd "pass imap.anarc.at"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPAccount anarcat-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C anarcat@imap.anarc.at /usr/lib/dovecot/imap"

IMAPStore anarcat-remote
Account anarcat-tunnel

# Maildir side, AKA "Near"
MaildirStore anarcat-local
# Maildir/top/sub/sub
#SubFolders Verbatim
# Maildir/.top.sub.sub
SubFolders Maildir++
# Maildir/top/.sub/.sub
# SubFolders legacy
# The trailing "/" is important
#Path ~/Maildir-mbsync/
Inbox ~/Maildir/

# what binds Maildir and IMAP
Channel anarcat
Far :anarcat-remote:
Near :anarcat-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
#Patterns *
Patterns * !register  !.register
# Automatically create missing mailboxes, both locally and on the server
Create Both
#Create Near
# Sync the movement of messages between folders and deletions, add after making sure the sync works
Expunge Both
# Propagate mailbox deletion
Remove both

IMAPAccount anarcat-register-imap
Host imap.anarc.at
User register
PassCmd "pass imap.anarc.at-register"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPAccount anarcat-register-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C register@imap.anarc.at /usr/lib/dovecot/imap"

IMAPStore anarcat-register-remote
Account anarcat-register-tunnel

MaildirStore anarcat-register-local
SubFolders Maildir++
Inbox ~/Maildir/.register/

Channel anarcat-register
Far :anarcat-register-remote:
Near :anarcat-register-local:
Create Both
Expunge Both
Remove both

Note that it may be out of sync with my live (and private) configuration file, as I do not publish my "dotfiles" repository publicly for security reasons.

OfflineIMAP

I've used OfflineIMAP for a long time before switching to SMD. I don't exactly remember why or when I started using it, but I do remember it became painfully slow as I started using notmuch, and would sometimes crash mysteriously. It's been a while, so my memory is hazy on that.

It also kind of died in a fire when Python 2 stop being maintained. The main author moved on to a different project, imapfw which could serve as a framework to build IMAP clients, but never seemed to implement all of the OfflineIMAP features and certainly not configuration file compatibility. Thankfully, a new team of volunteers ported OfflineIMAP to Python 3 and we can now test that new version to see if it is an improvement over mbsync.

Crash on full sync

The first thing that happened on a full sync is this crash:

Copy message from RemoteAnarcat:junk:
 ERROR: Copying message 30624 [acc: Anarcat]
  decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Thread 'Copy message from RemoteAnarcat:junk' terminated with exception:
Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/imaputil.py", line 406, in utf7m_decode
    for c in binary.decode():
AttributeError: 'memoryview' object has no attribute 'decode'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 802, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 342, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 908, in _fetch_from_imap
    ndata1 = self.parser['8bit-RFC'].parsebytes(data[0][1])
  File "/usr/lib/python3.9/email/parser.py", line 123, in parsebytes
    return self.parser.parsestr(text, headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 298, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 256, in _parsegen
    if self._cur.get_content_type() == 'message/delivery-status':
  File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 445, in parse
    kwds['parse_tree'] = parse_tree = cls.value_parser(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2675, in parse_content_type_header
    ctype.append(parse_mime_parameters(value[1:]))
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2569, in parse_mime_parameters
    token, value = get_parameter(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2492, in get_parameter
    token, value = get_value(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2403, in get_value
    token, value = get_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1294, in get_quoted_string
    token, value = get_bare_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1223, in get_bare_quoted_string
    token, value = get_encoded_word(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1064, in get_encoded_word
    text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
  File "/usr/lib/python3.9/email/_encoded_words.py", line 181, in decode
    string = bstring.decode(charset)
AttributeError: decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')


Last 1 debug messages logged for Copy message from RemoteAnarcat:junk prior to exception:
thread: Register new thread 'Copy message from RemoteAnarcat:junk' (account 'Anarcat')
ERROR: Exceptions occurred during the run!
ERROR: Copying message 30624 [acc: Anarcat]
  decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 802, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 342, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 908, in _fetch_from_imap
    ndata1 = self.parser['8bit-RFC'].parsebytes(data[0][1])
  File "/usr/lib/python3.9/email/parser.py", line 123, in parsebytes
    return self.parser.parsestr(text, headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 298, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 256, in _parsegen
    if self._cur.get_content_type() == 'message/delivery-status':
  File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 445, in parse
    kwds['parse_tree'] = parse_tree = cls.value_parser(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2675, in parse_content_type_header
    ctype.append(parse_mime_parameters(value[1:]))
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2569, in parse_mime_parameters
    token, value = get_parameter(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2492, in get_parameter
    token, value = get_value(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2403, in get_value
    token, value = get_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1294, in get_quoted_string
    token, value = get_bare_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1223, in get_bare_quoted_string
    token, value = get_encoded_word(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1064, in get_encoded_word
    text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
  File "/usr/lib/python3.9/email/_encoded_words.py", line 181, in decode
    string = bstring.decode(charset)

Folder junk [acc: Anarcat]:
 Copy message UID 30626 (29008/49310) RemoteAnarcat:junk -> LocalAnarcat:junk
Command exited with non-zero status 100
5252.91user 535.86system 3:21:00elapsed 47%CPU (0avgtext+0avgdata 846304maxresident)k
96344inputs+26563792outputs (1189major+2155815minor)pagefaults 0swaps

That only transferred about 8GB of mail, which gives us a transfer rate of 5.3Mbit/s, more than 5 times slower than mbsync. This bug is possibly limited to the bullseye version of offlineimap3 (the lovely 0.0~git20210225.1e7ef9e+dfsg-4), while the current sid version (the equally gorgeous 0.0~git20211018.e64c254+dfsg-1) seems unaffected.

Tolerable performance

The new release still crashes, except it does so at the very end, which is an improvement, since the mails do get transferred:

 *** Finished account 'Anarcat' in 511:12
ERROR: Exceptions occurred during the run!
ERROR: Exception parsing message with ID (<20190619152034.BFB8810E07A@marcos.anarc.at>) from imaplib (response type: bytes).
 AttributeError: decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 910, in _fetch_from_imap
    raise OfflineImapError(

ERROR: Exception parsing message with ID (<40A270DB.9090609@alternatives.ca>) from imaplib (response type: bytes).
 AttributeError: decoding with 'x-mac-roman' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 910, in _fetch_from_imap
    raise OfflineImapError(

ERROR: IMAP server 'RemoteAnarcat' does not have a message with UID '32686'

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 889, in _fetch_from_imap
    raise OfflineImapError(reason, severity)

Command exited with non-zero status 1
8273.52user 983.80system 8:31:12elapsed 30%CPU (0avgtext+0avgdata 841936maxresident)k
56376inputs+43247608outputs (811major+4972914minor)pagefaults 0swaps
"offlineimap  -o " took 8 hours 31 mins 15 secs

This is 8h31m for transferring 12G, which is around 3.1Mbit/s. That is nine times slower than mbsync, almost an order of magnitude!

Now that we have a full sync, we can test incremental synchronization. That is also much slower:

===> multitime results
1: sh -c "offlineimap -o || true"
            Mean        Std.Dev.    Min         Median      Max
real        24.639      0.513       23.946      24.526      25.708      
user        23.912      0.473       23.404      23.795      24.947      
sys         1.743       0.105       1.607       1.729       2.002

That is also an order of magnitude slower than mbsync, and significantly slower than what you'd expect from a sync process. ~30 seconds is long enough to make me impatient and distracted; 3 seconds, less so: I can wait and see the results almost immediately.

Integrity check

That said: this is still on a gigabit link. It's technically possible that OfflineIMAP performs better than mbsync over a slow link, but I Haven't tested that theory.

The OfflineIMAP mail spool is missing quite a few messages as well:

anarcat@angela:~(main)$ find Maildir-offlineimap -type f -type f -a \! -name '.*' | wc -l 
381463
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' | wc -l 
385247

... although that's probably all either new messages or the register folder, so OfflineIMAP might actually be in a better position there. But digging in more, it seems like the actual per-folder diff is fairly similar to mbsync: a few messages missing here and there. Considering OfflineIMAP's instability and poor performance, I have not looked any deeper in those discrepancies.

Other projects to evaluate

Those are all the options I have considered, in alphabetical order

  • doveadm-sync: requires dovecot on both ends, can tunnel over SSH, may have performance issues in incremental sync, written in C
  • fdm: fetchmail replacement, IMAP/POP3/stdin/Maildir/mbox,NNTP support, SOCKS support (for Tor), complex rules for delivering to specific mailboxes, adding headers, piping to commands, etc. discarded because no (real) support for keeping mail on the server, and written in C
  • getmail: fetchmail replacement, IMAP/POP3 support, supports incremental runs, classification rules, Python
  • interimap: syncs two IMAP servers, apparently faster than doveadm and offlineimap, but requires running an IMAP server locally, Perl
  • isync/mbsync: TLS client certs and SSH tunnels, fast, incremental, IMAP/POP/Maildir support, multiple mailbox, trash and recursion support, and generally has good words from multiple Debian and notmuch people (Arch tutorial), written in C, review above
  • mail-sync: notify support, happens over any piped transport (e.g. ssh), diff/patch system, requires binary on both ends, mentions UUCP in the manpage, mentions rsmtp which is a nice name for rsendmail. not evaluated because it seems awfully complex to setup, Haskell
  • nncp: treat the local spool as another mail server, not really compatible with my "multiple clients" setup, Golang
  • offlineimap3: requires IMAP, used the py2 version in the past, might just still work, first sync painful (IIRC), ways to tunnel over SSH, review above, Python

Most projects were not evaluated due to lack of time.

Conclusion

I'm now using mbsync to sync my mail. I'm a little disappointed by the synchronisation times over the slow link, but I guess that's on par for the course if we use IMAP. We are bound by the network speed much more than with custom protocols. I'm also worried about the C implementation and the crashes I have witnessed, but I am encouraged by the fast upstream response.

Time will tell if I will stick with that setup. I'm certainly curious about the promises of interimap and mail-sync, but I have ran out of time on this project.

...
@blog November 17, 2021 - 00:00 • 22 days ago
Tor and the humans who use it

Today, the Tor Project is holding our second annual State of the Onion, an event where Tor teams and the larger Tor community come together to share what we've accomplished this year, and what we're looking forward to in 2022.

This is an opportunity to talk about the many technical advancements our teams have made this year --- and it's also an opportunity to stop and look at the larger picture. With every release, every UX change, and every new translation, we aim to make it easier for people around the world to access the free and uncensored internet.

When we started our year-end fundraising campaign --- the theme of which is Privacy is a Human Right --- I stopped to think about what privacy means to me. Why does this mission resonate with me?

It goes way back. I grew up in the southern U.S., in the Bible Belt, which is a conservative and strictly religious place. As a queer kid I just did not fit in with this conservative culture.

The internet was a place where I could find out about a world outside of this narrow life experience. I was able to talk to people who were more like me, and it gave me hope and strength to know that a different life was possible.

But I couldn’t have done this without privacy. Had these conversations been public with my friends or my family or my peer group, I could have faced some very real negative consequences. The opportunity to explore and research and speak in private is probably what kept me going. In the grand scheme of things, my experience isn’t as high-stakes as many other folks who need privacy, but it’s why this idea --- that everyone on earth has the human right to privacy --- means so much to me.

And that’s what we’re doing at Tor. We're making privacy possible.

The best way to demonstrate Tor's impact (and what you're making possible with your donations) is to share the stories of people who use our tools and how Tor makes it possible for these people to exercise their human right to privacy. I went through the many excellent anonymous submissions to our Tor Stories survey (link) and picked out some stories that demonstrate clearly what Tor makes possible for millions of every day.

What do you make possible with your donation to Tor?

Privacy.

“I use Tor as [my] everyday browser. Especially when I research doctors and other very personal stuff, it feels better, 'cause hopefully there won't be data for sale, telling the world about my assumed medical condition.”

Censorship circumvention.

“I'm Chinese. In China, Google and Wikipedia are blocked. I can't stand [it]. Sometimes I use Tor to [get] across the GFW... Tor has provided me with a lot of help.”

Safety.

“I'm from the United States... I am a domestic violence victim...

My husband track[s] my online activity in real time, selectively block[s] my website access, interfere[s] with my online experience, attempt[s] to steal my account credentials, delete[s] data from my devices, and in a multitude of ways violate[s] my right to privacy.

I started using Tor to protect my privacy... If not for Tor, I would not have another option for online privacy.”

Dissent.

“Tor Browser / Orbot / Tails helped me so much during the 2019-2020 Hong Kong protests as a pro-democracy activist in Hong Kong. It allowed me to read / write / organize freely during the protests. [Tor] has become especially important after CCP imposed its National Security Law on Hong Kong, criminalizing political dissent. Tor Browser / Orbot / Tails allows me to continue my political activism and read / write without self-censorship on Twitter using my pseudonymous account.”

Freedom.

“I live in [Central Europe]...

Tor helped me very much. I educated myself without state propaganda. It got me out of the social bubble and I was able to get insights, ideas, and views from many different social groups.

I would never be free without Tor. And I am not talking just about digital freedom. I am talking about physical freedom as well."

And these are just a few examples. Privacy is a human right. Stand up for that right with a donation to Tor. Right now is an excellent time to make a gift, as all donations are currently matched, 1:1, by Friends of Tor.

...
@blog November 17, 2021 - 00:00 • 22 days ago
Help Censored Users, Run a Tor Bridge

Run a Tor Bridge campaign

Bridges are private Tor relays that serve as stepping stones into the network. When the Tor network is blocked, users can get a bridge to circumvent censorship. Thanks to our community of bridge operators, users in China, Belarus, Iran, and Kazakhstan can connect to the Tor network and access the free and open Internet.

We currently have approximately 1,200 bridges, 900 of which support the obfs4 obfuscation protocol. Unfortunately, these numbers have been decreasing since the beginning of this year. It's not enough to have many bridges: eventually, all of them could find themselves in block lists. We therefore need a constant trickle of new bridges that aren't blocked anywhere yet. This is where we need your help.

The Tor network size 2020 - 2021

The goal of this campaign is to bring more than 200 obfs4 bridges online by the end of this year. We will wrap up the campaign on January 7, 2022. To show our appreciation for your volunteer work, we're offering unique and exclusive Tor reward kits. For example, if you run 10 obfs4 bridges for one year, you can get the Golden Gate bridge kit, including 1 Tor hoodie, 2 Tor T-shirts, and a sticker pack. Some of these kits are limited.

1. Golden Gate bridge (limited to 10 kits)

  • Run 10 obfs4 bridges for 1 year.
  • Reward kit: 1 Tor hoodie + 2 Tor T-shirt + stickers pack.

2. Helix bridge (limited to 20 kits)

  • Run 5 obfs4 bridges for 1 year.
  • Reward kit: 1 Tor T-shirt + stickers pack.

3. University bridge kit (limited to 10 kits)

  • Run 2 obfs4 bridges for 1 year in your university.
  • Reward kit: 1 Tor T-shirt + stickers pack.

4. Rialto bridge (randomly select 10 new bridge operator)

  • Run 1 obfs4 bridge for 1 year and you will be part of the 'reward lottery'.
  • We will randomly select 10 new bridge operators to receive a metallic roots Tor t-shirt as a token of our gratitude for your help defending the open internet.

Bridge campaign rewards

Setting Up A Bridge

To set up an obfs4 bridge, check out our newly revised installation instructions. We have guides for several Linux distributions, FreeBSD, OpenBSD, and docker. Note that an obfs4 bridge needs both an open OR and an open obfs4 port. If you run into any trouble while setting up your bridge, check out our help page and our new Forum.

Once you have set up your bridge, find your bridge line (our post-install instructions explain how) and send an email to frontdesk@torproject.org. If you are running more than one bridge, it's preferable to send all of them once.

Technical requirements

To join the bridges campaign, you must follow these requirements:

  • Static IPv4 address. Although Tor bridges can operate behind dynamic IP addresses, this scenario is not that optimal while thinking about others who need to regularly configure the new IP addresses manually. IPv6 is definitely a plus, but it's not required.

  • Obfs4 pluggable transport configured. As this being the pluggable transport with higher probabilities of passing through global censorship, we opted to choose this one.

  • Uptime 24/7. Serving to the networking 24/7 is vital for bridges, as those who really need to workaround censorship depend on Tor being always available;

  • Only 2 bridges per IPv4 address.

  • Operators running more than 2 bridges should avoid sequential IP addresses. Sequential IP addresses make it easier for a group of bridges being blocked by entities that want filters against the whole network CIDR i.e. /24 for IPv4 or /64 for IPv6. If someone wants to block one bridge, and the whole netmask is used to block this particular bridge, and others are on the same address space, those other bridges will be blocked too, as a side effect.

  • Avoid running bridges on the same IP address of your (public) relay.

  • Relay operators running a large chunk of Tor exit nodes are discouraged to run bridges.

Other campaign rules

  • Participants should claim their reward by commenting on the Bridge's topic on the Tor Forum and sending an email with their full bridge line to frontdesk@torproject.org.

  • Bridges will be tested and validated by the Tor Project staff.

  • Rewards for the Golden Gate kit will follow this timeline:

    • 1 month - Tor Stickers
    • 3 months - Tor T-shirt
    • 6 months - 2nd Tor T-shirt
    • 9 months - Hoodie
  • Rewards for Helix kit will follow this timeline:

    • 1 month - Tor Stickers
    • 5 months - Tor T-shirt
  • Bridge operators must follow the Tor relay good practices.

  • Due to our limited staff capacity at the end of year, Golden Gate and Helix operators should expect to receive their first reward in January 2022.

Other ways to help

If you're not technical enough to run a bridge, but want to help censored users, there are other ways you can help:

  • Run a Snowflake proxy. You do not need a dedicated server and can run a proxy by simply installing an extension in your browser. The extension is available for Firefox and Chrome. There is no need to worry about which websites people are accessing through your proxy. Their visible browsing IP address will match their Tor exit node, not yours.
  • Make a donation to the Tor Project to support our work developing and sharing tools for privacy and freedom online.
  • Help translate Tor materials and documentation including information on how to set up a bridge.
  • Share your support of running and using Tor bridges on social media.
...
@blog November 15, 2021 - 14:27 • 24 days ago
New Release: Tor Browser 11.0.1
New Release: Tor Browser 11.0.1 sysrqb November 15, 2021

Tor Browser 11.0.1 is now available from the Tor Browser download page and also from our distribution directory.

This version provides important bug fixes on Windows, MacOS, and Linux, and includes blockchain explorer Blockchair as a search option.

Search with Blockchair

Blockchair available as a search option in Tor Browser 11.

Tor Browser users can now explore the data from 17 different blockchains from a single search engine with Blockchair. Type Blockchair and hit tab or select the Search with Blockchair shortcut to search directly from the address bar. Or, if you prefer, you can also set Blockchair as your Default Search Engine from within Tor Browser’s Search settings (about:preferences#search).

Full changelog

The full changelog since Tor Browser 11.0 is:

  • Windows, MacOS & Linux:
    • Tor Launcher 0.2.32
    • Bug 40059: YEC activist sign empty in about:tor on RTL locales
    • Bug 40383: Workaround issue in https-e wasm
    • Bug 40438: Add Blockchair as a search engine
    • Bug 40689: Change Blockchair Search provider’s HTTP method
    • Bug 40690: Browser chrome breaks when private browsing mode is turned off
    • Bug 40700: Switch Firefox recommendations off by default

Known issues

Tor Browser 11.0.1 comes with a number of known issues (please check the following list before submitting a new bug report):

  • Bug 40668 : DocumentFreezer & file scheme
  • Bug 40382 : Fonts don’t render
  • Bug 40679 : Missing features on first-time launch in esr91 on MacOS
  • Bug 40667: AV1 videos shows as corrupt files in Windows 8.1
  • Bug 40666 : Switching svg.disable affects NoScript settings
  • Bug 40693 : Potential Wayland dependency
  • Bug 40692: Picture-in-Picture is enabled on tbb 11.0a10
  • Bug 40705: “visit our website” link on about:tbupdate pointing to different locations
  • Bug 40706: Fix issue in https-e wasm
  • Bug 40698: Addon menus missing content in TB11

Join the discussion

Create an account and join the discussion on the Tor Project's brand new forum:

Tor Browser 11.0.1 discussion thread

...
@blog November 15, 2021 - 00:00 • 24 days ago
New Release: Tor Browser 11.0.1

Tor Browser 11.0.1 is now available from the Tor Browser download page and also from our distribution directory.

This version provides important bug fixes on Windows, MacOS, and Linux, and includes blockchain explorer Blockchair as a search option.

Blockchair available as search option

Blockchair available as a search option in Tor Browser 11.

Tor Browser users can now explore the data from 17 different blockchains from a single search engine with Blockchair. Type Blockchair and hit tab or select the Search with Blockchair shortcut to search directly from the address bar. Or, if you prefer, you can also set Blockchair as your Default Search Engine from within Tor Browser’s Search settings (about:preferences#search).

Full changelog

The full changelog since Tor Browser 11.0 is:

  • Windows, MacOS & Linux:
    • Tor Launcher 0.2.32
    • Bug 40059: YEC activist sign empty in about:tor on RTL locales
    • Bug 40383: Workaround issue in https-e wasm
    • Bug 40438: Add Blockchair as a search engine
    • Bug 40689: Change Blockchair Search provider’s HTTP method
    • Bug 40690: Browser chrome breaks when private browsing mode is turned off
    • Bug 40700: Switch Firefox recommendations off by default

Known issues

Tor Browser 11.0.1 comes with a number of known issues (please check the following list before submitting a new bug report):

  • Bug 40668 : DocumentFreezer & file scheme
  • Bug 40382 : Fonts don’t render
  • Bug 40679 : Missing features on first-time launch in esr91 on MacOS
  • Bug 40667: AV1 videos shows as corrupt files in Windows 8.1
  • Bug 40666 : Switching svg.disable affects NoScript settings
  • Bug 40693 : Potential Wayland dependency
  • Bug 40692: Picture-in-Picture is enabled on tbb 11.0a10
  • Bug 40705: “visit our website” link on about:tbupdate pointing to different locations
  • Bug 40706: Fix issue in https-e wasm

Join the discussion

Create an account and join the discussion on the Tor Project's brand new forum:

Tor Browser 11.0.1 discussion thread

...
@kushal November 14, 2021 - 11:38 • 25 days ago
first few weeks in stockholm

Completed 2 weeks in the rented apartment. The day we got it, the owner called at 10pm to tell us that he is selling the place. He decided to skip this important information till we get in. So that he can earn the rent for whatever time he has before selling. So, now we have to find a new place and every other plan is out of order. Anwesha even unpacked most of the bags on that evening, now we are slowly packing again. The worst thing is that Py will have to go to another school (most probably based on where we can find a place).

Very slowly finding out all the things at work, as the brain is being used exclusively to find a new place to live. But, excited to see so much of Python usage within Sunet.

...
@ooni November 11, 2021 - 00:00 • 28 days ago
A multi-perspective view of Internet censorship in Myanmar
In the wake of a military coup in February 2021, Myanmar experienced unprecedented levels of internet censorship. In response, we collaborated with CAIDA’s Internet Outage Detection and Analysis (IODA) team and Myanmar ICT for Development Organization (MIDO) on publishing a research report which documents a series of nightly internet outages and the blocking of social media, Wikipedia, and circumvention tool sites in Myanmar following the military coup. In the months that followed, we continued to examine internet censorship in Myanmar quite closely. ...
@blog November 9, 2021 - 22:29 • 29 days ago
You’re Invited: State of the Onion 2021
You’re Invited: State of the Onion 2021 isabela November 09, 2021

Last year, we held our first 100% virtual State of the Onion, a compilation of updates from the Tor Project's different teams discussing highlights of their work during the year and what we are excited about in the upcoming year. Our 2020 State of the Onion was our first time doing livestream iteration, and it was not only a great success because it allowed us to reach thousands of people all around the world, but also because it allowed for more projects from our community to participate, giving an opportunity for them to also share their updates.

We are happy to announce that we will be hosting our 2021 State of the Onion livestream on November 17 from 17:00 - 19:00 UTC.

This year we have another exciting program for you. Following last year's program structure, we will start with updates from the Tor Project's teams followed by updates from people in our community. Isabela Bagueros, our executive director, will be our host for the event.

Program

Note: More details about each session will be posted on the Tor Forum discussion post related to the 2021 State of the Onion.

Opening — Isabela Bagueros, Executive Director

Part I — The Tor Project

  1. Censorship Circumvention, Cecylia
  2. Applications & Tor Browser, Richard
  3. UX Team, Duncan
  4. Applications & VPN, Matt Finkel
  5. Community Team, Gus
  6. Network Health, GeKo
  7. Metrics, Hiro
  8. Shadow, Jim Newsome
  9. Tor network, little "t" tor, Mike/ahf
  10. Arti, Nick Mathewson
  11. TPA, Anarcat
  12. Year End Fundraising, Al

Intermission — Roger Dingledine, Co-Founder

Part 2 — Tor Community

We have also invited other projects who are part of the Tor community to present and share their updates.

  1. SecureDrop, Conor Schaefer - Chief Technology Officer at Freedom of the Press Foundation
  2. Library Freedom Project, Alison Macrina - Library Freedom Project
  3. Ricochet Refresh, Richard
  4. Open Observatory of Network Interference, Arturo Filastò - Project Lead
  5. The Guardian Project, Nathan Freitas - Founder of the Guardian Project
  6. Tails, Sajolida
  7. Calyx Institute, Nick Merrill - Founder, Executive Director

The State of the Onion will be streamed over our YouTube, Facebook, and Twitter accounts.

You can join the conversation on Twitter using the hashtag: #StateOfTheOnion2021, or post your questions and comments in the Youtube chat.

...
@blog November 9, 2021 - 00:00 • 1 months ago
You’re Invited: State of the Onion 2021

Last year, we held our first 100% virtual State of the Onion, a compilation of updates from the Tor Project's different teams discussing highlights of their work during the year and what we are excited about in the upcoming year. Our 2020 State of the Onion was our first time doing livestream iteration, and it was not only a great success because it allowed us to reach thousands of people all around the world, but also because it allowed for more projects from our community to participate, giving an opportunity for them to also share their updates.

We are happy to announce that we will be hosting our 2021 State of the Onion livestream on November 17 from 17:00 - 19:00 UTC.

This year we have another exciting program for you. Following last year's program structure, we will start with updates from the Tor Project's teams followed by updates from people in our community. Isabela Bagueros, our executive director, will be our host for the event.

Program

Note: More details about each session will be posted on the Tor Forum discussion post related to the 2021 State of the Onion.

Opening — Isabela Bagueros, Executive Director

Part I — The Tor Project

  1. Censorship Circumvention, Cecylia
  2. Applications & Tor Browser, Richard
  3. UX Team, Duncan
  4. Applications & VPN, Matt Finkel
  5. Community Team, Gus
  6. Network Health, GeKo
  7. Metrics, Hiro
  8. Shadow, Jim Newsome
  9. Tor network, little "t" tor, Mike/ahf
  10. Arti, Nick Mathewson
  11. TPA, Anarcat
  12. Year End Fundraising, Al

Intermission — Roger Dingledine, Co-Founder

Part 2 — Tor Community

We have also invited other projects who are part of the Tor community to present and share their updates.

  1. SecureDrop, Conor Schaefer - Chief Technology Officer at Freedom of the Press Foundation
  2. Library Freedom Project, Alison Macrina - Library Freedom Project
  3. Ricochet Refresh, Richard
  4. Open Observatory of Network Interference, Arturo Filastò - Project Lead
  5. The Guardian Project, Nathan Freitas - Founder of the Guardian Project
  6. Tails, Sajolida
  7. Calyx Institute, Nick Merrill - Founder, Executive Director

The State of the Onion will be streamed over our YouTube, Facebook, and Twitter accounts.

You can join the conversation on Twitter using the hashtag: #StateOfTheOnion2021, or post your questions and comments in the Youtube chat.

...
@blog November 8, 2021 - 05:51 • 1 months ago
New Release: Tor Browser 11.0
New Release: Tor Browser 11.0 duncan November 07, 2021

Tor Browser 11.0 is now available from the Tor Browser download page and our distribution directory. This is the first stable release based on Firefox ESR 91, and includes an important update to Tor 0.4.6.8.

Update: The version of Tor Browser for Windows available on the Tor Browser download page has been temporarily reverted to 10.5.10 while we investigate an issue with NoScript. Please see Bug 40695 for updates.

What’s new?

Tor Browser 11's connection screen in light and dark themes

Tor Browser gets a new look

Earlier this year, Firefox’s user interface underwent a significant redesign aimed at simplifying the browser chrome, streamlining menus and featuring an all-new tab design. Firefox ESR 91 introduces the new design to Tor Browser for the first time.

Tor Browser 11's redesigned icons

To ensure it lives up to the new experience, each piece of custom UI in Tor Browser has been modernized to match Firefox’s new look and feel. That includes everything from updating the fundamentals like color, typography and buttons to redrawing each of our icons to match the new thinner icon style.

In addition to the browser chrome itself, the connection screen, circuit display, security levels and onion site errors all received a sprucing-up too – featuring some small but welcome quality of life improvements to each.

Invalid Onion Site Address error resulting from v2 deprecation

Final deprecation of v2 onion services

Last year we announced that v2 onion services would be deprecated in late 2021, and since its 10.5 release Tor Browser has been busy warning users who visit v2 onion sites of their upcoming retirement. At long last, that day has finally come. Since updating to Tor 0.4.6.8 v2 onion services are no longer reachable in Tor Browser, and users will receive an “Invalid Onion Site Address” error instead.

Should you receive this error when attempting to visit a previously working v2 address, there is nothing wrong with your browser – instead, the issue lies with the site itself. If you wish, you can notify the onion site’s administrator about the problem and encourage them to upgrade to a v3 onion service as soon as possible.

It’s easy to tell if you still have any old v2 addresses saved in your bookmarks that are in need of removal or updating too: although both end in .onion, the more secure v3 addresses are 56 characters long compared to v2’s modest 16 character length.

Known issues

Tor Browser 11.0 comes with a number of known issues:

  • Bug 40668: DocumentFreezer & file scheme
  • Bug 40671: Fonts don't render
  • Bug 40679: Missing features on first-time launch in esr91 on MacOS
  • Bug 40689: Change Blockchair Search provider's HTTP method
  • Bug 40667: AV1 videos shows as corrupt files in Windows 8.1
  • Bug 40677: Since the update to 11.0a9 some addons are inactive and need disabling-reenabling on each start
  • Bug 40666: Switching svg.disable affects NoScript settings
  • Bug 40690: Browser chrome breaks when private browsing mode is turned off
  • Bug 40693: Potential Wayland dependency (new)
  • Bug 40692: Picture-in-Picture is enabled on tbb 11.0a10 (new)
  • Bug 40700: Switch Firefox recommendations off by default (new)
  • Bug 40705: "visit our website" link on about:tbupdate pointing to different locations (new)
  • Bug 40706: Fix issue in https-e wasm (new)

Join the discussion

Last week we announced a new discussion and user support platform: the Tor Forum. Starting today, we’re going to be doing things a little bit differently – rather than commenting on the blog posts themselves, we’d like to invite you to create an account and join in the discussion on the forum instead:

Tor Browser 11.0 discussion thread

Give feedback

If you find a bug or have a suggestion for how we could improve this release, please let us know. Thanks to all of the teams across Tor, and the many volunteers, who contributed to this release.

Full changelog

The full changelog since Tor Browser 10.5.10 is:

  • Windows + OS X + Linux
    • Update Firefox to 91.3.0esr
    • Update Tor to tor-0.4.6.8
    • Bug 32624: localStorage is not shared between tabs
    • Bug 33125: Remove xpinstall.whitelist.add* as they don't do anything anymore
    • Bug 34188: Cleanup extensions.* prefs
    • Bug 40004: Convert tl-protocol to async.
    • Bug 40012: Watch all requested tor events
    • Bug 40027: Make torbutton_send_ctrl_cmd async
    • Bug 40042: Add missing parameter of createTransport
    • Bug 40043: Delete all plugin-related protections
    • Bug 40045: Teach the controller about status_client
    • Bug 40046: Support arbitrary watch events
    • Bug 40047: New string for Security Level panel
    • Bug 40048: Protonify Circuit Display Panel
    • Bug 40053: investigate fingerprinting potential of extended TextMetrics interface
    • Bug 40083: Make sure Region.jsm fetching is disabled
    • Bug 40177: Clean up obsolete preferences in our 000-tor-browser.js
    • Bug 40220: Make sure tracker cookie purging is disabled
    • Bug 40342: Set `gfx.bundled-fonts.activate = 1` to preserve current bundled fonts behaviour
    • Bug 40463: Disable network.http.windows10-sso.enabled in FF 91
    • Bug 40483: Deutsche Welle v2 redirect
    • Bug 40534: Cannot open URLs on command line with Tor Browser 10.5
    • Bug 40547: UX: starting in offline mode can result in difficulty to connect later
    • Bug 40548: Set network.proxy.failover_direct to false in FF 91
    • Bug 40561: Refactor about:torconnect implementation
    • Bug 40567: RFPHelper is not init until after about:torconnect bootstraps
    • Bug 40597: Implement TorSettings module
    • Bug 40600: Multiple pages as home page unreliable in 11.0a4
    • Bug 40616: UX: multiple about:torconnect
    • Bug 40624: TorConnect banner always visible in about:preferences#tor even after bootstrap
    • Bug 40626: Update Security Level styling to match Proton UI
    • Bug 40628: Checkbox wrong color in about:torconnect in dark mode theme
    • Bug 40630: Update New Identity and New Circuit icons
    • Bug 40631: site identity icons are not being displayed properly
    • Bug 40632: Proton'ify Circuit Display Panel
    • Bug 40634: Style updates for Onion Error Pages
    • Bug 40636: Fix about:torconnect 'Connect' border radius in about:preferences#tor
    • Bug 40641: Update Security Level selection in about:preferences to match style as tracking protection option bubbles
    • Bug 40648: Replace onion pattern divs/css with tiling SVG
    • Bug 40653: Onion Available text not aligned correctly in toolbar in ESR91
    • Bug 40655: esr91 is suggesting to make Tor Browser the default browse
    • Bug 40657: esr91 is missing "New identity" in hamburger menu
    • Bug 40680: Prepare update to localized assets for YEC
    • Bug 40686: Update Onboarding link for 11.0
  • Build System
    • Windows + OS X + Linux
      • Update Go to 1.16.9
      • Bug 40048: Remove projects/clang-source
      • Bug 40347: Make the list of toolchain updates needed for firefox91
      • Bug 40363: Change bsaes git url
      • Bug 40366: Use bullseye to build https-everywhere
      • Bug 40368: Use system's python3 for https-everywhere
    • Windows + Linux
    • Windows
      • Bug 28240: switch from SJLJ exception handling to Dwarf2 in mingw for win32
      • Bug 40306: Update Windows toolchain to switch to mozilla91
      • Bug 40376: Use python3 for running pe_checksum_fix.py
    • OS X
      • Bug 40307: Update macOS toolchain to switch to mozilla91
    • Linux
      • Bug 40222: Bump GCC to 10.3.0 for Linux
      • Bug 40305: Update Linux toolchain to switch to mozilla91
      • Bug 40353: Temporarily disable rlbox for linux builds
...
@blog November 8, 2021 - 00:00 • 1 months ago
New Release: Tor Browser 11.0

Tor Browser 11.0 is now available from the Tor Browser download page and our distribution directory. This is the first stable release based on Firefox ESR 91, and includes an important update to Tor 0.4.6.8.

Update: The version of Tor Browser for Windows available on the Tor Browser download page has been temporarily reverted to 10.5.10 while we investigate an issue with NoScript. Please see Bug 40695 for updates.

What’s new?

Tor Browser 11's connection screen in light and dark themes

Tor Browser gets a new look

Earlier this year, Firefox’s user interface underwent a significant redesign aimed at simplifying the browser chrome, streamlining menus and featuring an all-new tab design. Firefox ESR 91 introduces the new design to Tor Browser for the first time.

Tor Browser 11's redesigned icons

To ensure it lives up to the new experience, each piece of custom UI in Tor Browser has been modernized to match Firefox’s new look and feel. That includes everything from updating the fundamentals like color, typography and buttons to redrawing each of our icons to match the new thinner icon style.

In addition to the browser chrome itself, the connection screen, circuit display, security levels and onion site errors all received a sprucing-up too – featuring some small but welcome quality of life improvements to each.

Invalid Onion Site Address error resulting from v2 deprecation

Final deprecation of v2 onion services

Last year we announced that v2 onion services would be deprecated in late 2021, and since its 10.5 release Tor Browser has been busy warning users who visit v2 onion sites of their upcoming retirement. At long last, that day has finally come. Since updating to Tor 0.4.6.8 v2 onion services are no longer reachable in Tor Browser, and users will receive an “Invalid Onion Site Address” error instead.

Should you receive this error when attempting to visit a previously working v2 address, there is nothing wrong with your browser – instead, the issue lies with the site itself. If you wish, you can notify the onion site’s administrator about the problem and encourage them to upgrade to a v3 onion service as soon as possible.

It’s easy to tell if you still have any old v2 addresses saved in your bookmarks that are in need of removal or updating too: although both end in .onion, the more secure v3 addresses are 56 characters long compared to v2’s modest 16 character length.

Known issues

Tor Browser 11.0 comes with a number of known issues:

  • Bug 40668: DocumentFreezer & file scheme
  • Bug 40671: Fonts don't render
  • Bug 40679: Missing features on first-time launch in esr91 on MacOS
  • Bug 40689: Change Blockchair Search provider's HTTP method
  • Bug 40667: AV1 videos shows as corrupt files in Windows 8.1
  • Bug 40677: Since the update to 11.0a9 some addons are inactive and need disabling-reenabling on each start
  • Bug 40666: Switching svg.disable affects NoScript settings
  • Bug 40690: Browser chrome breaks when private browsing mode is turned off
  • Bug 40693: Potential Wayland dependency (new)
  • Bug 40692: Picture-in-Picture is enabled on tbb 11.0a10 (new)
  • Bug 40700: Switch Firefox recommendations off by default (new)
  • Bug 40705: "visit our website" link on about:tbupdate pointing to different locations (new)
  • Bug 40706: Fix issue in https-e wasm (new)

Join the discussion

Last week we announced a new discussion and user support platform: the Tor Forum. Starting today, we’re going to be doing things a little bit differently – rather than commenting on the blog posts themselves, we’d like to invite you to create an account and join in the discussion on the forum instead:

Tor Browser 11.0 discussion thread

Give feedback

If you find a bug or have a suggestion for how we could improve this release, please let us know. Thanks to all of the teams across Tor, and the many volunteers, who contributed to this release.

Full changelog

The full changelog since Tor Browser 10.5.10 is:

  • Windows + OS X + Linux
    • Update Firefox to 91.3.0esr
    • Update Tor to tor-0.4.6.8
    • Bug 32624: localStorage is not shared between tabs
    • Bug 33125: Remove xpinstall.whitelist.add* as they don't do anything anymore
    • Bug 34188: Cleanup extensions.* prefs
    • Bug 40004: Convert tl-protocol to async.
    • Bug 40012: Watch all requested tor events
    • Bug 40027: Make torbutton_send_ctrl_cmd async
    • Bug 40042: Add missing parameter of createTransport
    • Bug 40043: Delete all plugin-related protections
    • Bug 40045: Teach the controller about status_client
    • Bug 40046: Support arbitrary watch events
    • Bug 40047: New string for Security Level panel
    • Bug 40048: Protonify Circuit Display Panel
    • Bug 40053: investigate fingerprinting potential of extended TextMetrics interface
    • Bug 40083: Make sure Region.jsm fetching is disabled
    • Bug 40177: Clean up obsolete preferences in our 000-tor-browser.js
    • Bug 40220: Make sure tracker cookie purging is disabled
    • Bug 40342: Set `gfx.bundled-fonts.activate = 1` to preserve current bundled fonts behaviour
    • Bug 40463: Disable network.http.windows10-sso.enabled in FF 91
    • Bug 40483: Deutsche Welle v2 redirect
    • Bug 40534: Cannot open URLs on command line with Tor Browser 10.5
    • Bug 40547: UX: starting in offline mode can result in difficulty to connect later
    • Bug 40548: Set network.proxy.failover_direct to false in FF 91
    • Bug 40561: Refactor about:torconnect implementation
    • Bug 40567: RFPHelper is not init until after about:torconnect bootstraps
    • Bug 40597: Implement TorSettings module
    • Bug 40600: Multiple pages as home page unreliable in 11.0a4
    • Bug 40616: UX: multiple about:torconnect
    • Bug 40624: TorConnect banner always visible in about:preferences#tor even after bootstrap
    • Bug 40626: Update Security Level styling to match Proton UI
    • Bug 40628: Checkbox wrong color in about:torconnect in dark mode theme
    • Bug 40630: Update New Identity and New Circuit icons
    • Bug 40631: site identity icons are not being displayed properly
    • Bug 40632: Proton'ify Circuit Display Panel
    • Bug 40634: Style updates for Onion Error Pages
    • Bug 40636: Fix about:torconnect 'Connect' border radius in about:preferences#tor
    • Bug 40641: Update Security Level selection in about:preferences to match style as tracking protection option bubbles
    • Bug 40648: Replace onion pattern divs/css with tiling SVG
    • Bug 40653: Onion Available text not aligned correctly in toolbar in ESR91
    • Bug 40655: esr91 is suggesting to make Tor Browser the default browse
    • Bug 40657: esr91 is missing "New identity" in hamburger menu
    • Bug 40680: Prepare update to localized assets for YEC
    • Bug 40686: Update Onboarding link for 11.0
  • Build System
    • Windows + OS X + Linux
      • Update Go to 1.16.9
      • Bug 40048: Remove projects/clang-source
      • Bug 40347: Make the list of toolchain updates needed for firefox91
      • Bug 40363: Change bsaes git url
      • Bug 40366: Use bullseye to build https-everywhere
      • Bug 40368: Use system's python3 for https-everywhere
    • Windows + Linux
    • Windows
      • Bug 28240: switch from SJLJ exception handling to Dwarf2 in mingw for win32
      • Bug 40306: Update Windows toolchain to switch to mozilla91
      • Bug 40376: Use python3 for running pe_checksum_fix.py
    • OS X
      • Bug 40307: Update macOS toolchain to switch to mozilla91
    • Linux
      • Bug 40222: Bump GCC to 10.3.0 for Linux
      • Bug 40305: Update Linux toolchain to switch to mozilla91
      • Bug 40353: Temporarily disable rlbox for linux builds
...
@ooni November 8, 2021 - 00:00 • 1 months ago
Investigating Internet shutdowns through Mozilla telemetry
More than 200 million users worldwide use the Firefox web browser (developed by Mozilla) every month. If access to the Internet is shut down in a country, Mozilla should expect to see a dramatic drop in Firefox usage from that country. Given how widespread the use of Firefox is around the world, could Mozilla telemetry be a valuable resource for the Internet freedom community to investigate Internet shutdowns? To explore this question, the Open Observatory of Network Interference (OONI) and Internet Outage Detection and Analysis (IODA) teams joined forces to analyze a dataset of potential outage signals gathered through regular Mozilla telemetry, access to which was provided by Mozilla as part of a relevant research project to validate assumptions about the data. ...
@blog November 5, 2021 - 00:05 • 1 months ago
New Release: Tor Browser 11.0a10 (Windows/macOS/Linux)
New Release: Tor Browser 11.0a10 (Windows/macOS/Linux) sysrqb November 04, 2021

Tor Browser 11.0a10 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable Windows/macOS/Linux release instead.

This version updates Firefox to version 91.3.0esr on Windows, macOS, and Linux. This version includes important security updates to Firefox on Windows, macOS, and Linux.

Please report any issues and regressions you experience since upgrading from an earlier version based on Firefox 78esr.

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services very soon. Please see the deprecation F.A.Q. entry regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 11.0a9:

  • Windows + OS X + Linux
    • Update Firefox to 91.3.0esr
    • Update Tor to 0.4.7.2-alpha
    • Bug 32624: localStorage is not shared between tabs [tor-browser]
    • Bug 33125: Remove xpinstall.whitelist.add* as they don't do anything anymore [tor-browser]
    • Bug 34188: Cleanup extensions.* prefs [tor-browser]
    • Bug 40053: investigate fingerprinting potential of extended TextMetrics interface
    • Bug 40083: Make sure Region.jsm fetching is disabled
    • Bug 40177: Clean up obsolete preferences in our 000-tor-browser.js
    • Bug 40220: Make sure tracker cookie purging is disabled
    • Bug 40342: Set `gfx.bundled-fonts.activate = 1` to preserve current bundled fonts behaviour
    • Bug 40463: Disable network.http.windows10-sso.enabled in FF 91
    • Bug 40483: Deutsche Welle v2 redirect
    • Bug 40548: Set network.proxy.failover_direct to false in FF 91
    • Bug 40630: New Identity and New Circuit icons
    • Bug 40641: Update Security Level selection in about:preferences to match style as tracking protection option bubbles
    • Bug 40648: Replace onion pattern divs/css with tiling SVG
    • Bug 40653: Onion Available text not aligned correctly in toolbar in ESR91
    • Bug 40655: esr91 is suggesting to make Tor Browser the default browse
    • Bug 40657: esr91 is missing "New identity" in hamburger menu
    • Bug 40680: Prepare update to localized assets for YEC
  • Build System
    • Windows + OS X + Linux
      • Bug 40366: Use bullseye to build https-everywhere
      • Bug 40368: Use system's python3 for https-everywhere
...
@blog November 5, 2021 - 00:00 • 1 months ago
New Release: Tor Browser 11.0a10 (Windows/macOS/Linux)

Tor Browser 11.0a10 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable Windows/macOS/Linux release instead.

This version updates Firefox to version 91.3.0esr on Windows, macOS, and Linux. This version includes important security updates to Firefox on Windows, macOS, and Linux.

Please report any issues and regressions you experience since upgrading from an earlier version based on Firefox 78esr.

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services very soon. Please see the deprecation F.A.Q. entry regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 11.0a9:

  • Windows + OS X + Linux
    • Update Firefox to 91.3.0esr
    • Update Tor to 0.4.7.2-alpha
    • Bug 32624: localStorage is not shared between tabs [tor-browser]
    • Bug 33125: Remove xpinstall.whitelist.add* as they don't do anything anymore [tor-browser]
    • Bug 34188: Cleanup extensions.* prefs [tor-browser]
    • Bug 40053: investigate fingerprinting potential of extended TextMetrics interface
    • Bug 40083: Make sure Region.jsm fetching is disabled
    • Bug 40177: Clean up obsolete preferences in our 000-tor-browser.js
    • Bug 40220: Make sure tracker cookie purging is disabled
    • Bug 40342: Set `gfx.bundled-fonts.activate = 1` to preserve current bundled fonts behaviour
    • Bug 40463: Disable network.http.windows10-sso.enabled in FF 91
    • Bug 40483: Deutsche Welle v2 redirect
    • Bug 40548: Set network.proxy.failover_direct to false in FF 91
    • Bug 40630: New Identity and New Circuit icons
    • Bug 40641: Update Security Level selection in about:preferences to match style as tracking protection option bubbles
    • Bug 40648: Replace onion pattern divs/css with tiling SVG
    • Bug 40653: Onion Available text not aligned correctly in toolbar in ESR91
    • Bug 40655: esr91 is suggesting to make Tor Browser the default browse
    • Bug 40657: esr91 is missing "New identity" in hamburger menu
    • Bug 40680: Prepare update to localized assets for YEC
  • Build System
    • Windows + OS X + Linux
      • Bug 40366: Use bullseye to build https-everywhere
      • Bug 40368: Use system's python3 for https-everywhere
...
@anarcat November 4, 2021 - 15:46 • 1 months ago
A Python contextmanager gotcha

Dear lazy web...

I've had this code sitting around as a wtf.py for a while. I've been meaning to understand what's going on and write a blog post about it for a while, but I'm lacking the time. Now that I have a few minutes, I actually sat down to look at it and I think I figured it out:

from contextlib import contextmanager


@contextmanager
def bad():
    print('in the context manager')
    try:
        print("yielding value")
        yield 'value'
    finally:
        return print('cleaning up')


@contextmanager
def good():
    print('in the context manager')
    try:
        print("yielding value")
        yield 'value'
    finally:
        print('cleaning up')


with bad() as v:
    print('got v = %s' % v)
    raise Exception('exception not raised!')  # SILENCED!

print("this code is reached")
with good() as v:
    print('got v = %s' % v)
    raise Exception('expection normally raised')

print("NOT REACHED (expected)")

For those, like me, who need a walkthrough, here's what the above does:

  1. define a bad context manager (the things you use with with statements) with contextlib.contextmanager) which:

    1. prints a debug statement
    2. return a value
    3. then returns and prints a debug statement
  2. define a good context manager in much the same way, except it doesn't return, it just prints statement

  3. use the bad context manager to show how it bypasses an exception

  4. use the good context manager to show how it correctly raises the exception

The output of this code (in Debian 11 bullseye, Python 3.9.2) is:

in the context manager
yielding value
got v = value
cleaning up
this code is reached
in the context manager
yielding value
got v = value
cleaning up
Traceback (most recent call last):
  File "/home/anarcat/wikis/anarc.at/wtf.py", line 31, in <module>
    raise Exception('expection normally raised')
Exception: expection normally raised

What is surprising to me, with this code, is not only does the exception not get raised, but also the return statement doesn't seem to actually execute, or at least not in the parent scope: if it would, this code is reached wouldn't be printed and the rest of the code wouldn't run either.

So what's going on here? Now I know that I should be careful with return in my context manager, but why? And why is it silencing the exception?

The reason why it's being silenced is this little chunk in the with documentation:

If the suite was exited due to an exception, and the return value from the exit() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement.

This feels a little too magic. If you write a context manager with __exit__(), you're kind of forced to lookup again what that API is. But the contextmanager decorator hides that away and it's easy to make that mistake...

Credits to the Python tips book for teaching me about that trick in the first place.

...
@blog November 1, 2021 - 08:26 • 1 months ago
Tor Forum: a new discussion platform for the Tor Community
Tor Forum: a new discussion platform for the Tor Community Gus November 01, 2021

Communicating and finding help online is crucial to building a solid community. After many years of using emails, mailing lists, blog comments, and IRC to help Tor users, we believe that time has come to improve our discussion channels.

Today, we're happy to announce a new discussion and user support platform: the Tor Forum.

The new forum is powered by Discourse: a modern, friendly, and free and open source software. The forum posts are publicly readable, and you don't need to log in to navigate and access the content. It's also possible to install the Discourse App on your mobile device and receive notifications. For users who like the traditional mailing list format, Discourse features email integration. The new forum is compatible and works with Tor Browser (security slider level set 'Safer').

Currently, the Tor Forum is fully hosted by Discourse, and because they do not support onion services yet, it won't have an onion site soon. That is also why the domain is torproject.net, because of our system security policy on using *.torproject.org only for sites we host in our own infrastructure.

In the last few years, the Tor Project has improved internal tools and platforms to make it easy to contribute to Tor software development and to participate in our community. For example, we revamped our websites, moved from Trac to GitLab, connected our chat channels on Matrix, and now we're launching the Tor Forum.

We invite the Tor community to join the Tor Forum and contribute with us!

Moderation policy: Don't be a jerk. Be awesome instead.

All discussions on the Tor Forum will follow the Tor Project's Code of Conduct. Before creating an account, users should read and agree with the moderation policy and our Code of Conduct. Moderators will enforce this policy to keep our community open, welcoming, and inclusive. Keeping a community healthy and welcoming requires an active community and moderation. Users with higher trust levels will be able to flag harmful posts and report to moderators.

The future of mailing lists, blog comments, and Frontdesk

Multiple user support and feedback channels makes it difficult for users to find information and ask questions in a venue where they will receive timely and accurate answers. Plus, maintaining disconnected channels is unsustainable for Tor staff. Both of these challenges can be addressed in the same way: by modernizing and streamlining our support and feedback channels. The Tor Forum will begin to take the place of some of these spaces.

At the beginning of 2022, we will deprecate and archive some of the unused and unmaintained mailing lists and move these discussions to the Tor Forum. The forum has the capacity to receive mails from specific categories and reply through a mail client. Popular and functional mailing lists like tor-dev and tor-project won't be archived and will continue to exist.

Soon, we will migrate the Tor Blog to a static generated website, Lektor. The blog comments will be hosted on Discourse in the new Tor Forum setup. This will make the comment section of each blog post easier to maintain and more open, as moderation will be distributed among trusted Tor Forum users.

Additionally, we will soon begin to inform/point users who come to our frontdesk@torproject.org email service to a public resource. For users living in censored countries where the Tor Project domains are blocked, like China, Iran, Turkey, Egypt, and others, we'll continue providing support by email and chat.

Introducing the Tor Forum categories

We are starting the Tor Forum with a few categories. Depending on the forum usage, we can edit, remove, and create new categories. At the moment, you will find these discussion categories:

1. Feedback

Most people use Tor with Tor Browser, and we would love to know how we can improve the user experience and/or user interface. This section is for feedback about Tor Browser, Tor Browser Alpha, the Tor Project Websites, Localization, and our own Forum. User feedback is limited to products and tools developed by the Tor Project.

2. Support

Whether you're using Tor Browser for Android, setting up a relay, or trying to circumvent censorship, you can find help in this category.

Remember that you should check their support channels for help with Tails, OnionShare, Ricochet Refresh, and other tools. User support is limited to products and tools developed by the Tor Project.

3. In your language

If you want to write about Tor in your language, have nice articles to suggest, or simply want to ask a question, please comment on this category.

4. Events

A list of upcoming Tor events. (This category might get busier again after the pandemic.)

5. Mailing lists

We've mirrored two mailing lists (tor-project and tor-relays), and users can read and receive notifications on Discourse. It's convenient to read the mailing lists without being subscribed to them using the Discourse App. But, on the other hand, it's just a read-only mirror. Users can't reply to mailing lists using Discourse.

6. General Discussion

Anything related to Tor ecosystem which does not fit into the Support or feedback category

How to join the Tor Forum

1. Visit the Tor Forum website: https://forum.torproject.net

2. Click on "Sign up" and register your account. A valid email is required.

3. You will receive an automatic email from torproject1@discoursemail.com with a verification link. Click on this link to finish your registration.

Or you can login using your GitHub or Discord account.

We look forward to seeing you there!

...
@blog November 1, 2021 - 00:00 • 1 months ago
Tor Forum: a new discussion platform for the Tor Community

Communicating and finding help online is crucial to building a solid community. After many years of using emails, mailing lists, blog comments, and IRC to help Tor users, we believe that time has come to improve our discussion channels.

Today, we're happy to announce a new discussion and user support platform: the Tor Forum.

The new forum is powered by Discourse: a modern, friendly, and free and open source software. The forum posts are publicly readable, and you don't need to log in to navigate and access the content. It's also possible to install the Discourse App on your mobile device and receive notifications. For users who like the traditional mailing list format, Discourse features email integration. The new forum is compatible and works with Tor Browser (security slider level set 'Safer').

Currently, the Tor Forum is fully hosted by Discourse, and because they do not support onion services yet, it won't have an onion site soon. That is also why the domain is torproject.net, because of our system security policy on using *.torproject.org only for sites we host in our own infrastructure.

In the last few years, the Tor Project has improved internal tools and platforms to make it easy to contribute to Tor software development and to participate in our community. For example, we revamped our websites, moved from Trac to GitLab, connected our chat channels on Matrix, and now we're launching the Tor Forum.

We invite the Tor community to join the Tor Forum and contribute with us!

Moderation policy: Don't be a jerk. Be awesome instead.

All discussions on the Tor Forum will follow the Tor Project's Code of Conduct. Before creating an account, users should read and agree with the moderation policy and our Code of Conduct. Moderators will enforce this policy to keep our community open, welcoming, and inclusive. Keeping a community healthy and welcoming requires an active community and moderation. Users with higher trust levels will be able to flag harmful posts and report to moderators.

The future of mailing lists, blog comments, and Frontdesk

Multiple user support and feedback channels makes it difficult for users to find information and ask questions in a venue where they will receive timely and accurate answers. Plus, maintaining disconnected channels is unsustainable for Tor staff. Both of these challenges can be addressed in the same way: by modernizing and streamlining our support and feedback channels. The Tor Forum will begin to take the place of some of these spaces.

At the beginning of 2022, we will deprecate and archive some of the unused and unmaintained mailing lists and move these discussions to the Tor Forum. The forum has the capacity to receive mails from specific categories and reply through a mail client. Popular and functional mailing lists like tor-dev and tor-project won't be archived and will continue to exist.

Soon, we will migrate the Tor Blog to a static generated website, Lektor. The blog comments will be hosted on Discourse in the new Tor Forum setup. This will make the comment section of each blog post easier to maintain and more open, as moderation will be distributed among trusted Tor Forum users.

Additionally, we will soon begin to inform/point users who come to our frontdesk@torproject.org email service to a public resource. For users living in censored countries where the Tor Project domains are blocked, like China, Iran, Turkey, Egypt, and others, we'll continue providing support by email and chat.

Introducing the Tor Forum categories

We are starting the Tor Forum with a few categories. Depending on the forum usage, we can edit, remove, and create new categories. At the moment, you will find these discussion categories:

1. Feedback

Most people use Tor with Tor Browser, and we would love to know how we can improve the user experience and/or user interface. This section is for feedback about Tor Browser, Tor Browser Alpha, the Tor Project Websites, Localization, and our own Forum. User feedback is limited to products and tools developed by the Tor Project.

2. Support

Whether you're using Tor Browser for Android, setting up a relay, or trying to circumvent censorship, you can find help in this category.

Remember that you should check their support channels for help with Tails, OnionShare, Ricochet Refresh, and other tools. User support is limited to products and tools developed by the Tor Project.

3. In your language

If you want to write about Tor in your language, have nice articles to suggest, or simply want to ask a question, please comment on this category.

4. Events

A list of upcoming Tor events. (This category might get busier again after the pandemic.)

5. Mailing lists

We've mirrored two mailing lists (tor-project and tor-relays), and users can read and receive notifications on Discourse. It's convenient to read the mailing lists without being subscribed to them using the Discourse App. But, on the other hand, it's just a read-only mirror. Users can't reply to mailing lists using Discourse.

6. General Discussion

Anything related to Tor ecosystem which does not fit into the Support or feedback category

How to join the Tor Forum

1. Visit the Tor Forum website: https://forum.torproject.net

2. Click on "Sign up" and register your account. A valid email is required.

3. You will receive an automatic email from torproject1@discoursemail.com with a verification link. Click on this link to finish your registration.

Or you can login using your GitHub or Discord account.

We look forward to seeing you there!

...
@kushal October 29, 2021 - 08:46 • 1 months ago
Continuing the journey at SUNET

SUNET logo

From this week I started working for SUNET as a public interest technologist. We are under Vetenskapsrådet, this also means from now on I am a central government employee in Sweden.

I will be helping out various in open source projects and services provided SUNET, focusing on privacy and security. I will also continue working on all of the upstream projects I maintain, including SecureDrop.

...