Planet Tor

@blog May 10, 2021 - 14:50 • 2 days ago
New release: Tor 0.4.5.8
New release: Tor 0.4.5.8 nickm May 10, 2021

We have a new stable release today. If you build Tor from source, you can download the source code for Tor 0.4.5.8 on the download page. Packages should be available within the next several weeks, with a new Tor Browser likely next week.

Tor 0.4.5.8 fixes several bugs in earlier version, backporting fixes from the 0.4.6.x series.

Changes in version 0.4.5.8 - 2021-05-10

  • Minor features (compatibility, Linux seccomp sandbox, backport from 0.4.6.3-rc):
    • Add a workaround to enable the Linux sandbox to work correctly with Glibc 2.33. This version of Glibc has started using the fstatat() system call, which previously our sandbox did not allow. Closes ticket 40382; see the ticket for a discussion of trade-offs.
  • Minor features (compilation, backport from 0.4.6.3-rc):
    • Make the autoconf script build correctly with autoconf versions 2.70 and later. Closes part of ticket 40335.

 

  • Minor features (fallback directory list, backport from 0.4.6.2-alpha):
    • Regenerate the list of fallback directories to contain a new set of 200 relays. Closes ticket 40265.
  • Minor features (geoip data):
    • Update the geoip files to match the IPFire Location Database, as retrieved on 2021/05/07.
  • Minor features (onion services):
    • Add warning message when connecting to now deprecated v2 onion services. As announced, Tor 0.4.5.x is the last series that will support v2 onions. Closes ticket 40373.
  • Minor bugfixes (bridge, pluggable transport, backport from 0.4.6.2-alpha):
    • Fix a regression that made it impossible start Tor using a bridge line with a transport name and no fingerprint. Fixes bug 40360; bugfix on 0.4.5.4-rc.
  • Minor bugfixes (build, cross-compilation, backport from 0.4.6.3-rc):
    • Allow a custom "ar" for cross-compilation. Our previous build script had used the $AR environment variable in most places, but it missed one. Fixes bug 40369; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (channel, DoS, backport from 0.4.6.2-alpha):
    • Fix a non-fatal BUG() message due to a too-early free of a string, when listing a client connection from the DoS defenses subsystem. Fixes bug 40345; bugfix on 0.4.3.4-rc.
  • Minor bugfixes (compiler warnings, backport from 0.4.6.3-rc):
    • Fix an indentation problem that led to a warning from GCC 11.1.1. Fixes bug 40380; bugfix on 0.3.0.1-alpha.
  • Minor bugfixes (controller, backport from 0.4.6.1-alpha):
    • Fix a "BUG" warning that would appear when a controller chooses the first hop for a circuit, and that circuit completes. Fixes bug 40285; bugfix on 0.3.2.1-alpha.
  • Minor bugfixes (onion service, client, memory leak, backport from 0.4.6.3-rc):
    • Fix a bug where an expired cached descriptor could get overwritten with a new one without freeing it, leading to a memory leak. Fixes bug 40356; bugfix on 0.3.5.1-alpha.
  • Minor bugfixes (testing, BSD, backport from 0.4.6.2-alpha):
    • Fix pattern-matching errors when patterns expand to invalid paths on BSD systems. Fixes bug 40318; bugfix on 0.4.5.1-alpha. Patch by Daniel Pinto.
...
@blog May 10, 2021 - 14:46 • 2 days ago
New release candidate: Tor 0.4.6.3-rc
New release candidate: Tor 0.4.6.3-rc nickm May 10, 2021

There's a new release candidate available for download. If you build Tor from source, you can download the source code for Tor 0.4.6.3-rc from the download page on the website. Packages should be available over the coming weeks, with a new Tor Browser release likely next week.

Tor 0.4.6.3-rc is the first release candidate in its series. It fixes a few small bugs from previous versions, and adds a better error message when trying to use (no longer supported) v2 onion services.

Though we anticipate that we'll be doing a bit more clean-up between now and the stable release, we expect that our remaining changes will be fairly simple. There will likely be at least one more release candidate before 0.4.6.x is stable.

Changes in version 0.4.6.3-rc - 2021-05-10

  • Major bugfixes (onion service, control port):
    • Make the ADD_ONION command properly configure client authorization. Before this fix, the created onion failed to add the client(s). Fixes bug 40378; bugfix on 0.4.6.1-alpha.
  • Minor features (compatibility, Linux seccomp sandbox):
    • Add a workaround to enable the Linux sandbox to work correctly with Glibc 2.33. This version of Glibc has started using the fstatat() system call, which previously our sandbox did not allow. Closes ticket 40382; see the ticket for a discussion of trade-offs.

 

  • Minor features (compilation):
    • Make the autoconf script build correctly with autoconf versions 2.70 and later. Closes part of ticket 40335.
  • Minor features (geoip data):
    • Update the geoip files to match the IPFire Location Database, as retrieved on 2021/05/07.
  • Minor features (onion services):
    • Add a warning message when trying to connect to (no longer supported) v2 onion services. Closes ticket 40373.
  • Minor bugfixes (build, cross-compilation):
    • Allow a custom "ar" for cross-compilation. Our previous build script had used the $AR environment variable in most places, but it missed one. Fixes bug 40369; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (compiler warnings):
    • Fix an indentation problem that led to a warning from GCC 11.1.1. Fixes bug 40380; bugfix on 0.3.0.1-alpha.
  • Minor bugfixes (logging, relay):
    • Emit a warning if an Address is found to be internal and tor can't use it. Fixes bug 40290; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (onion service, client, memory leak):
    • Fix a bug where an expired cached descriptor could get overwritten with a new one without freeing it, leading to a memory leak. Fixes bug 40356; bugfix on 0.3.5.1-alpha.
...
@blog May 5, 2021 - 14:33 • 7 days ago
Check the status of Tor services with status.torproject.org
Check the status of Tor services with status.torproject.org anarcat May 05, 2021

The Tor Project now has a status page which shows the state of our major services.

You can check status.torproject for news about major outages in Tor services, including v3 and v2 onion services, directory authorities, our website (torproject.org), and the check.torproject.org tool. The status page also displays outages related to Tor internal services, like our GitLab instance.

This post documents why we launched status.torproject.org, how the service was built, and how it works.

Why a status page

The first step in setting up a service page was to realize we needed one in the first place. I surveyed internal users at the end of 2020 to see what could be improved, and one of the suggestions that came up was to "document downtimes of one hour or longer" and generally improve communications around monitoring. The latter is still on the sysadmin roadmap, but a status page seemed like a good solution for the former.

We already have two monitoring tools in the sysadmin team: Icinga (a fork of Nagios) and Prometheus, with Grafana dashboards. But those are hard to understand for users. Worse, they also tend to generate false positives, and don't clearly show users which issues are critical.

In the end, a manually curated dashboard provides important usability benefits over an automated system, and all major organisations have one.

Picking the right tool

It wasn't my first foray in status page design. In another life, I had setup a status page using a tool called Cachet. That was already a great improvement over the previous solutions, which were to use first a wiki and then a blog to post updates. But Cachet is a complex Laravel app, which also requires a web browser to update. It generally requires more maintenance than what we'd like, needing silly things like a SQL database and PHP web server.

So when I found cstate, I was pretty excited. It's basically a theme for the Hugo static site generator, which means that it's a set of HTML, CSS, and a sprinkle of Javascript. And being based on Hugo means that the site is generated from a set of Markdown files and the result is just plain HTML that can be hosted on any web server on the planet.

Deployment

At first, I wanted to deploy the site through GitLab CI, but at that time we didn't have GitLab pages set up. Even though we do have GitLab pages set up now, it's not (yet) integrated with our mirroring infrastructure. So, for now, the source is hosted and built in our legacy git and Jenkins services.

It is nice to have the content hosted in a git repository: sysadmins can just edit Markdown in the git repository and push to deploy changes, no web browser required. And it's trivial to setup a local environment to preview changes:

hugo serve --baseUrl=http://localhost/
firefox https://localhost:1313/

Only the sysadmin team and gitolite administrators have access to the repository, at this stage, but that could be improved if necessary. Merge requests can also be issued on the GitLab repository and then pushed by authorized personnel later on, naturally.

Availability

One of the concerns I have is that the site is hosted inside our normal mirror infrastructure. Naturally, if an outage occurs there, the site goes down. But I figured it's a bridge we'll cross when we get there. Because it's so easy to build the site from scratch, it's actually trivial to host a copy of the site on any GitLab server, thanks to the .gitlab-ci.yml file shipped (but not currently used) in the repository. If push comes to shove, we can just publish the site elsewhere and point DNS there.

And, of course, if DNS fails us, then we're in trouble, but that's the situation anyway: we can always register a new domain name for the status page when we need to. It doesn't seem like a priority at the moment.

Comments and feedback are welcome!

...
@pastly May 3, 2021 - 15:30 • 9 days ago
How I set up my websites with Tor and Nginx

I was recently asked how I setup my websites to:

  1. Redirect HTTP to HTTPS when not accessed via an onion service.
  2. Serve the website over HTTPS when not accessed via an onion service.
  3. Serve the website over HTTP when accessed via an onion service.

I will further explain:

  • How the .onion available button is obtained in my setup.
  • How to add an onion Alt-Svc that works.

I have a very simple setup. I have a tor daemon running on the same machine as nginx. As most of my websites are static, nginx serves their files directly in most cases. There is no software between tor and nginx; if there is for you, that drastically changes things and this post may be of little use to you. If you have extra software "behind" nginx (e.g. a python app generating a dynamic website), most likely this post will still be useful to you. For example, instead of telling nginx this like I do:

location / {
        try_files $uri $uri/ =404;
}

You might be telling nginx this:

location / {
        include proxy_params;
        proxy_pass http://unix:/path/to/some/app.sock;
}

I use Certbot (Let's Encrypt) for my CA, and it automatically generates some of the nginx config you will see below.

All of the nginx config blocks are in one file, /etc/nginx/sites-available/flashflow.pastly.xyz. As is standard with nginx on Debian, there's a symlink to that file in /etc/nginx/sites-enabled/ and /etc/nginx/nginx.conf was already set to load files in /etc/nginx/sites-enabled/.

This post uses flashflow.pastly.xyz and its onion address as an example. Whenever you see flashflow.pastly.xyz or its onion address, mentally replace the domains with your own.

Redirect HTTP to HTTPS when not accessed via an onion service.

This is entirely handled by nginx and uses a server {} block automatically generated by Certbot. It is this:

server {
    if ($host = flashflow.pastly.xyz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
    listen 80;
    listen [::]:80;
    server_name flashflow.pastly.xyz;
    return 404; # managed by Certbot
}

All this block does is redirect to HTTPS. It is used when the user is visiting flashflow.pastly.xyz on port 80, as indicated by the server_name and listen lines.

Serve the website over HTTPS when not accessed via an onion service.

This is entirely handled by nginx. Again as the server_name and listen lines indicate, this block is used when the user is visiting flashflow.pastly.xyz on port 443 (using TLS). This is overwhelmingly automatically generated by Certbot too.

I slightly simplified this block as presented here. We will edit this block later in this post to add Onion-Location and Alt-Svc headers.

server {
    server_name flashflow.pastly.xyz;
    root /var/www/flashflow.pastly.xyz;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/flashflow.pastly.xyz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/flashflow.pastly.xyz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Serve the website over HTTP when accessed via an onion service.

This is the nginx config block. It is a simplified version of the previous one, as it is also actually serving the website, but with plain HTTP and when the user is visiting the onion service, not flashflow.pastly.xyz.

server {
    listen 80;
    listen [::]:80;
    server_name jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion;
    root /var/www/flashflow.pastly.xyz;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

These are the relevant lines from the tor's torrc. We will edit this block later in this post to add Alt-Svc support.

HiddenServiceDir /var/lib/tor/flashflow.pastly.xyz_service
HiddenServicePort 80

In this post I've shared two server {} blocks that tell nginx to listen on port 80. Nginx knows to use this block for onion service connections because the server_name (the hostname that the user's browser is telling nginx it wants to visit) is the onion service. Nginx uses the other server {} block with port 80 when the user's browser tells nginx that it wants to visit flashflow.pastly.xyz.

After adding those lines to the torrc, I reloaded tor (restart not required). Then I could learn what the onion address is:

$ cat /var/lib/tor/flashflow.pastly.xyz_service/hostname 
jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion

And from there knew what to put on the server_name line.

Whenever I edited nginx's config, I reloaded nginx when done (systemctl reload nginx) and verified it didn't say there was an error.

Whenever I edited tor's config, I reloaded tor when done (systemctl reload tor@default) and verified by checking tor's logs that there was no error (journalctl -eu tor@default) and that tor is still running (systemctl status tor@default).

How the .onion available button is obtained in my setup.

Verify that the preceeding steps are working. Verify that:

  1. Visiting http://flashflow.pastly.xyz redirects to https://flashflow.pastly.xyz to serve the website.
  2. Visiting http://jsd33qlp6[...]d.onion serves the website.

This button advertises the fact that the website is also available at an onion service, which improves users' security and may even improve their performance. Further, if they've configured Tor Browser to do so, Tor Browser can automatically redirect to the onion service instead of presenting a button for the user to maybe click.

Find the 2nd server {} block you added, the one that listens on port 443. We are now going to add a single line to it that instructs nginx to add an HTTP header in its responses.

server {
    server_name flashflow.pastly.xyz;
    [... lines omitted ...]
    location / {
        try_files $uri $uri/ =404;
        # ADD THE BELOW LINE
        add_header Onion-Location http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion$request_uri;
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    [... lines omitted ...]
}

Reload nginx and verify it didn't say there was an error.

Visiting https://flashflow.pastly.xyz should now result in a purple .onion available button appearing in the URL bar when the page is done loading. Clicking it will take the user from https://flashflow.pastly.xyz/foo/bar to http://jsd33qlp6[...]d.onion/foo/bar.

How to add an onion Alt-Svc that works.

Verify that the preceeding steps are working. Verify that:

  1. Visiting http://flashflow.pastly.xyz redirects to https://flashflow.pastly.xyz to serve the website.
  2. Visiting http://jsd33qlp6[...]d.onion serves the website.
  3. (Optional) visiting https://flashflow.pastly.xyz results in a purple .onion available button in the URL bar.

This is another HTTP header that tells the browser there is another way to fetch the given resource that it should consider using in the future instead. The Alt-Svc header is used in contexts entirely outside of Tor, but it can also be used to tell Tor Browser to consider secretly fetching content from this host from an onion service in the future.

Common gotcha: The onion service must also support HTTPS. The onion service does not need a TLS certificate that is valid for the onion address: it should just use the same certificate as the regular web service, even though it is invalid for the onion service. The browser verifies that the certificate it gets from jsd33qlp6[...]d.onion is valid for flashflow.pastly.xyz when using the .onion as an Alt-Svc for the .xyz.

Add to the torrc the following line:

HiddenServiceDir /var/lib/tor/flashflow.pastly.xyz_service
HiddenServicePort 80
# ADD THE BELOW LINE
HiddenServicePort 443

Reload tor when done (systemctl reload tor@default) and verify by checking tor's logs that there was no error (journalctl -eu tor@default) and that tor is still running (systemctl status tor@default).

Find the 2nd server {} block you added, the one that listens on port 443. We are now going to add a single line to it that instructs nginx to add an HTTP header in its responses, and edit the server_name line to list the onion service.

server {
    # EDIT THE BELOW LINE
    server_name flashflow.pastly.xyz jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion;
    [... lines omitted ...]
    location / {
        try_files $uri $uri/ =404;
        # ADD THE BELOW LINE
        add_header Alt-Svc 'h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;';
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    [... lines omitted ...]
}

Reload nginx and verify it didn't say there was an error.

You can verify the Alt-Svc header is being sent by, well, inspecting the headers that nginx sends when you request either https://flashflow.pastly.xyz or https://jsd33qlp6[...]d.onion.

$ curl --head https://flashflow.pastly.xyz
HTTP/2 200 
server: nginx/1.14.2
[... lines omitted ...]
onion-location: http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion/
alt-svc: h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;
[... lines omitted ...]


# the --insecure flag tells curl to keep going even though it will see a
# cert that isn't valid for the onion service. This is expected, as
# explained previously.
$ torsocks curl --insecure --head https://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion
HTTP/2 200 
server: nginx/1.14.2
[... lines omitted ...]
onion-location: http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion/
alt-svc: h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;
[... lines omitted ...]

Verifying that Tor Browser actually uses the headers is harder and beyond the scope of this post. The basic idea is to abuse Alt-Svc to serve something different up via the onion service and check that you get the different content after a couple of page refreshes.

...
@anarcat April 28, 2021 - 20:05 • 14 days ago
Building a status page service with Hugo

The Tor Project now has a status page which shows the state of our major services.

You can check status.torprojet.org for news about major outages in Tor services, including v3 and v2 onion services, directory authorities, our website (torproject.org), and the check.torproject.org tool. The status page also displays outages related to Tor internal services, like our GitLab instance.

This post documents why we launched status.torproject.org, how the service was built, and how it works.

Why a status page

The first step in setting up a service page was to realize we needed one in the first place. I surveyed internal users at the end of 2020 to see what could be improved, and one of the suggestions that came up was to "document downtimes of one hour or longer" and generally improve communications around monitoring. The latter is still on the sysadmin roadmap, but a status page seemed like a good solution for the former.

We already have two monitoring tools in the sysadmin team: Icinga (a fork of Nagios) and Prometheus, with Grafana dashboards. But those are hard to understand for users. Worse, they also tend to generate false positives, and don't clearly show users which issues are critical.

In the end, a manually curated dashboard provides important usability benefits over an automated system, and all major organisations have one.

Picking the right tool

It wasn't my first foray in status page design. In another life, I had setup a status page using a tool called Cachet. That was already a great improvement over the previous solutions, which were to use first a wiki and then a blog to post updates. But Cachet is a complex Laravel app, which also requires a web browser to update. It generally requires more maintenance than what we'd like, needing silly things like a SQL database and PHP web server.

So when I found cstate, I was pretty excited. It's basically a theme for the Hugo static site generator, which means that it's a set of HTML, CSS, and a sprinkle of Javascript. And being based on Hugo means that the site is generated from a set of Markdown files and the result is just plain HTML that can be hosted on any web server on the planet.

Deployment

At first, I wanted to deploy the site through GitLab CI, but at that time we didn't have GitLab pages set up. Even though we do have GitLab pages set up now, it's not (yet) integrated with our mirroring infrastructure. So, for now, the source is hosted and built in our legacy git and Jenkins services.

It is nice to have the content hosted in a git repository: sysadmins can just edit Markdown in the git repository and push to deploy changes, no web browser required. And it's trivial to setup a local environment to preview changes:

hugo serve --baseUrl=http://localhost/
firefox https://localhost:1313/

Only the sysadmin team and gitolite administrators have access to the repository, at this stage, but that could be improved if necessary. Merge requests can also be issued on the GitLab repository and then pushed by authorized personnel later on, naturally.

Availability

One of the concerns I have is that the site is hosted inside our normal mirror infrastructure. Naturally, if an outage occurs there, the site goes down. But I figured it's a bridge we'll cross when we get there. Because it's so easy to build the site from scratch, it's actually trivial to host a copy of the site on any GitLab server, thanks to the .gitlab-ci.yml file shipped (but not currently used) in the repository. If push comes to shove, we can just publish the site elsewhere and point DNS there.

And, of course, if DNS fails us, then we're in trouble, but that's the situation anyway: we can always register a new domain name for the status page when we need to. It doesn't seem like a priority at the moment.

Comments and feedback are welcome!


This article was first published on the Tor Project Blog.

...
@blog April 27, 2021 - 08:23 • 15 days ago
Defend Dissent with Tor
Defend Dissent with Tor Gus April 27, 2021

Guest post by Glencora Borradaile

After 4 years of giving digital security trainings to activists and teaching a course called "Communications Security and Social Movements", I've compiled all my materials into an open, digital book - Defend Dissent: Digital Suppression and Cryptographic Defense of Social Movements hosted by Oregon State University where I am an Associate Professor. The book is intended for an introductory, non-major college audience, and I hope it will find use outside the university setting.

Defend Dissent has three parts:

  • Part 1 covers the basics of cryptography: basic encryption, how keys are exchanged, how passwords protect accounts and how encryption can help provide anonymity.  When I give digital security trainings, I don't spend a lot of time here, but I still want people to know (for example) what end-to-end encryption is and why we want it.
  • Part 2 gives brief context for how surveillance is used to suppress social movements, with a US focus.
  • Part 3 contains what you might consider more classic material for digital security training, focusing on the different places your data might be vulnerable and the tactics you can use to defend your data.

Each chapter ends with a story that brings social context to the material in that chapter (even in Part 1!) - from surveillance used against contemporary US protests to the African National Congress' use of partially manual encryption in fighting apartheid in South Africa in the 80s.

It should be no surprise that Tor is a star of Defend Dissent, ending out Parts 1 and 3. The anonymity that the Tor technology enables turns the internet into what it should be: a place to communicate without everyone knowing your business. As a professor, I love teaching Tor. It is a delightful combination of encryption, key exchange, probability and threat modeling.

In Defend Dissent, I aim to make Tor easy to understand, and use a three-step example to explain Tor to audiences who may have never used it before: There are just three steps to understanding how Tor works:

1. Encryption allows you to keep the content of your communications private from anyone who doesn't have the key. But it doesn't protect your identity or an eavesdropper from knowing who you are communicating with and when.

Encryption keeps the content of your communications private

2. Assata can send Bobby an encrypted message even if they haven't met ahead of time to agree on a key for encryption. This concept can be used to allow Assata and Bobby to agree on a single encryption key. (Put an encryption key in the box.)

Exchanging a secure message without sharing a key

3. When Assata accesses Tor, the Tor Browser picks three randomly chosen nodes (her entry, relay and exit nodes) from amongst thousands in the Tor network. Assata's Tor Browser agrees on a key with the entry node, then agrees on a key with the relay node by communicating with the relay node through the entry node, and so on. Assata's Tor Browser encrypts the message with the exit key, then with the relay key and then with the entry key and sends the message along. The entry node removes one layer of encryption and so on. (Like removing the layers of an onion ...) This way, the relay doesn't know who Assata is - just that it is relaying a message through the Tor network.

I'm excited to share this accessible resource and to teach the world more about Tor, encryption, and secure communication. Even if you're a technical expert, Defend Dissent may help you talk to others in your life about how to use Tor and why these kinds of tools are so vital to social movements, change, and dissent.

For more details on how Tor works you can read the four chapters of Defend Dissent that lead to Anonymous Routing: What is Encryption?, Modern Cryptography, Exchanging Keys for Encryption, and Metadata.
Or discover other topics in defending social movements with cryptography.

...
@blog April 27, 2021 - 02:13 • 15 days ago
New Release: Tor Browser 10.5a15
New Release: Tor Browser 10.5a15 sysrqb April 26, 2021

Tor Browser 10.5a15 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This version updates Firefox to 78.10esr and Fenix to 88.1.1. In addition, Tor Browser 10.5a15 updates Tor to 0.4.6.2-alpha. This version includes important security updates to Firefox for Desktop and security updates for Android.

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

Note: This version is not completely reproducible. We are investigating non-determinism in the Android Tor Browser build. Tor Browser for Windows, macOS and Linux are reproducible.

The full changelog since Tor Browser 10.5a14:

  • All Platforms
    • Update Tor to 0.4.6.2-alpha
  • Windows + OS X + Linux
    • Update Firefox to 78.10.0esr
    • Bug 40408: Disallow SVG Context Paint in all web content
  • Android
    • Update Fenix to 88.1.1
    • Bug 40051: Rebase android-components patches for Fenix 88
    • Bug 40158: Rebase Fenix patches to Fenix 88.1.1
    • Bug 40399: Rebase 10.5 patches on 88.0
  • Build System
    • All Platforms
      • Update Go to 1.15.11
    • Android
      • Bug 40259: Update components for mozilla88-based Fenix
...
@nickm April 27, 2021 - 00:00 • 15 days ago
Implementing Rust futures when you only have async functions

Rust doesn't yet support asynchronous functions in traits, but several important async-related traits (like AsyncRead and Stream ) define their interface using functions that return Poll. So, what can you do when you have a function that is async, and you need to use it to implement one of these traits?

(I'll be assuming that you already know a little about pinning, futures, and async programming in Rust. That's not because they're easy “everybody-should-know-it” topics, but because I'm still learning them myself, and I don't understand them well enough be a good teacher. You can probably keep reading if you don't understand them well . )

A little background

Here's a situation I ran into earlier this year. In the end, I only solved it with help from Daniel Franke, so I decided that I should write up the solution here in case it can help somebody else.

I've been working on Arti, an implementation of the Tor protocols in Rust. After a bunch of hacking, I finally got to the point where I had a DataStream type that provides an anonymous connection over the Tor network:

impl DataStream {
    pub async fn read(&mut self, buf: &mut[u8]) -> io::Result<usize>
    { ... }
    pub async fn write(&mut self, buf: &[u8]) -> io::Result<usize>
    { ... }
}

Now, there's a lot of complexity hiding in those ellipses. Tor isn't a simple protocol: when it's trying to read, it may need to wait for data to arrive. It may also need to send messages in response to arriving data. It might need to update internal state, or even tear down an entire Tor circuit because of a protocol error. Async functions made it possible to implement all of this stuff in a more-or-less comprehensible way, so rewriting those functions to explicitly return a typed future was not an option.

But I wanted DataStream to implement AsyncRead and AsyncWrite, so I could use it with other code in the Rust async ecosystem. So let's look at AsyncRead (because it's simpler than AsyncWrite). The only required method in AsyncRead is:

pub fn poll_read(
    self: Pin<&mut Self>,
    cx: &mut Context<'_>,
    buf: &mut [u8]
) -> Poll<io::Result<usize>>

This read() has to check whether there's data can be read into buf immediately, without blocking. If there is, we read the data and return the number of bytes we read. Otherwise, we have to schedule ourself on cx, and return Poll::Pending.1

Moving forward, getting stuck

Compare poll_read to the read function in DataStream. First off, there's a mismatch between how these two functions use their output buffers. Because DataStream::read is async, it returns a future that will hang on to its buffer until the future is finally ready. But poll_read has to return right away, and it can't store a reference to its buffer at all. So I started by defining a wrapper variant of DataStream to implements the behavior that poll_read would need:2

pub struct DataReaderImpl {
    s: DataStream,
    pending: Vec<u8>
    offset: usize,
    len: usize,
}
pub struct DataReaderImpl {
    s: DataStream,
    pending: Vec<u8>
}
impl DataReaderImpl {
    fn new(s: DataStream) -> DataReaderImpl {
        DataReaderImpl {
            s,
            pending: Vec::new(),
        }
    }
    // Load up to 1k into the pending buffer.
    async fn fill_buf(&mut self) -> io::Result<usize> {
        let mut data = vec![0;1024];
        let len = self.s.read(&mut data[..]).await?;
        data.truncate(len);
        self.pending.extend(data);
        Ok(len)
    }
    // pull bytes from the pending buffer into `buf`.
    fn extract_bytes(&mut self, buf: &mut [u8]) -> usize {
        let n = cmp::min(buf.len(), self.pending.len());
        buf[..n].copy_from_slice(&self.pending[..n]);
        self.pending.drain(0..n);
        n
    }
}

Then, I thought, it ought to be easy to write AsyncRead! Here was my first try:

// This won't work...
impl AsyncRead for DataReaderImpl {
    fn poll_read(mut self: Pin<&mut Self>,
                 cx: &mut Context<'_>,
                 buf: &mut [u8]) -> Poll<io::Result<usize>> {
       if self.pending.is_empty() {
            // looks like we need more bytes.
            let fut = self.fill_buf();
            futures::pin_mut!(fut);
            match fut.poll(cx) {
                Poll::Ready(Err(e)) =>
                    return Poll::Ready(Err(e)),
                Poll::Ready(Ok(n)) =>
                    if n == 0 {
                        return Poll::Ready(Ok(0)); // EOF
                    }
                Poll::Pending =>
                    todo!("crud, where do i put the future?"), // XXXX
            }
        }

        // We have some data; move it out to the caller.
        let n = self.extract_bytes(buf);
        Poll::Ready(Ok(n))
    }
}

Almost there! But what do I do if the future says it's pending? I need to store it and poll it again later the next time I call this function. But to do that, I won't be able to pin the future to the stack! I'll have to store it in the structure instead. And since the future comes from an async function, it won't have a type that I can name; I'll have to store it as a Box<dyn Future>.

Oh hang on, it'll need to be pinned. And sometimes there won't be a read in progress, so I won't have a future at all. Maybe I store it in an Option<Pin<Box<dyn Future>>>?

(This is the point where I had to take a break and figure out pin-projection3.)

But after I played around with that for a while, I hit the final snag: ultimately, I was trying to create a self-referential structure4, which you can't do in safe Rust. You see, the future returned by DataReaderImpl::fill_buf needs to hold a reference to the DataReaderImpl, and so the future needs to outlive the DataReaderImpl. That means you can't store it in the DataReaderImpl. You can't even store it and the DataReaderImpl in the same struct: that creates self-reference.

So what could I do? Was I supposed to use unsafe code or some tricky crate to make a self-referential struct anyway? Was my solution fundamentally flawed? Was I even trying to do something possible‽

I asked for help on Twitter. Fortunately, Daniel Franke got back to me, looked at my busted code, and walked me through the actual answer.

Hold the future or the reader: not both!

Here's the trick: We define an enum that holds the DataReaderImpl or the future that its fill_buf function returns, but not both at once. That way, we never have a self-referential structure!

First we had to define a new variation on fill_buf that will take ownership of the reader when it's called, and return ownership once it's done:

impl DataReaderImpl {
    async fn owning_fill_buf(mut self) -> (Self, io::Result<usize>) {
        let r = self.fill_buf().await;
        (self, r)
    }
}

Then we had to define an enum that could hold either the future or the DataReaderImpl object, along with a wrapper struct to hold the enum.

type OwnedResult = (DataReaderImpl, io::Result<usize>);
enum State {
    Closed,
    Ready(DataReaderImpl),
    Reading(Pin<Box<dyn Future<Output=OwnedResult>>>),
}
struct DataReader {
    state: Option<State>
}

Note that the DataReader struct holds an Option<State>—we'll want to modify the state object destructively, so we'll need to take ownership of the state in poll_read and then replace it with something else.5

With this groundwork in place we could finally give an implementation of AsyncRead that works:

impl AsyncRead for DataReader {
    fn poll_read(mut self: Pin<&mut Self>,
                 cx: &mut Context<'_>,
                 buf: &mut [u8]) -> Poll<io::Result<usize>> {
        // We're taking this temporarily. We have to put
        // something back before we return.
        let state = self.state.take().unwrap();

        // We own the state, so we can destructure it.
        let mut future = match state {
            State::Closed => {
                self.state = Some(State::Closed);
                return Poll::Ready(Ok(0));
            }
            State::Ready(mut imp) => {
                let n = imp.extract_bytes(buf);
                if n > 0 {
                    self.state = Some(State::Ready(imp));
                    // We have data, so we can give it back now.
                    return Poll::Ready(Ok(n));
                }
                // Nothing available; launch a read and poll it.
                Box::pin(imp.owning_fill_buf())
            }
            // If we have a future, we need to poll it.
            State::Reading(fut) => fut,
        };

        // Now we have a future for an in-progress read.
        // Can it make any progress?
        match future.as_mut().poll(cx) {
            Poll::Ready((_imp, Err(e))) => { // Error
                self.state = Some(State::Closed);
                Poll::Ready(Err(e))
            }
            Poll::Ready((_imp, Ok(0))) => { // EOF
                self.state = Some(State::Closed);
                Poll::Ready(Ok(0))
            }
            Poll::Ready((mut imp, Ok(_))) => {
                // We have some data!
                let n = imp.extract_bytes(buf);
                self.state = Some(State::Ready(imp));
                debug_assert!(n > 0);
                Poll::Ready(Ok(n))
            }
            Poll::Pending => {
                // We're pending; remember the future
                // and tell the caller.
                self.state = Some(State::Reading(future));
                Poll::Pending
            }
        }
    }
}

Now when poll_read() takes ownership of the previous state, it either owns a DataReaderImpl or a future returned by owning_fill_buf()—but never both at once, so we don't have any self-reference problems. When poll_read() done, it has to put a new valid state back before it returns.

Conclusions

For the current version of all this code, have a look at tor_proto::stream::data in Arti. Note that the code in Arti is more complex than what I have in this post, and some of that complexity is probably unnecessary: I've been learning more about Rust as I go along.

I hope that some day there's an easier way to do all of this (with real asynchronous traits, maybe?) but in the meantime, I hope that this write-up will be useful to somebody else.


1

We might also have to report an EOF as Poll::Ready(Ok(0)), or an error as Poll::Ready(Err(_). But let's keep this simple.

2

At this point I started writing my code really inefficiently, since I was just trying to get it to work. In the interest of clarity, I'll leave it as inefficient code here too.

3

It didn't turn out to be what I needed in the end, but I'm glad I learned about it: it has been the answer for a lot of other problems later on.

4

Self-referential structures in Rust require unsafe code and pinning. I spent a semi-unpleasant hour or two looking through example code here just to see what would be involved, and tried learning the rental crate, in case it would help.

5

We could probably use std::mem::replace for this too, but I don't expect there would be a performance difference.

...
@anarcat April 25, 2021 - 01:02 • 17 days ago
Lost article ideas

I wrote for LWN for about two years. During that time, I wrote (what seems to me an impressive) 34 articles, but I always had a pile of ideas in the back of my mind. Those are ideas, notes, and scribbles lying around. Some were just completely abandoned because they didn't seem a good fit for LWN.

Concretely, I stored those in branches in a git repository, and used the branch name (and, naively, the last commit log) as indicators of the topic.

This was the state of affairs when I left:

remotes/private/attic/novena                    822ca2bb add letter i sent to novena, never published
remotes/private/attic/secureboot                de09d82b quick review, add note and graph
remotes/private/attic/wireguard                 5c5340d1 wireguard review, tutorial and comparison with alternatives
remotes/private/backlog/dat                     914c5edf Merge branch 'master' into backlog/dat
remotes/private/backlog/packet                  9b2c6d1a ham radio packet innovations and primer
remotes/private/backlog/performance-tweaks      dcf02676 config notes for http2
remotes/private/backlog/serverless              9fce6484 postponed until kubecon europe
remotes/private/fin/cost-of-hosting             00d8e499 cost-of-hosting article online
remotes/private/fin/kubecon                     f4fd7df2 remove published or spun off articles
remotes/private/fin/kubecon-overview            21fae984 publish kubecon overview article
remotes/private/fin/kubecon2018                 1edc5ec8 add series
remotes/private/fin/netconf                     3f4b7ece publish the netconf articles
remotes/private/fin/netdev                      6ee66559 publish articles from netdev 2.2
remotes/private/fin/pgp-offline                 f841deed pgp offline branch ready for publication
remotes/private/fin/primes                      c7e5b912 publish the ROCA paper
remotes/private/fin/runtimes                    4bee1d70 prepare publication of runtimes articles
remotes/private/fin/token-benchmarks            5a363992 regenerate timestamp automatically
remotes/private/ideas/astropy                   95d53152 astropy or python in astronomy
remotes/private/ideas/avaneya                   20a6d149 crowdfunded blade-runner-themed GPLv3 simcity-like simulator
remotes/private/ideas/backups-benchmarks        fe2f1f13 review of backup software through performance and features
remotes/private/ideas/cumin                     7bed3945 review of the cumin automation tool from WM foundation
remotes/private/ideas/future-of-distros         d086ca0d modern packaging problems and complex apps
remotes/private/ideas/on-dying                  a92ad23f another dying thing
remotes/private/ideas/openpgp-discovery         8f2782f0 openpgp discovery mechanisms (WKD, etc), thanks to jonas meurer
remotes/private/ideas/password-bench            451602c0 bruteforce estimates for various password patterns compared with RSA key sizes
remotes/private/ideas/prometheus-openmetrics    2568dbd6 openmetrics standardizing prom metrics enpoints
remotes/private/ideas/telling-time              f3c24a53 another way of telling time
remotes/private/ideas/wallabako                 4f44c5da talk about wallabako, read-it-later + kobo hacking
remotes/private/stalled/bench-bench-bench       8cef0504 benchmarking http benchmarking tools
remotes/private/stalled/debian-survey-democracy 909bdc98 free software surveys and debian democracy, volunteer vs paid work

Wow, what a mess! Let's see if I can make sense of this:

Attic

Those are articles that I thought about, then finally rejected, either because it didn't seem worth it, or my editors rejected it, or I just moved on:

  • novena: the project is ooold now, didn't seem to fit a LWN article. it was basically "how can i build my novena now" and "you guys rock!" it seems like the MNT Reform is the brain child of the Novena now, and I dare say it's even cooler!
  • secureboot: my LWN editors were critical of my approach, and probably rightly so - it's a really complex subject and I was probably out of my depth... it's also out of date now, we did manage secureboot in Debian
  • wireguard: LWN ended up writing extensive coverage, and I was biased against Donenfeld because of conflicts in a previous project

Backlog

Those were articles I was planning to write about next.

  • dat: I already had written Sharing and archiving data sets with Dat, but it seems I had more to say... mostly performance issues, beaker, no streaming, limited adoption... to be investigated, I guess?
  • packet: a primer on data communications over ham radio, and the cool new tech that has emerged in the free software world. those are mainly notes about Pat, Direwolf, APRS and so on... just never got around to making sense of it or really using the tech...
  • performance-tweaks: "optimizing websites at the age of http2", the unwritten story of the optimization of this website with HTTP/2 and friends
  • serverless: god. one of the leftover topics at Kubecon, my notes on this were thin, and the actual subject, possibly even thinner... the only lie worse than the cloud is that there's no server at all! concretely, that's a pile of notes about Kubecon which I wanted to sort through. Probably belongs in the attic now.

Fin

Those are finished articles, they were published on my website and LWN, but the branches were kept because previous drafts had private notes that should not be published.

Ideas

A lot of those branches were actually just an empty commit, with the commitlog being the "pitch", more or less. I'd send that list to my editors, sometimes with a few more links (basically the above), and they would nudge me one way or the other.

Sometimes they would actively discourage me to write about something, and I would do it anyways, send them a draft, and they would patiently make me rewrite it until it was a decent article. This was especially hard with the terminal emulator series, which took forever to write and even got my editors upset when they realized I had never installed Fedora (I ended up installing it, and I was proven wrong!)

Stalled

Oh, and then there's those: those are either "ideas" or "backlog" that got so far behind that I just moved them out of the way because I was tired of seeing them in my list.

  • stalled/bench-bench-bench benchmarking http benchmarking tools, a horrible mess of links, copy-paste from terminals, and ideas about benchmarking... some of this trickled out into this benchmarking guide at Tor, but not much more than the list of tools
  • stalled/debian-survey-democracy: "free software surveys and Debian democracy, volunteer vs paid work"... A long standing concern of mine is that all Debian work is supposed to be volunteer, and paying explicitly for work inside Debian has traditionally been frowned upon, even leading to serious drama and dissent (remember Dunc-Tank)? back when I was writing for LWN, I was also doing paid work for Debian LTS. I also learned that a lot (most?) Debian Developers were actually being paid by their job to work on Debian. So I was confused by this apparent contradiction, especially given how the LTS project has been mostly accepted, while Dunc-Tank was not... See also this talk at Debconf 16. I had hopes that this study would show the "hunch" people have offered (that most DDs are paid to work on Debian) but it seems to show the reverse (only 36% of DDs, and 18% of all respondents paid). So I am still confused and worried about the sustainability of Debian.

What do you think?

So that's all I got. As people might have noticed here, I have much less time to write these days, but if there's any subject in there I should pick, what is the one that you would find most interesting?

Oh! and I should mention that you can write to LWN! If you think people should know more about some Linux thing, you can get paid to write for it! Pitch it to the editors, they won't bite. The worst that can happen is that they say "yes" and there goes two years of your life learning to write. Because no, you don't know how to write, no one does. You need an editor to write.

That's why this article looks like crap and has a smiley. :)

...
@anarcat April 24, 2021 - 17:56 • 18 days ago
A dead game clock

Time flies. Back in 2008, I wrote a game clock. Since then, what was first called "chess clock" was renamed to pychessclock and then Gameclock (2008). It shipped with Debian 6 squeeze (2011), 7 wheezy (4.0, 2013, with a new UI), 8 jessie (5.0, 2015, with a code cleanup, translation, go timers), 9 stretch (2017), and 10 buster (2019), phew! Eight years in Debian over 4 releases, not bad!

But alas, Debian 11 bullseye (2021) won't ship with Gameclock because both Python 2 and GTK 2 were removed from Debian. I lack the time, interest, and energy to port this program. Even if I could find the time, everyone is on their phone nowadays.

So finding the right toolkit would require some serious thinking about how to make a portable program that can run on Linux and Android. I care less about Mac, iOS, and Windows, but, interestingly, it feels it wouldn't be much harder to cover those as well if I hit both Linux and Android (which is already hard enough, paradoxically).

(And before you ask, no, Java is not an option for me thanks. If I switch to anything else than Python, it would be Golang or Rust. And I did look at some toolkit options a few years ago, was excited by none.)

So there you have it: that is how software dies, I guess. Alternatives include:

  • Chessclock - really old Ruby which made Gameclock rename
  • Ghronos - also really old Java app
  • Lichess - has a chess clock built into the app
  • Otter - if you squint a little

PS: Monkeysign also suffered the same fate, for what that's worth. Alternatives include caff, GNOME Keysign, and pius. Note that this does not affect the larger Monkeysphere project, which will ship with Debian bullseye.

...
@blog April 23, 2021 - 01:04 • 19 days ago
Domain Shadowing: Leveraging CDNs for Robust Blocking-Resistant Communications
Domain Shadowing: Leveraging CDNs for Robust Blocking-Resistant Communications Mingkui Wei April 22, 2021

We invited guest blog author, Mingkui Wei, to submit a summary of their research to the blog this week. This blog post is based on the upcoming Usenix Security paper (full version here). Note that the domain shadowing ideas presented herein are intended to be a building block for a future system that doesn't exist for end-users yet. We hope this post will help system designers to think in new ways, and use those ideas to build new censorship circumvention tools.

What is Domain Shadowing?
Domain shadowing is a new censorship circumvention technique that uses Content Distribution Networks (CDNs) as its leverage to achieve its goal, which is similar to domain fronting. However, domain shadowing works completely differently from domain fronting and is stronger in terms of blocking-resistance. Compared to domain fronting, one big difference among many is that the user in domain shadowing is in charge of the whole procedure. In other words, the complete system can be solely configured by the user without necessary assistance from neither the censored website nor an anti-censorship organization.

How Domain Shadowing Works
We start this section by explaining how domain names are resolved and translated by CDN.

CDNs act like a reverse proxy that hides the back-end domain and presents only the front-end domain to the public. CDNs typically take two approaches to accomplish the name translation, as shown in the following two figures. We make the following assumptions to facilitate the illustration: assume the publisher's (i.e. the person who wants to use CDN to distribute the content of their website) origin server is hosted on Amazon Web Service (AWS) and assigned a canonical name abc.aws.com, and the publisher wants to advertise the website using the domain example.com, which is hosted on GoDaddy's name server.

Figure 1 shows the name translation procedure used by most CDNs, and we use Fastly as an example. To use Fastly's service, the publisher will first log into their Fastly account and set example.com as the frontend, and abc.aws.com as backend. Then, the publisher will create a new CNAME record in their GoDaddy's name server, which resolves the domain example.com to a fixed domain global.ssl.fastly.net. The remaining steps in Figure 1 are intuitive.

There are also some CDNs who host their own TLD name server, such as Cloudflare. If this is the case, step 2 and step 3 in Figure 1 can be skipped (as shown in Figure 2).

Note that the last four steps on both figures show the difference of how a name resolution is conducted by CDN. Specifically, a regular DNS server will only respond to a DNS query with the location of the origin server and the document must be fetched by the client itself, while the CDN will actually fetch the web document for the client.

Name resolution by Fastly

Figure 1. Name resolution by Fastly

Name resolution by Cloudflare

Figure 2: Name resolution by Cloudflare

Based on the above introduction, we can now present how domain shadowing works. Domain shadowing takes advantage of the fact that when the domain binding (i.e. the connection between the frontend and the backend domains) is created, the CDN allows arbitrary domains to be set at the backend. As a result, a user can freely bind a frontend domain to any backend domain. To access a blocked domain (e.g. censored.com) within a censored area, a censored user only needs to take the following steps:

  1. The user registers a random domain as the "shadow" domain, for example: shadow.com. We assume the censor won't block this newly registered domain.
  2. The user subscribes to a CDN service that is accessible within the censored area, but the CDN itself is not censored. A practical example would be the CDN deploys all its edge servers outside the censored area.
  3. The user binds the shadow domain to the censored domain in the CDN service by setting the shadow domain as the frontend and the censored domain as the backend.
  4. The user creates a rule in their CDN account to rewrite the Host header of incoming requests from Host:shadow.com to Host:censored.com. This is an essential step since otherwise, the origin server of censored.com will receive an unrecognized Host header and unable to serve the request.
  5. Finally, to access the censored domain, the user sends a request to https://shadow.com within the censored area. The request will be sent to the CDN, which will rewrite the Host header and forwards the request to censored.com. After the response is received from censored.com, the CDN will return the response to the user "in the name" of https://shadow.com.

During this process, the censor will only see the user connect to the CDN using HTTPS and request resources from shadow.com, and thus will not block the traffic.

On a CDN that still supports domain fronting, we can apply domain fronting techniques to make domain shadowing stealthier. To do this, we still set shadow.com as the frontend and censored.com as the backend, but when an HTTPS request is issued from within the censored area, the user will request the front domain front.com and set the Host header to be shadow.com. This way, the censor only sees the user is communicating with the front domain and will not even suspect the user's behavior.

What’s the Benefit of Domain Shadowing?
Compared to its siblings, domain fronting, the obvious benefit of domain shadowing is that it can use any CDN (as long as the CDN supports domain shadowing, and based on our experiments most CDNs do) to access any domain. The censored domain does not need to be on the same CDN, which is a big limitation of domain fronting. Actually, the censored domain does not need to be a domain that uses CDN at all. This is a big leap compared to domain fronting, which can only access the domains on the same CDN as the front domain.

Another shortcoming of domain fronting is that it can be (and is being) painlessly disabled by CDNs by mandating the Host header of an HTTPS request must match the SNI of the TLS handshake. Domain shadowing, on the other hand, is harder to be disabled since allowing a user to configure the backend domains is a legitimate feature of CDNs.

Compared to other VPS-based schemes, domain shadowing is (possibly) faster and does not need dedicated third-party support. It is faster because compared to the proxy-on-VPS scheme that uses a self-deployed proxy to relay the traffic, domain shadowing's relay is actually all the CDN's edge servers that operate on the CDN's high-speed backbone network, and the whole infrastructure is optimized specifically to distribute content fast and reliably. The following figure compares the delay of fetching a web document directly from the origin server, using Psiphon, using proxy-over-EC2 (with 2 instances based on different hardware configuration), and using domain shadowing based on 5 different CDN providers (Fastly, Azure CDN, Google CDN, AWS Cloudfront, and StakePath). From the figure, we can see domain shadowing beats other schemes most of the time.

image: domain shadowing is (possibly) faster

Challenges of Domain Shadowing
At this moment, domain shadowing faces the following main challenges:

Complexity: The user must config the frontend and backend domains in their CDN account for every censored domain they want to visit. Although such configuration can be automated using the CDN’s API, the user still needs to have sufficient knowledge about relatively complex operations such as how to register to a CDN, enable API configuration and obtain API credentials, and how to register a domain.

Cost: Based on our survey, for 500 GB monthly data usage, the cost of using domain shadowing with a reputable CDN is about $40, which increases or decreases linearly with the data usage in general. If the user chooses to use an inexpensive CDN, the cost could be brought to under $10 per month. However, this still can't beat free tools such as Psiphon and Tor.

Security: By using domain shadowing, the browser "thinks" it is only communicating with the shadow domain, while the web documents are actually from all the different censored domains (see the following figure where we visit Facebook using Forbes.com as the shadow domain). Such domain transformation makes the Same-Origin-Policy no longer enforceable. While we can use a browser extension to help with this issue to some extent, the user must be aware and cautious about what websites to visit.

Facebook screenshot

Privacy: CDNs intercept all HTTP and HTTPS traffic. That is, when a CDN is involved, the HTTPS is no longer between the client and the origin server but between the client and the CDN edge server. Thus, the CDN is able to view and modify any and all traffic between the user and the target server. While this is very unlikely, especially for large and reputable CDNs, users should be aware of the possibility.

Conclusion
We explained domain shadowing, a new technique that achieves censorship circumvention using CDN as leverage. It differs from domain fronting, but can work hand-in-hand with domain fronting to achieve better blocking-resistance. While significant work is still needed to address all the challenges and make it deployable, we see domain shadowing as a promising technique to achieve better censorship circumvention.

...
@kushal April 21, 2021 - 07:46 • 21 days ago
Adding dunder methods to a Python class written in Rust

Last week I did two rounds of my Creating Python modules in Rust workshop. During the second session on Sunday, someone asked if we can create standard dunder methods, say __str__ or __repr__. I never did that before, and during the session I tried to read the docs and implement it. And I failed :)

Later I realized that I should have read the docs carefully. To add those methods, we will have to implement PyObjectProtocol for the Rust structure.

#[pyproto]
impl PyObjectProtocol for Ros {
    fn __repr__(&self) -> PyResult<String> {
        let cpus = self.sys.get_processors().len();
        let repr = format!("<Ros(CPUS: {})>", cpus);
        Ok(repr)
    }

    fn __str__(&self) -> PyResult<String> {
        let cpus = self.sys.get_processors().len();
        let repr = format!("<Ros(CPUS: {})>", cpus);
        Ok(repr)
    }
}
>>> from randomos import Ros
>>> r = Ros()
>>> r
<Ros (CPUS: 8)>
>>> str(r)
'<Ros (CPUS: 8)>'

This code example is in the ros-more branch of the code.

...
@blog April 20, 2021 - 15:59 • 22 days ago
New Release: Tor Browser 10.0.16
New Release: Tor Browser 10.0.16 sysrqb April 20, 2021

Tor Browser 10.0.16 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Firefox to 78.10esr. In addition, Tor Browser 10.0.16 updates NoScript to 11.2.4, and adds localization in Burmese. This version includes important security updates to Firefox for Desktop.

Warning:
Tor Browser will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

Note: New macOS Users, please report if you experience trouble with Gatekeeper when installing this Tor Browser version, and provide the error and the version of macOS you are using.

Note: The Android Tor Browser update will be available next week.

The full changelog since Desktop Tor Browser 10.0.15:

  • Windows + OS X + Linux
    • Update Firefox to 78.10.0esr
    • Update NoScript to 11.2.4
    • Bug 40007: Update domain fronting config for Moat
    • Bug 40390: Add Burmese as a new locale
    • Bug 40408: Disallow SVG Context Paint in all web content

Changes:

  • Updated on 2021-04-23 to include Mozilla's Security Advisory
...
@blog April 15, 2021 - 12:11 • 27 days ago
New Alpha Release: Tor 0.4.6.2-alpha
New Alpha Release: Tor 0.4.6.2-alpha nickm April 15, 2021

There's a new alpha release available for download. If you build Tor from source, you can download the source code for 0.4.6.2-alpha from the download page on the website. Packages should be available over the coming weeks, with a new alpha Tor Browser release likely some time next week.

Remember, this is an alpha release: you should only run this if you'd like to find and report more bugs than usual.

Tor 0.4.6.2-alpha is the second alpha in its series. It fixes several small bugs in previous releases, and solves other issues that had enabled denial-of-service attacks and affected integration with other tools.

Changes in version 0.4.6.2-alpha - 2021-04-15

  • Minor features (client):
    • Clients now check whether their streams are attempting to re-enter the Tor network (i.e. to send Tor traffic over Tor), and close them preemptively if they think exit relays will refuse them for this reason. See ticket 2667 for details. Closes ticket 40271.
  • Minor features (command line):
    • Add long format name "--torrc-file" equivalent to the existing command-line option "-f". Closes ticket 40324. Patch by Daniel Pinto.

 

  • Minor features (dormant mode):
    • Add a new 'DormantTimeoutEnabled' option to allow coarse-grained control over whether the client ever becomes dormant from inactivity. Most people won't need this. Closes ticket 40228.
  • Minor features (fallback directory list):
    • Regenerate the list of fallback directories to contain a new set of 200 relays. Closes ticket 40265.
  • Minor features (geoip data):
    • Update the geoip files to match the IPFire Location Database, as retrieved on 2021/04/13.
  • Minor features (logging):
    • Edit heartbeat log messages so that more of them begin with the string "Heartbeat: ". Closes ticket 40322; patch from 'cypherpunks'.
  • Minor bugfixes (bridge, pluggable transport):
    • Fix a regression that made it impossible start Tor using a bridge line with a transport name and no fingerprint. Fixes bug 40360; bugfix on 0.4.5.4-rc.
  • Minor bugfixes (channel, DoS):
    • Fix a non-fatal BUG() message due to a too-early free of a string, when listing a client connection from the DoS defenses subsystem. Fixes bug 40345; bugfix on 0.4.3.4-rc.
  • Minor bugfixes (compilation):
    • Fix a compilation warning about unused functions when building with a libc that lacks the GLOB_ALTDIRFUNC constant. Fixes bug 40354; bugfix on 0.4.5.1-alpha. Patch by Daniel Pinto.
  • Minor bugfixes (configuration):
    • Fix pattern-matching for directories on all platforms when using %include options in configuration files. This patch also fixes compilation on musl libc based systems. Fixes bug 40141; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (relay):
    • Move the "overload-general" line from extrainfo to the server descriptor. Fixes bug 40364; bugfix on 0.4.6.1-alpha.
  • Minor bugfixes (testing, BSD):
    • Fix pattern-matching errors when patterns expand to invalid paths on BSD systems. Fixes bug 40318; bugfix on 0.4.5.1-alpha. Patch by Daniel Pinto.
  • Documentation (manual):
    • Move the ServerTransport* options to the "SERVER OPTIONS" section. Closes issue 40331.
    • Indicate that the HiddenServiceStatistics option also applies to bridges. Closes ticket 40346.
    • Move the description of BridgeRecordUsageByCountry to the section "STATISTICS OPTIONS". Closes ticket 40323.
...
@blog April 14, 2021 - 00:50 • 28 days ago
New Release: Tor Browser 10.5a14
New Release: Tor Browser 10.5a14 sysrqb April 13, 2021

Tor Browser 10.5a14 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This release updates NoScript to 11.2.4 and updates the Snowflake pluggable transport. This release is the first version that is localized in Burmese, as well. Please report issues as comments here, or through the support channels

Important Note: Tor Browser Alpha versions do not support version 2 onion services. Please see the previously published deprecation timeline.

Note: Tor Browser 10.5 does not support CentOS 6.

The full changelog since Tor Browser 10.5a13:

  • All Platforms
    • Update NoScript to 11.2.4
    • Translations update
    • Bug 40261: Bump versions of snowflake and webrtc
    • Bug 40263: Update domain front for Snowflake
    • Bug 40390: Add Burmese as a new locale
  • Windows + OS X + Linux
    • Bug 40007: Update domain fronting config for Moat
...
@kushal April 13, 2021 - 06:05 • 29 days ago
Workshop on writing Python modules in Rust April 2020

I am conducting 2 repeat sessions for a workshop on "Writing Python modules in Rust".

The first session is on 16th April, 1500 UTC onwards, and the repeat session will be on 18th April 0900 UTC. More details can be found in this issue.

You don't have to have any prior Rust knowledge. I will be providing working code, and we will go very slowly to have a working Python module with useful functions in it.

If you are planning to attend or know anyone else who may want to join, then please point them to the issue link.

...
@blog April 5, 2021 - 13:58 • 1 months ago
New Release: Tor Browser 10.5a13
New Release: Tor Browser 10.5a13 sysrqb April 05, 2021

Tor Browser 10.5a13 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This release updates Firefox to 78.9.0esr for desktop and Firefox for Android to 87.0.0. Additionally, we update Tor to 0.4.6.1-alpha and OpenSSL to 1.1.1k and NoScript to 11.2.3. This release includes important security updates to Firefox for Desktop, and similar important security updates to Firefox for Android.

Important Note: This is the first Tor Browser version that does not support version 2 onion services. Please see the previously published deprecation timeline.

Note: Tor Browser 10.5 does not support CentOS 6.

The full changelog since Tor Browser 10.5a11 (windows, macOS, and Linux) and 10.5a12 (Android) is:

  • Windows + OS X + Linux
    • Update Firefox to 78.9.0esr
    • Update NoScript to 11.2.3
    • Update Openssl to 1.1.1k
    • Update Tor to 0.4.6.1-alpha
    • Translations update
    • Bug 40030: DuckDuckGo redirect to html doesn't work
    • Bug 40032: Remove Snowflake survey banner from TB-alpha
  • Android
    • Update Fenix to 87.0.0
    • Bug 40047: Rebase android-components patches for Fenix 87.0.0
    • Bug 40153: Rebase Fenix patches to Fenix 87.0.0
    • Bug 40365: Rebase 10.5 patches on 87.0
    • Bug 40383: Disable dom.enable_event_timing
  • Build System
    • Windows + OS X + Linux
      • Update Go to 1.15.10
      • Bug 23631: Use rootless containers [tor-browser-build]
      • Bug 40016: getfpaths is not setting origin_project
    • Windows
      • Bug 40252: Bump mingw-w64 and clang for Firefox 78.9
...
@blog March 28, 2021 - 02:11 • 2 months ago
New Release: Tor Browser 10.0.15
New Release: Tor Browser 10.0.15 sysrqb March 27, 2021

Update: 9 April 2021: Android Tor Browser 10.0.15 is now available.

Tor Browser 10.0.15 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Openssl to 1.1.1k. In addition, Tor Browser 10.0.15 includes a bugfix for when Javascript is disabled on websites.

Relay operators who use the Windows Expert Bundle are strongly encouraged to upgrade their relay.

Note: Tor Browser will stop supporting version 2 onion services in June (two months from now). Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Desktop Tor Browser 10.0.14 and Android Tor Browser 10.0.12 is:

  • Windows + OS X + Linux + Android
    • Update Openssl to 1.1.1k
    • Bug 40030: Add 'noscript' capability to NoScript
  • Android
    • Update Fenix to 87.0.0
    • Update NoScript to 11.2.4
    • Update Tor to 0.4.5.7
    • Translations update
    • Bug 40045: Add External App Prompt for Sharing Images
    • Bug 40047: Rebase android-components patches for Fenix 87.0.0
    • Bug 40151: Remove survey banner on TBA-stable
    • Bug 40153: Rebase Fenix patches to Fenix 87.0.0
    • Bug 40365: Rebase 10.5 patches on 87.0
    • Bug 40383: Disable dom.enable_event_timing
  • Build System
    • Android
      • Bug 40162: Build Fenix instrumented tests apk
      • Bug 40172: Move Gradle compilers out of android-toolchain to own gradle project
      • Bug 40241: Update components for mozilla87-based Fenix
...
@blog March 27, 2021 - 22:04 • 2 months ago
Onionize your Workflow with the Onion Guide Fanzine
Onionize your Workflow with the Onion Guide Fanzine Gaba March 27, 2021

At the Tor Project, we build technologies that allow anybody to access the Internet privately. We maintain the software that runs the Tor network as well as utilities and clients to use the network. We also collect anonymous data on the network that allows us to detect problems that may occur, and we connect with the users of the Tor network through training and feedback exercises that help to improve our tools.

In some places, the organizations and individuals we work with are in risk of persecution for the digital services they run. It could be reproductive rights services that are criminalized in some countries or content that is censored by an Internet provider or government. Or it could be that they need to protect their own users when accessing their content and find a way for their community to use Tor Browser for protection.
 
One way we help human rights defenders and organizations take back their right to privacy online is by helping them to use and set up onion services. Websites that are only accessible over Tor are called "onions" and end in the TLD .onion. For example, the DuckDuckGo onion is https://3g2upl4pq6kufc4m.onion. You can access these websites by using Tor Browser. The addresses must be shared with you by the website host, as onions are not indexed in search engines in the typical way that other websites are.
 
Last year, thanks to the support of Digital Defenders Partnership, we wrote a series of Onion Guides intended to make it easier for our partners to correctly and safely set up their own onion services. To create these Onion Guides, we collected and improved existing disparate information about the benefits of onion services and how to set them up for a website. 
 
During the last activity of this project, we ran a survey between December 2020 and January 2021. The participants were partner organizations and individuals who were known to use onion services and had received training from Tor in the past. All questions asked were related to the Onion Guides and onion services. Five people responded to this survey.
“[Tor] offers the possibility for those of us who do work for social transformation to access the Internet safely, without exposing ourselves or exposing our processes, but also, it is a tool that is there and can be even more accessible to different people in different territories.” - Survey response.
When asked if they can define onion services, all participants in this study gave different answers. Some related to specific services, like OnionShare and SecureDrop; others associated onion services to a service without metadata; only two participants answered that it is a service that can only be accessed over the Tor network.
 
When asked if onion services respond to the threats they or their organizations face, most participants answered YES. One of the participants answered NO. Same for the question asking if you feel safer using onion services.
 
When asked to define the best benefit of using onion services, most participants answered (a) anonymity; followed by (b) accessing digital security guides and tools; other mentions were: (c) sharing and storing documents and sensitive information; and (d) NAT punching.
 
When asked if they would recommend onion services to anyone, all survey participants answered YES, because of safety.
 
You can find the Onion Guide in our community portal, well as the section on Onion Services, in English, Spanish and Portuguese. Feel free to use it to set up your own .onion site, and let us know how it works for you.
...
@kushal March 25, 2021 - 14:52 • 2 months ago
The correct spelling is Tor

Tor spelling tweet

The correct spelling is Tor, not TOR or any other variations. Please use the correct spelling of the project.

...
@blog March 24, 2021 - 21:05 • 2 months ago
New Release: Tor Browser 10.0.14
New Release: Tor Browser 10.0.14 sysrqb March 24, 2021

Tor Browser 10.0.14 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Desktop Firefox to 78.9.0esr. In addition, Tor Browser 10.0.14 updates NoScript to 11.2.3, and Tor to 0.4.5.7. This version includes important security updates to Firefox for Desktop.

Note: An update for Android Tor Browser is not included in this release.

The full changelog since Desktop Tor Browser 10.0.13 is:

  • Windows + OS X + Linux
    • Update Firefox to 78.9.0esr
    • Update NoScript to 11.2.3
    • Update Tor to 0.4.5.7
    • Bug 40031: Remove survey banner on TB-stable
  • Build System
    • Windows
      • Bug 40249: Bump mingw-w64 and clang for Firefox 78.9
...
@blog March 24, 2021 - 16:58 • 2 months ago
Get a TLS certificate for your onion site
Get a TLS certificate for your onion site isabela March 24, 2021

We are happy to share the news of another important milestone for .onion services! You can now get DV certificates for your v3 onion site using HARICA, a Root CA Operator founded by Academic Network (GUnet), a civil society nonprofit from Greece.

Last year we wrote a blog post about the challenges and opportunities for onion services:

The future of TLS support for onion services is very encouraging. In March of this year, the CA/Browser Forum approved an amendment to the domain validation (DV) TLS certificate baseline requirements which now allows certificate authorities (CAs) to issue certificates containing v3 .onion addresses. This means, in the not-too-distant-future, a CA like Let's Encrypt can issue a certificate for an onion service and Tor Browser will "just work." In addition, for onion services that do not want to rely on certificate authorities, we are exploring alternative designs like Same Origin Onion Certificate for inclusion in Tor Browser.

We are happy that the ‘not-too-distant-future’ was indeed quite close. We hope that more CAs do the same. 

Why would an .onion site need a TLS certificate? This is a great question. Especially because .onion services provide pretty much the same protections offered by an HTTPS connection regarding protecting the data in transit from man in the middle attacks and validating that the user is indeed connecting the server the domain in the browser bar is requesting. Onion services do the same thing, so why would an .onion site need a TLS certificate?

Our Community portal page about onion services give you a list of reasons why a service admin would need a TLS certificate as part of their implementation. Here are some of them:

  • Websites with complex setups and that are serving HTTP and HTTPS content
  • To help the user verify that the .onion address is indeed the site you are hosting (this would be a manual check done by the user looking at the cert registration information)
  • Some services work with protocols, frameworks, and other infrastructure that has HTTPS connection as a requirement
  • In case your web server and your tor process are in different machines

Previously, .onion site administrators who needed a TLS certificate had to either hack other solutions or spend a significant amount of money purchasing an EV certificate. Now with HARICA, acquiring a certificate has become more accessible, but we know that free certificates are ideal and are looking forward to that moment.

We are happy to see people acquiring certificates for their onions. Remember to do it for a v3 onion address since v2 will be deprecated very soon:

2. July 15th, 2021
0.4.6.x: Tor will no longer support v2 and support will be removed from the code base.

If you would like to give it a try, here is a great tutorial by Kushal from the Tor community.
 

...
@anarcat March 23, 2021 - 02:03 • 2 months ago
Major email crash with syncmaildir

TL:DR; lost half my mail (150,000 messages, ~6GB) last night. Cause uncertain, but possibly a combination of a dead CMOS battery, systemd OnCalendar=daily, a (locking?) bug in syncmaildir, and generally, a system too exotic and complicated.

The crash

So I somehow lost half my mail:

anarcat@angela:~(main)$ du -sh Maildir/
7,9G    Maildir/

anarcat@curie:~(main)$ du -sh Maildir
14G     Maildir

anarcat@marcos:~$ du -sh Maildir
8,0G    Maildir

Those are three different machines:

  • angela: my laptop, not always on
  • curie: my workstation, mostly always on
  • marcos: my mail server, always on

Those mails are synchronized using a rather exotic system based on SSH, syncmaildir and rsendmail.

The anomaly started on curie:

-- Reboot --
mar 22 16:13:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:13:00 curie smd-pull[4801]: rm: impossible de supprimer '/home/anarcat/.smd/workarea/Maildir': Le dossier n'est pas vide
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:13:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:14:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:14:00 curie smd-pull[7025]:  4091 ?        00:00:00 smd-push
mar 22 16:14:00 curie smd-pull[7025]: Already running.
mar 22 16:14:00 curie smd-pull[7025]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:14:00 curie smd-pull[7025]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 4091) run(rm /home/anarcat/.smd/lock))
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:14:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

Then it seems like smd-push (from curie) started destroying the universe for some reason:

mar 22 16:20:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:20:00 curie smd-pull[9319]:  4091 ?        00:00:00 smd-push
mar 22 16:20:00 curie smd-pull[9319]: Already running.
mar 22 16:20:00 curie smd-pull[9319]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:20:00 curie smd-pull[9319]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(ru
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:20:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:21:34 curie smd-push[4091]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:21:35 curie smd-push[9374]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:21:35 curie systemd[3199]: smd-push.service: Succeeded.

Notice the del-mails(293920) there: it is actively trying to destroy basically every email in my mail spool.

Then somehow push and pull started both at once:

mar 22 16:21:35 curie systemd[3199]: Started push emails with syncmaildir.
mar 22 16:21:35 curie systemd[3199]: Starting push emails with syncmaildir...
mar 22 16:22:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:22:00 curie smd-pull[10333]:  9455 ?        00:00:00 smd-push
mar 22 16:22:00 curie smd-pull[10333]: Already running.
mar 22 16:22:00 curie smd-pull[10333]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:22:00 curie smd-pull[10333]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: Data transmission failed.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marco
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open requested file.
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:22:00 curie smd-push[9455]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start push emails with syncmaildir.

There it seems push tried to destroy the universe again: del-mails(293920).

Interestingly, the push started again in parallel with the pull, right that minute:

mar 22 16:22:00 curie systemd[3199]: Starting push emails with syncmaildir...

... but didn't complete for a while, here's pull trying to start again:

mar 22 16:24:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:24:00 curie smd-pull[12051]: 10466 ?        00:00:00 smd-push
mar 22 16:24:00 curie smd-pull[12051]: Already running.
mar 22 16:24:00 curie smd-pull[12051]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:24:00 curie smd-pull[12051]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 10466) run(rm /home/anarcat/.smd/lock))
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

... and the long push finally resolving:

mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: No such file or directory
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open requested file.
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie smd-push[10466]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start push emails with syncmaildir.
mar 22 16:24:00 curie systemd[3199]: Starting push emails with syncmaildir...

This pattern repeats until 16:35, when that locking issue silently recovered somehow:

mar 22 16:35:03 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:35:41 curie smd-pull[20788]: default: smd-client@localhost: TAGS: stats::new-mails(5), del-mails(1), bytes-received(21885), xdelta-received(6863398)
mar 22 16:35:42 curie smd-pull[21373]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:35:42 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:35:42 curie systemd[3199]: Started pull emails with syncmaildir.
mar 22 16:36:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:36:36 curie smd-pull[21738]: default: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(214)
mar 22 16:36:37 curie smd-pull[21816]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:36:37 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:36:37 curie systemd[3199]: Started pull emails with syncmaildir.

... notice that huge xdelta-received there, that's 7GB right there. Mysteriously, the curie mail spool survived this, possibly because smd-pull started failing again:

mar 22 16:38:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:38:00 curie smd-pull[23556]: 21887 ?        00:00:00 smd-push
mar 22 16:38:00 curie smd-pull[23556]: Already running.
mar 22 16:38:00 curie smd-pull[23556]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:38:00 curie smd-pull[23556]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 21887) run(rm /home/anarcat/.smd/lock))
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:38:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

That could have been when i got on angela to check my mail, and it was busy doing the nasty removal stuff... although the times don't match. Here is when angela came back online:

anarcat@angela:~(main)$ last
anarcat  :0           :0               Mon Mar 22 19:57   still logged in
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 19:57   still running
anarcat  :0           :0               Mon Mar 22 17:43 - 18:47  (01:03)
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 17:39   still running

Then finally the sync on curie started failing with:

mar 22 16:46:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:46:42 curie smd-pull[27455]: smd-server: ERROR: Client aborted, removing /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.new and /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.mtime.new
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: Failed to copy Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: The destination already exists but its content differs.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: To fix this problem you have two options:
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - rename Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S by hand so that Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   can be copied without replacing it.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   Executing `cd; mv -n "Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "Maildir/.koumbit/cur/1616446002.1.localhost"` should work.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - run smd-push so that your changes to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   are propagated to the other mailbox
mar 22 16:46:42 curie smd-pull[27455]: default: smd-client@localhost: TAGS: error::context(copy-message) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "/home/anarcat/.smd/workarea/Maildir/.koumbit/tmp/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S") run(smd-push default))
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:46:42 curie systemd[3199]: Failed to start pull emails with syncmaildir.

It went on like this until I found the problem. This is, presumably, a good thing because those emails were not being destroyed.

On angela, things looked like this:

-- Reboot --
mar 22 17:39:29 angela systemd[1677]: Started run notmuch new at least once a day.
mar 22 17:39:29 angela systemd[1677]: Started run smd-pull regularly.
mar 22 17:40:46 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open Maildir/.tor/new/1616446842.M285912P26118.marcos,S=8860,W=8996: Maildir/.tor/new/1616446842.M285912P26118.marcos,S=886
0,W=8996: No such file or directory
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open requested file.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: Data transmission failed.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: This problem is transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: server sent ABORT or connection died
mar 22 17:43:18 angela smd-pull[3916]: default: smd-server@smd-server-anarcat: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested
-actions(retry)
mar 22 17:43:18 angela smd-pull[3916]: default: smd-client@localhost: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Failed with result 'exit-code'.
mar 22 17:43:18 angela systemd[1677]: Failed to start pull emails with syncmaildir.
mar 22 17:43:18 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:29 angela smd-pull[4847]: default: smd-client@localhost: TAGS: stats::new-mails(29), del-mails(0), bytes-received(401519), xdelta-received(38914)
mar 22 17:43:29 angela smd-pull[5600]: register: smd-client@localhost: TAGS: stats::new-mails(2), del-mails(0), bytes-received(92150), xdelta-received(471)
mar 22 17:43:29 angela systemd[1677]: smd-pull.service: Succeeded.
mar 22 17:43:29 angela systemd[1677]: Started pull emails with syncmaildir.
mar 22 17:43:29 angela systemd[1677]: Starting push emails with syncmaildir...
mar 22 17:43:32 angela smd-push[5693]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(217)
mar 22 17:43:33 angela smd-push[6575]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(219)
mar 22 17:43:33 angela systemd[1677]: smd-push.service: Succeeded.
mar 22 17:43:33 angela systemd[1677]: Started push emails with syncmaildir.

Notice how long it took to get the first error, in that first failure: it failed after 3 minutes! Presumably that's when it started deleting all that mail. And this is during pull, not push, so the error didn't come from angela.

Affected data

It seems 2GB of mail from my main INBOX was destroyed. Another 2.4GB of spam (kept for training purposes) was also destroyed, along with 700MB of Sent mail. The rest is hard to figure out, because the folders are actually still there, just smaller. So I relied on ncdu to figure out the size changes.

(Note that I don't really archive (or delete much of) my mail since I use notmuch, which is why the INBOX is so large...)

Concretely, according to the notmuch-new.service which still runs periodically on marcos, here are the changes that happened on the server:

mar 22 16:17:12 marcos notmuch[10729]: Added 7 new messages to the database. Removed 57985 messages. Detected 1372 file renames.
mar 22 16:22:43 marcos notmuch[12826]: No new mail. Removed 143842 messages. Detected 6072 file renames.
mar 22 16:27:02 marcos notmuch[13969]: No new mail. Removed 82071 messages. Detected 1783 file renames.
mar 22 16:29:45 marcos notmuch[15079]: Added 22743 new messages to the database. Detected 1 file rename.
mar 22 16:31:48 marcos notmuch[16196]: Added 22779 new messages to the database. Removed 5 messages.
mar 22 16:33:11 marcos notmuch[17192]: Added 3711 new messages to the database.
mar 22 16:40:41 marcos notmuch[19122]: Added 74558 new messages to the database. Detected 1 file rename.
mar 22 16:43:21 marcos notmuch[20325]: Added 9061 new messages to the database. Detected 4 file renames.
mar 22 17:43:08 marcos notmuch[7420]: Added 1793 new messages to the database. Detected 6 file renames.

That is basically the entire mail spool destroyed at first (283 898 messages), and then bits and pieces of it progressively re-added (134 645 messages), somehow, so 149 253 mails were lost, presumably.

Recovery

I disabled the services all over the place:

systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer

(Well, technically, I did that only on angela, as I thought the problem was there. Luckily, curie kept going but it seems like it was harmless.)

I made a backup of the mail spool on curie:

tar cf - Maildir/ | pv -s 14G | gzip -c > Maildir.tgz

Then I crossed my fingers and ran smd-push -v -s, as that was suggested by smd error codes themselves. That thankfully started restoring mail. It failed a few times on weird cases of files being duplicates, but I resolved this by following the instructions. Or mostly: I actually deleted the files instead of moving them, which made smd even unhappier (if there ever was such a thing). I had to recreate some of those files, so, lesson learned: do follow the advice smd gives you, even if it seems useless or strange.

But then smd-push was humming along, uploading tens of thousands of messages, saturating the upload in the office, refilling the mail spool on the server... yaay!... ?

Except... well, of course that didn't quite work: the mail spool in the office eventually started to grow beyond the size of the mail spool on the workstation. That is what smd-push eventually settled on:

default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(151697), del-mails(0), bytes-received(7539147811), xdelta-received(10881198)

It recreated 151 697 emails, adding about 2000 emails to the pool, kind of from nowhere at all. On marcos, before:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,0 GiB [##########] /.notmuch
  717,3 MiB [#         ] /.Archives.2014
  498,2 MiB [#         ] /.feeds.debian-planet
  453,1 MiB [#         ] /.Archives.2012
  414,5 MiB [#         ] /.debian
  408,2 MiB [#         ] /.quoifaire
  389,8 MiB [          ] /.rapports
  356,6 MiB [          ] /.tor
  182,6 MiB [          ] /.koumbit
  179,8 MiB [          ] /tmp
   56,8 MiB [          ] /.nn
   43,0 MiB [          ] /.act-mtl
   32,6 MiB [          ] /.feeds.sysadvent
   31,7 MiB [          ] /.feeds.releases
   31,4 MiB [          ] /.Sent.2005
   26,3 MiB [          ] /.sage
   25,5 MiB [          ] /.freedombox
   24,0 MiB [          ] /.feeds.git-annex
   21,1 MiB [          ] /.Archives.2011
   19,1 MiB [          ] /.Sent.2003
   16,7 MiB [          ] /.bugtraq
   16,2 MiB [          ] /.mlug
 Total disk usage:   8,0 GiB  Apparent size:   7,6 GiB  Items: 184426

After:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,7 GiB [##########] /.notmuch
    2,7 GiB [#####     ] /.junk
    1,9 GiB [###       ] /cur
  717,3 MiB [#         ] /.Archives.2014
  659,3 MiB [#         ] /.Sent
  513,9 MiB [#         ] /.Archives.2012
  498,2 MiB [#         ] /.feeds.debian-planet
  449,6 MiB [          ] /.Archives.2015
  414,5 MiB [          ] /.debian
  408,2 MiB [          ] /.quoifaire
  389,8 MiB [          ] /.rapports
  380,8 MiB [          ] /.Archives.2013
  356,6 MiB [          ] /.tor
  261,1 MiB [          ] /.Archives.2011
  240,9 MiB [          ] /.koumbit
  183,6 MiB [          ] /.Archives.2010
  179,8 MiB [          ] /tmp
  128,4 MiB [          ] /.lists
  106,1 MiB [          ] /.inso-interne
  103,0 MiB [          ] /.github
   75,0 MiB [          ] /.nanog
   69,8 MiB [          ] /.full-disclosure
 Total disk usage:  16,2 GiB  Apparent size:  15,5 GiB  Items: 341143

That is 156 717 files more.

On curie:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------------------------------------
    2,7 GiB [##########] /.junk
    2,3 GiB [########  ] /.notmuch
    1,9 GiB [######    ] /cur
  661,2 MiB [##        ] /.Archives.2014
  655,3 MiB [##        ] /.Sent
  512,0 MiB [#         ] /.Archives.2012
  447,3 MiB [#         ] /.Archives.2015
  438,5 MiB [#         ] /.feeds.debian-planet
  406,5 MiB [#         ] /.quoifaire
  383,6 MiB [#         ] /.debian
  378,6 MiB [#         ] /.Archives.2013
  303,3 MiB [#         ] /.tor
  296,0 MiB [#         ] /.rapports
  237,6 MiB [          ] /.koumbit
  233,2 MiB [          ] /.Archives.2011
  182,1 MiB [          ] /.Archives.2010
  127,0 MiB [          ] /.lists
  104,8 MiB [          ] /.inso-interne
  102,7 MiB [          ] /.register
   89,6 MiB [          ] /.github
   67,1 MiB [          ] /.full-disclosure
   66,5 MiB [          ] /.nanog
 Total disk usage:  13,3 GiB  Apparent size:  12,6 GiB  Items: 342465

Interestingly, there are more files, but less disk usage. It's possible the notmuch database there is more efficient. So maybe there's nothing to worry about.

Last night's marcos backup has:

root@marcos:/home/anarcat# find /mnt/home/anarcat/Maildir | pv -l | wc -l
 341k 0:00:16 [20,4k/s] [                             <=>                                                                                                                                     ]
341040

... 341040 files, which seems about right, considering some mail was delivered during the day. An audit can be performed with hashdeep:

borg mount /media/sdb2/borg/::marcos-auto-2021-03-22 /mnt
hashdeep -c sha256 -r /mnt/home/anarcat/Maildir | pv -l -s 341k > Maildir-backup-manifest.txt

And then compared with:

hashdeep -c sha256 -k Maildir-backup-manifest.txt Maildir/

Some extra files should show up in the Maildir, and very few should actually be missing, because I shouldn't have deleted mail from the previous day the next day, or at least very few. The actual summary hashdeep gave me was:

hashdeep: Audit failed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 339080
Files partially matched: 0
            Files moved: 782
        New files found: 107
  Known files not found: 106

So 106 files added, 107 deleted. Seems good enough for me...

Postfix was stopped at Mar 22 21:12:59 to try and stop external events from confusing things even further. I reviewed the delivery log to see if mail that came in during the problem window disappeared:

grep 'dovecot:.*stored mail into mailbox' /var/log/mail.log |
  tail -20 |
  sed 's/.*msgid=<//;s/>.*//' | 
  while read msgid; do 
    notmuch count --exclude=false id:$msgid |
      grep 0 && echo $msgid missing;
  done

And things looked okay. Now of course if we go further back, we find mail I actually deleted (because I do do that sometimes), so it's hard to use this log as an audit trail. We can only hope that the curie spool is sufficiently coherent to be relied on.

Worst case, we'll have to restore from last night's backup, but that's getting far away now: I get hundreds of mails a day in that mail spool, and reseting back to last night does not seem like a good idea.

A dry run of smd-pull on angela seems to agree that it's missing some files:

default: smd-client@localhost: TAGS: stats::new-mails(154914), del-mails(0), bytes-received(0), xdelta-received(0)

... a number of mails somewhere in between the other two, go figure. A "wet" run of this was started, without deletion (-n), which gave us:

default: smd-client@localhost: TAGS: stats::new-mails(154911), del-mails(0), bytes-received(7658160107), xdelta-received(10837609)

Strange that it sync'd three less emails, but that's still better than nothing, and we have a mail spool on angela again:

anarcat@angela:~(main)$ notmuch new
purging with prefix '.': spam moved (0), ham moved (0), deleted (0), done
Note: Ignoring non-mail file: /home/anarcat/Maildir//.uidvalidity
Processed 1779 total files in 26s (66 files/sec.).
Added 1190 new messages to the database. Removed 3 messages. Detected 593 file renames.
tagging with prefix '.': spam, sent, feeds, koumbit, tor, lists, rapports, folders, done.

Notice how only 1190 messages were re-added, that is because I killed notmuch before it had time to remove all those mails from its database.

Possible causes

I am totally at a loss as to why smd started destroying everything like it did. But a few things come to mind:

  1. I rewired my office on that day.

  2. This meant unplugging curie, the workstation.

  3. It has a bad CMOS battery (known problem), so it jumped around the time continuum a few times, sometimes by years.

  4. The smd services are ran from a systemd unit with OnCalendar=*:0/2. I have heard that it's possible that major time jumps "pile up" execution of jobs, and it seems this happened in this case.

  5. It's possible that locking in smd is not as great as it could be, and that it corrupted its internal data structures on curie, which led it to command a destruction of the remote mail spool.

It's also possible that there was a disk failure on the server, marcos. But since it's running on a (software) RAID-1 array, and no errors have been found (according to dmesg), I don't think that's a plausible hypothesis.

Lessons learned

  1. follow what smd says, even if it seems useless or strange.

  2. trust but verify: just backup everything before you do anything, especially the largest data set.

  3. daily backups are not great for email, unless you're ready to lose a day of email (which I'm not).

  4. hashdeep is great. I keep finding new use cases for it. Last time it was to audit my camera SD card to make sure I didn't forget anything, and now this. it's fast and powerful.

  5. borg is great too. the FUSE mount was especially useful, and it was pretty fast to explore the backup, even through that overhead: checksumming 15GB of mail took about 35 minutes, which gives a respectable 8MB/s, probably bottlenecked by the crap external USB drive I use for backups (!).

  6. I really need to finish my backup system so that I have automated offsite backups, although in this case that would actually have been much slower (certainly not 8MB/s!).

Workarounds and solutions

I setup fake-hwclock on curie, so that the next power failure will not upset my clock that badly.

I am thinking of switching to ZFS or BTRFS for most of my filesystems, so that I can use filesystem snapshots (including remotely!) as a backup strategy. This seems so much more powerful than crawling the filesystem for changes, and allows for truly offsite backups protected from an attacker (hopefully). But it's a long way there.

I'm also thinking of rebuilding my mail setup without smd. It's not the first time something like this happens with smd. It's the first time I am more confident it's the root cause of the problem, however, and it makes me really nervous for the future.

I have used offlineimap in the past and it seems it was finally ported to Python 3 so that could be an option again. isync/mbsync is another option, which I tried before but do not remember why I didn't switch. A complete redesign with something like getmail and/or nncp could also be an option. But alas, I lack the time to go crazy with those experiments.

Somehow, doing like everyone else and just going with Google still doesn't seem to be an option for me. Screw big tech. But I am afraid they will win, eventually.

In any case, I'm just happy I got mail again, strangely.

...
@blog March 21, 2021 - 16:27 • 2 months ago
New Release: Tor Browser 10.5a12 (Android Only)
New Release: Tor Browser 10.5a12 (Android Only) sysrqb March 21, 2021

Tor Browser 10.5a12 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release for Android instead.

This release updates Fenix to 87.0.0-beta.2. Additionally, we update NoScript to 11.2.3 and Tor to 0.4.6.1-alpha.

Note: This is the first Tor Browser version that does not support version 2 onion services. Please see the previously published deprecation timeline.

The full changelog since Tor Browser 10.5a11 is:

  • Android
    • Update Fenix to 87.0.0-beta.2
    • Update NoScript to 11.2.3
    • Update Tor to 0.4.6.1-alpha
    • Translations update
    • Bug 40030: DuckDuckGo redirect to html doesn't work
    • Bug 40043: Rebase android-components patches for Fenix 87 beta 2 builds
    • Bug 40045: Add External App Prompt for Sharing Images
    • Bug 40150: Rebase Fenix patches to Fenix 87 beta 2
    • Bug 40361: Rebase tor-browser patches to 87.0b4
  • Build System
    • Android
      • Update Go to 1.15.10
      • Bug 23631: Use rootless containers
      • Bug 40016: getfpaths is not setting origin_project
      • Bug 40172: Move Gradle compilers out of android-toolchain to own gradle project
      • Bug 40241: Update components for mozilla87-based Fenix
...