Planet Tor

@blog April 15, 2021 - 12:11 • a day ago
New Alpha Release: Tor 0.4.6.2-alpha
New Alpha Release: Tor 0.4.6.2-alpha nickm April 15, 2021

There's a new alpha release available for download. If you build Tor from source, you can download the source code for 0.4.6.2-alpha from the download page on the website. Packages should be available over the coming weeks, with a new alpha Tor Browser release likely some time next week.

Remember, this is an alpha release: you should only run this if you'd like to find and report more bugs than usual.

Tor 0.4.6.2-alpha is the second alpha in its series. It fixes several small bugs in previous releases, and solves other issues that had enabled denial-of-service attacks and affected integration with other tools.

Changes in version 0.4.6.2-alpha - 2021-04-15

  • Minor features (client):
    • Clients now check whether their streams are attempting to re-enter the Tor network (i.e. to send Tor traffic over Tor), and close them preemptively if they think exit relays will refuse them for this reason. See ticket 2667 for details. Closes ticket 40271.
  • Minor features (command line):
    • Add long format name "--torrc-file" equivalent to the existing command-line option "-f". Closes ticket 40324. Patch by Daniel Pinto.

 

  • Minor features (dormant mode):
    • Add a new 'DormantTimeoutEnabled' option to allow coarse-grained control over whether the client ever becomes dormant from inactivity. Most people won't need this. Closes ticket 40228.
  • Minor features (fallback directory list):
    • Regenerate the list of fallback directories to contain a new set of 200 relays. Closes ticket 40265.
  • Minor features (geoip data):
    • Update the geoip files to match the IPFire Location Database, as retrieved on 2021/04/13.
  • Minor features (logging):
    • Edit heartbeat log messages so that more of them begin with the string "Heartbeat: ". Closes ticket 40322; patch from 'cypherpunks'.
  • Minor bugfixes (bridge, pluggable transport):
    • Fix a regression that made it impossible start Tor using a bridge line with a transport name and no fingerprint. Fixes bug 40360; bugfix on 0.4.5.4-rc.
  • Minor bugfixes (channel, DoS):
    • Fix a non-fatal BUG() message due to a too-early free of a string, when listing a client connection from the DoS defenses subsystem. Fixes bug 40345; bugfix on 0.4.3.4-rc.
  • Minor bugfixes (compilation):
    • Fix a compilation warning about unused functions when building with a libc that lacks the GLOB_ALTDIRFUNC constant. Fixes bug 40354; bugfix on 0.4.5.1-alpha. Patch by Daniel Pinto.
  • Minor bugfixes (configuration):
    • Fix pattern-matching for directories on all platforms when using %include options in configuration files. This patch also fixes compilation on musl libc based systems. Fixes bug 40141; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (relay):
    • Move the "overload-general" line from extrainfo to the server descriptor. Fixes bug 40364; bugfix on 0.4.6.1-alpha.
  • Minor bugfixes (testing, BSD):
    • Fix pattern-matching errors when patterns expand to invalid paths on BSD systems. Fixes bug 40318; bugfix on 0.4.5.1-alpha. Patch by Daniel Pinto.
  • Documentation (manual):
    • Move the ServerTransport* options to the "SERVER OPTIONS" section. Closes issue 40331.
    • Indicate that the HiddenServiceStatistics option also applies to bridges. Closes ticket 40346.
    • Move the description of BridgeRecordUsageByCountry to the section "STATISTICS OPTIONS". Closes ticket 40323.
...
@blog April 14, 2021 - 00:50 • 3 days ago
New Release: Tor Browser 10.5a14
New Release: Tor Browser 10.5a14 sysrqb April 13, 2021

Tor Browser 10.5a14 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This release updates NoScript to 11.2.4 and updates the Snowflake pluggable transport. This release is the first version that is localized in Burmese, as well. Please report issues as comments here, or through the support channels

Important Note: Tor Browser Alpha versions do not support version 2 onion services. Please see the previously published deprecation timeline.

Note: Tor Browser 10.5 does not support CentOS 6.

The full changelog since Tor Browser 10.5a13:

  • All Platforms
    • Update NoScript to 11.2.4
    • Translations update
    • Bug 40261: Bump versions of snowflake and webrtc
    • Bug 40263: Update domain front for Snowflake
    • Bug 40390: Add Burmese as a new locale
  • Windows + OS X + Linux
    • Bug 40007: Update domain fronting config for Moat
...
@kushal April 13, 2021 - 06:05 • 4 days ago
Workshop on writing Python modules in Rust April 2020

I am conducting 2 repeat sessions for a workshop on "Writing Python modules in Rust".

The first session is on 16th April, 1500 UTC onwards, and the repeat session will be on 18th April 0900 UTC. More details can be found in this issue.

You don't have to have any prior Rust knowledge. I will be providing working code, and we will go very slowly to have a working Python module with useful functions in it.

If you are planning to attend or know anyone else who may want to join, then please point them to the issue link.

...
@blog April 5, 2021 - 13:58 • 11 days ago
New Release: Tor Browser 10.5a13
New Release: Tor Browser 10.5a13 sysrqb April 05, 2021

Tor Browser 10.5a13 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This release updates Firefox to 78.9.0esr for desktop and Firefox for Android to 87.0.0. Additionally, we update Tor to 0.4.6.1-alpha and OpenSSL to 1.1.1k and NoScript to 11.2.3. This release includes important security updates to Firefox for Desktop, and similar important security updates to Firefox for Android.

Important Note: This is the first Tor Browser version that does not support version 2 onion services. Please see the previously published deprecation timeline.

Note: Tor Browser 10.5 does not support CentOS 6.

The full changelog since Tor Browser 10.5a11 (windows, macOS, and Linux) and 10.5a12 (Android) is:

  • Windows + OS X + Linux
    • Update Firefox to 78.9.0esr
    • Update NoScript to 11.2.3
    • Update Openssl to 1.1.1k
    • Update Tor to 0.4.6.1-alpha
    • Translations update
    • Bug 40030: DuckDuckGo redirect to html doesn't work
    • Bug 40032: Remove Snowflake survey banner from TB-alpha
  • Android
    • Update Fenix to 87.0.0
    • Bug 40047: Rebase android-components patches for Fenix 87.0.0
    • Bug 40153: Rebase Fenix patches to Fenix 87.0.0
    • Bug 40365: Rebase 10.5 patches on 87.0
    • Bug 40383: Disable dom.enable_event_timing
  • Build System
    • Windows + OS X + Linux
      • Update Go to 1.15.10
      • Bug 23631: Use rootless containers [tor-browser-build]
      • Bug 40016: getfpaths is not setting origin_project
    • Windows
      • Bug 40252: Bump mingw-w64 and clang for Firefox 78.9
...
@blog March 28, 2021 - 02:11 • 20 days ago
New Release: Tor Browser 10.0.15
New Release: Tor Browser 10.0.15 sysrqb March 27, 2021

Update: 9 April 2021: Android Tor Browser 10.0.15 is now available.

Tor Browser 10.0.15 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Openssl to 1.1.1k. In addition, Tor Browser 10.0.15 includes a bugfix for when Javascript is disabled on websites.

Relay operators who use the Windows Expert Bundle are strongly encouraged to upgrade their relay.

Note: Tor Browser will stop supporting version 2 onion services in June (two months from now). Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Desktop Tor Browser 10.0.14 and Android Tor Browser 10.0.12 is:

  • Windows + OS X + Linux + Android
    • Update Openssl to 1.1.1k
    • Bug 40030: Add 'noscript' capability to NoScript
  • Android
    • Update Fenix to 87.0.0
    • Update NoScript to 11.2.4
    • Update Tor to 0.4.5.7
    • Translations update
    • Bug 40045: Add External App Prompt for Sharing Images
    • Bug 40047: Rebase android-components patches for Fenix 87.0.0
    • Bug 40151: Remove survey banner on TBA-stable
    • Bug 40153: Rebase Fenix patches to Fenix 87.0.0
    • Bug 40365: Rebase 10.5 patches on 87.0
    • Bug 40383: Disable dom.enable_event_timing
  • Build System
    • Android
      • Bug 40162: Build Fenix instrumented tests apk
      • Bug 40172: Move Gradle compilers out of android-toolchain to own gradle project
      • Bug 40241: Update components for mozilla87-based Fenix
...
@blog March 27, 2021 - 22:04 • 20 days ago
Onionize your Workflow with the Onion Guide Fanzine
Onionize your Workflow with the Onion Guide Fanzine Gaba March 27, 2021

At the Tor Project, we build technologies that allow anybody to access the Internet privately. We maintain the software that runs the Tor network as well as utilities and clients to use the network. We also collect anonymous data on the network that allows us to detect problems that may occur, and we connect with the users of the Tor network through training and feedback exercises that help to improve our tools.

In some places, the organizations and individuals we work with are in risk of persecution for the digital services they run. It could be reproductive rights services that are criminalized in some countries or content that is censored by an Internet provider or government. Or it could be that they need to protect their own users when accessing their content and find a way for their community to use Tor Browser for protection.
 
One way we help human rights defenders and organizations take back their right to privacy online is by helping them to use and set up onion services. Websites that are only accessible over Tor are called "onions" and end in the TLD .onion. For example, the DuckDuckGo onion is https://3g2upl4pq6kufc4m.onion. You can access these websites by using Tor Browser. The addresses must be shared with you by the website host, as onions are not indexed in search engines in the typical way that other websites are.
 
Last year, thanks to the support of Digital Defenders Partnership, we wrote a series of Onion Guides intended to make it easier for our partners to correctly and safely set up their own onion services. To create these Onion Guides, we collected and improved existing disparate information about the benefits of onion services and how to set them up for a website. 
 
During the last activity of this project, we ran a survey between December 2020 and January 2021. The participants were partner organizations and individuals who were known to use onion services and had received training from Tor in the past. All questions asked were related to the Onion Guides and onion services. Five people responded to this survey.
“[Tor] offers the possibility for those of us who do work for social transformation to access the Internet safely, without exposing ourselves or exposing our processes, but also, it is a tool that is there and can be even more accessible to different people in different territories.” - Survey response.
When asked if they can define onion services, all participants in this study gave different answers. Some related to specific services, like OnionShare and SecureDrop; others associated onion services to a service without metadata; only two participants answered that it is a service that can only be accessed over the Tor network.
 
When asked if onion services respond to the threats they or their organizations face, most participants answered YES. One of the participants answered NO. Same for the question asking if you feel safer using onion services.
 
When asked to define the best benefit of using onion services, most participants answered (a) anonymity; followed by (b) accessing digital security guides and tools; other mentions were: (c) sharing and storing documents and sensitive information; and (d) NAT punching.
 
When asked if they would recommend onion services to anyone, all survey participants answered YES, because of safety.
 
You can find the Onion Guide in our community portal, well as the section on Onion Services, in English, Spanish and Portuguese. Feel free to use it to set up your own .onion site, and let us know how it works for you.
...
@kushal March 25, 2021 - 14:52 • 22 days ago
The correct spelling is Tor

Tor spelling tweet

The correct spelling is Tor, not TOR or any other variations. Please use the correct spelling of the project.

...
@blog March 24, 2021 - 21:05 • 23 days ago
New Release: Tor Browser 10.0.14
New Release: Tor Browser 10.0.14 sysrqb March 24, 2021

Tor Browser 10.0.14 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Desktop Firefox to 78.9.0esr. In addition, Tor Browser 10.0.14 updates NoScript to 11.2.3, and Tor to 0.4.5.7. This version includes important security updates to Firefox for Desktop.

Note: An update for Android Tor Browser is not included in this release.

The full changelog since Desktop Tor Browser 10.0.13 is:

  • Windows + OS X + Linux
    • Update Firefox to 78.9.0esr
    • Update NoScript to 11.2.3
    • Update Tor to 0.4.5.7
    • Bug 40031: Remove survey banner on TB-stable
  • Build System
    • Windows
      • Bug 40249: Bump mingw-w64 and clang for Firefox 78.9
...
@blog March 24, 2021 - 16:58 • 23 days ago
Get a TLS certificate for your onion site
Get a TLS certificate for your onion site isabela March 24, 2021

We are happy to share the news of another important milestone for .onion services! You can now get DV certificates for your v3 onion site using HARICA, a Root CA Operator founded by Academic Network (GUnet), a civil society nonprofit from Greece.

Last year we wrote a blog post about the challenges and opportunities for onion services:

The future of TLS support for onion services is very encouraging. In March of this year, the CA/Browser Forum approved an amendment to the domain validation (DV) TLS certificate baseline requirements which now allows certificate authorities (CAs) to issue certificates containing v3 .onion addresses. This means, in the not-too-distant-future, a CA like Let's Encrypt can issue a certificate for an onion service and Tor Browser will "just work." In addition, for onion services that do not want to rely on certificate authorities, we are exploring alternative designs like Same Origin Onion Certificate for inclusion in Tor Browser.

We are happy that the ‘not-too-distant-future’ was indeed quite close. We hope that more CAs do the same. 

Why would an .onion site need a TLS certificate? This is a great question. Especially because .onion services provide pretty much the same protections offered by an HTTPS connection regarding protecting the data in transit from man in the middle attacks and validating that the user is indeed connecting the server the domain in the browser bar is requesting. Onion services do the same thing, so why would an .onion site need a TLS certificate?

Our Community portal page about onion services give you a list of reasons why a service admin would need a TLS certificate as part of their implementation. Here are some of them:

  • Websites with complex setups and that are serving HTTP and HTTPS content
  • To help the user verify that the .onion address is indeed the site you are hosting (this would be a manual check done by the user looking at the cert registration information)
  • Some services work with protocols, frameworks, and other infrastructure that has HTTPS connection as a requirement
  • In case your web server and your tor process are in different machines

Previously, .onion site administrators who needed a TLS certificate had to either hack other solutions or spend a significant amount of money purchasing an EV certificate. Now with HARICA, acquiring a certificate has become more accessible, but we know that free certificates are ideal and are looking forward to that moment.

We are happy to see people acquiring certificates for their onions. Remember to do it for a v3 onion address since v2 will be deprecated very soon:

2. July 15th, 2021
0.4.6.x: Tor will no longer support v2 and support will be removed from the code base.

If you would like to give it a try, here is a great tutorial by Kushal from the Tor community.
 

...
@anarcat March 23, 2021 - 02:03 • 25 days ago
Major email crash with syncmaildir

TL:DR; lost half my mail (150,000 messages, ~6GB) last night. Cause uncertain, but possibly a combination of a dead CMOS battery, systemd OnCalendar=daily, a (locking?) bug in syncmaildir, and generally, a system too exotic and complicated.

The crash

So I somehow lost half my mail:

anarcat@angela:~(main)$ du -sh Maildir/
7,9G    Maildir/

anarcat@curie:~(main)$ du -sh Maildir
14G     Maildir

anarcat@marcos:~$ du -sh Maildir
8,0G    Maildir

Those are three different machines:

  • angela: my laptop, not always on
  • curie: my workstation, mostly always on
  • marcos: my mail server, always on

Those mails are synchronized using a rather exotic system based on SSH, syncmaildir and rsendmail.

The anomaly started on curie:

-- Reboot --
mar 22 16:13:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:13:00 curie smd-pull[4801]: rm: impossible de supprimer '/home/anarcat/.smd/workarea/Maildir': Le dossier n'est pas vide
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:13:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:14:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:14:00 curie smd-pull[7025]:  4091 ?        00:00:00 smd-push
mar 22 16:14:00 curie smd-pull[7025]: Already running.
mar 22 16:14:00 curie smd-pull[7025]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:14:00 curie smd-pull[7025]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 4091) run(rm /home/anarcat/.smd/lock))
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:14:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

Then it seems like smd-push (from curie) started destroying the universe for some reason:

mar 22 16:20:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:20:00 curie smd-pull[9319]:  4091 ?        00:00:00 smd-push
mar 22 16:20:00 curie smd-pull[9319]: Already running.
mar 22 16:20:00 curie smd-pull[9319]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:20:00 curie smd-pull[9319]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(ru
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:20:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:21:34 curie smd-push[4091]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:21:35 curie smd-push[9374]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:21:35 curie systemd[3199]: smd-push.service: Succeeded.

Notice the del-mails(293920) there: it is actively trying to destroy basically every email in my mail spool.

Then somehow push and pull started both at once:

mar 22 16:21:35 curie systemd[3199]: Started push emails with syncmaildir.
mar 22 16:21:35 curie systemd[3199]: Starting push emails with syncmaildir...
mar 22 16:22:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:22:00 curie smd-pull[10333]:  9455 ?        00:00:00 smd-push
mar 22 16:22:00 curie smd-pull[10333]: Already running.
mar 22 16:22:00 curie smd-pull[10333]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:22:00 curie smd-pull[10333]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: Data transmission failed.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marco
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open requested file.
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:22:00 curie smd-push[9455]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start push emails with syncmaildir.

There it seems push tried to destroy the universe again: del-mails(293920).

Interestingly, the push started again in parallel with the pull, right that minute:

mar 22 16:22:00 curie systemd[3199]: Starting push emails with syncmaildir...

... but didn't complete for a while, here's pull trying to start again:

mar 22 16:24:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:24:00 curie smd-pull[12051]: 10466 ?        00:00:00 smd-push
mar 22 16:24:00 curie smd-pull[12051]: Already running.
mar 22 16:24:00 curie smd-pull[12051]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:24:00 curie smd-pull[12051]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 10466) run(rm /home/anarcat/.smd/lock))
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

... and the long push finally resolving:

mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: No such file or directory
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open requested file.
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie smd-push[10466]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start push emails with syncmaildir.
mar 22 16:24:00 curie systemd[3199]: Starting push emails with syncmaildir...

This pattern repeats until 16:35, when that locking issue silently recovered somehow:

mar 22 16:35:03 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:35:41 curie smd-pull[20788]: default: smd-client@localhost: TAGS: stats::new-mails(5), del-mails(1), bytes-received(21885), xdelta-received(6863398)
mar 22 16:35:42 curie smd-pull[21373]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:35:42 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:35:42 curie systemd[3199]: Started pull emails with syncmaildir.
mar 22 16:36:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:36:36 curie smd-pull[21738]: default: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(214)
mar 22 16:36:37 curie smd-pull[21816]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:36:37 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:36:37 curie systemd[3199]: Started pull emails with syncmaildir.

... notice that huge xdelta-received there, that's 7GB right there. Mysteriously, the curie mail spool survived this, possibly because smd-pull started failing again:

mar 22 16:38:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:38:00 curie smd-pull[23556]: 21887 ?        00:00:00 smd-push
mar 22 16:38:00 curie smd-pull[23556]: Already running.
mar 22 16:38:00 curie smd-pull[23556]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:38:00 curie smd-pull[23556]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 21887) run(rm /home/anarcat/.smd/lock))
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:38:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

That could have been when i got on angela to check my mail, and it was busy doing the nasty removal stuff... although the times don't match. Here is when angela came back online:

anarcat@angela:~(main)$ last
anarcat  :0           :0               Mon Mar 22 19:57   still logged in
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 19:57   still running
anarcat  :0           :0               Mon Mar 22 17:43 - 18:47  (01:03)
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 17:39   still running

Then finally the sync on curie started failing with:

mar 22 16:46:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:46:42 curie smd-pull[27455]: smd-server: ERROR: Client aborted, removing /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.new and /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.mtime.new
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: Failed to copy Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: The destination already exists but its content differs.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: To fix this problem you have two options:
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - rename Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S by hand so that Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   can be copied without replacing it.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   Executing `cd; mv -n "Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "Maildir/.koumbit/cur/1616446002.1.localhost"` should work.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - run smd-push so that your changes to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   are propagated to the other mailbox
mar 22 16:46:42 curie smd-pull[27455]: default: smd-client@localhost: TAGS: error::context(copy-message) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "/home/anarcat/.smd/workarea/Maildir/.koumbit/tmp/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S") run(smd-push default))
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:46:42 curie systemd[3199]: Failed to start pull emails with syncmaildir.

It went on like this until I found the problem. This is, presumably, a good thing because those emails were not being destroyed.

On angela, things looked like this:

-- Reboot --
mar 22 17:39:29 angela systemd[1677]: Started run notmuch new at least once a day.
mar 22 17:39:29 angela systemd[1677]: Started run smd-pull regularly.
mar 22 17:40:46 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open Maildir/.tor/new/1616446842.M285912P26118.marcos,S=8860,W=8996: Maildir/.tor/new/1616446842.M285912P26118.marcos,S=886
0,W=8996: No such file or directory
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open requested file.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: Data transmission failed.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: This problem is transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: server sent ABORT or connection died
mar 22 17:43:18 angela smd-pull[3916]: default: smd-server@smd-server-anarcat: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested
-actions(retry)
mar 22 17:43:18 angela smd-pull[3916]: default: smd-client@localhost: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Failed with result 'exit-code'.
mar 22 17:43:18 angela systemd[1677]: Failed to start pull emails with syncmaildir.
mar 22 17:43:18 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:29 angela smd-pull[4847]: default: smd-client@localhost: TAGS: stats::new-mails(29), del-mails(0), bytes-received(401519), xdelta-received(38914)
mar 22 17:43:29 angela smd-pull[5600]: register: smd-client@localhost: TAGS: stats::new-mails(2), del-mails(0), bytes-received(92150), xdelta-received(471)
mar 22 17:43:29 angela systemd[1677]: smd-pull.service: Succeeded.
mar 22 17:43:29 angela systemd[1677]: Started pull emails with syncmaildir.
mar 22 17:43:29 angela systemd[1677]: Starting push emails with syncmaildir...
mar 22 17:43:32 angela smd-push[5693]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(217)
mar 22 17:43:33 angela smd-push[6575]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(219)
mar 22 17:43:33 angela systemd[1677]: smd-push.service: Succeeded.
mar 22 17:43:33 angela systemd[1677]: Started push emails with syncmaildir.

Notice how long it took to get the first error, in that first failure: it failed after 3 minutes! Presumably that's when it started deleting all that mail. And this is during pull, not push, so the error didn't come from angela.

Affected data

It seems 2GB of mail from my main INBOX was destroyed. Another 2.4GB of spam (kept for training purposes) was also destroyed, along with 700MB of Sent mail. The rest is hard to figure out, because the folders are actually still there, just smaller. So I relied on ncdu to figure out the size changes.

(Note that I don't really archive (or delete much of) my mail since I use notmuch, which is why the INBOX is so large...)

Concretely, according to the notmuch-new.service which still runs periodically on marcos, here are the changes that happened on the server:

mar 22 16:17:12 marcos notmuch[10729]: Added 7 new messages to the database. Removed 57985 messages. Detected 1372 file renames.
mar 22 16:22:43 marcos notmuch[12826]: No new mail. Removed 143842 messages. Detected 6072 file renames.
mar 22 16:27:02 marcos notmuch[13969]: No new mail. Removed 82071 messages. Detected 1783 file renames.
mar 22 16:29:45 marcos notmuch[15079]: Added 22743 new messages to the database. Detected 1 file rename.
mar 22 16:31:48 marcos notmuch[16196]: Added 22779 new messages to the database. Removed 5 messages.
mar 22 16:33:11 marcos notmuch[17192]: Added 3711 new messages to the database.
mar 22 16:40:41 marcos notmuch[19122]: Added 74558 new messages to the database. Detected 1 file rename.
mar 22 16:43:21 marcos notmuch[20325]: Added 9061 new messages to the database. Detected 4 file renames.
mar 22 17:43:08 marcos notmuch[7420]: Added 1793 new messages to the database. Detected 6 file renames.

That is basically the entire mail spool destroyed at first (283 898 messages), and then bits and pieces of it progressively re-added (134 645 messages), somehow, so 149 253 mails were lost, presumably.

Recovery

I disabled the services all over the place:

systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer

(Well, technically, I did that only on angela, as I thought the problem was there. Luckily, curie kept going but it seems like it was harmless.)

I made a backup of the mail spool on curie:

tar cf - Maildir/ | pv -s 14G | gzip -c > Maildir.tgz

Then I crossed my fingers and ran smd-push -v -s, as that was suggested by smd error codes themselves. That thankfully started restoring mail. It failed a few times on weird cases of files being duplicates, but I resolved this by following the instructions. Or mostly: I actually deleted the files instead of moving them, which made smd even unhappier (if there ever was such a thing). I had to recreate some of those files, so, lesson learned: do follow the advice smd gives you, even if it seems useless or strange.

But then smd-push was humming along, uploading tens of thousands of messages, saturating the upload in the office, refilling the mail spool on the server... yaay!... ?

Except... well, of course that didn't quite work: the mail spool in the office eventually started to grow beyond the size of the mail spool on the workstation. That is what smd-push eventually settled on:

default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(151697), del-mails(0), bytes-received(7539147811), xdelta-received(10881198)

It recreated 151 697 emails, adding about 2000 emails to the pool, kind of from nowhere at all. On marcos, before:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,0 GiB [##########] /.notmuch
  717,3 MiB [#         ] /.Archives.2014
  498,2 MiB [#         ] /.feeds.debian-planet
  453,1 MiB [#         ] /.Archives.2012
  414,5 MiB [#         ] /.debian
  408,2 MiB [#         ] /.quoifaire
  389,8 MiB [          ] /.rapports
  356,6 MiB [          ] /.tor
  182,6 MiB [          ] /.koumbit
  179,8 MiB [          ] /tmp
   56,8 MiB [          ] /.nn
   43,0 MiB [          ] /.act-mtl
   32,6 MiB [          ] /.feeds.sysadvent
   31,7 MiB [          ] /.feeds.releases
   31,4 MiB [          ] /.Sent.2005
   26,3 MiB [          ] /.sage
   25,5 MiB [          ] /.freedombox
   24,0 MiB [          ] /.feeds.git-annex
   21,1 MiB [          ] /.Archives.2011
   19,1 MiB [          ] /.Sent.2003
   16,7 MiB [          ] /.bugtraq
   16,2 MiB [          ] /.mlug
 Total disk usage:   8,0 GiB  Apparent size:   7,6 GiB  Items: 184426

After:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,7 GiB [##########] /.notmuch
    2,7 GiB [#####     ] /.junk
    1,9 GiB [###       ] /cur
  717,3 MiB [#         ] /.Archives.2014
  659,3 MiB [#         ] /.Sent
  513,9 MiB [#         ] /.Archives.2012
  498,2 MiB [#         ] /.feeds.debian-planet
  449,6 MiB [          ] /.Archives.2015
  414,5 MiB [          ] /.debian
  408,2 MiB [          ] /.quoifaire
  389,8 MiB [          ] /.rapports
  380,8 MiB [          ] /.Archives.2013
  356,6 MiB [          ] /.tor
  261,1 MiB [          ] /.Archives.2011
  240,9 MiB [          ] /.koumbit
  183,6 MiB [          ] /.Archives.2010
  179,8 MiB [          ] /tmp
  128,4 MiB [          ] /.lists
  106,1 MiB [          ] /.inso-interne
  103,0 MiB [          ] /.github
   75,0 MiB [          ] /.nanog
   69,8 MiB [          ] /.full-disclosure
 Total disk usage:  16,2 GiB  Apparent size:  15,5 GiB  Items: 341143

That is 156 717 files more.

On curie:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------------------------------------
    2,7 GiB [##########] /.junk
    2,3 GiB [########  ] /.notmuch
    1,9 GiB [######    ] /cur
  661,2 MiB [##        ] /.Archives.2014
  655,3 MiB [##        ] /.Sent
  512,0 MiB [#         ] /.Archives.2012
  447,3 MiB [#         ] /.Archives.2015
  438,5 MiB [#         ] /.feeds.debian-planet
  406,5 MiB [#         ] /.quoifaire
  383,6 MiB [#         ] /.debian
  378,6 MiB [#         ] /.Archives.2013
  303,3 MiB [#         ] /.tor
  296,0 MiB [#         ] /.rapports
  237,6 MiB [          ] /.koumbit
  233,2 MiB [          ] /.Archives.2011
  182,1 MiB [          ] /.Archives.2010
  127,0 MiB [          ] /.lists
  104,8 MiB [          ] /.inso-interne
  102,7 MiB [          ] /.register
   89,6 MiB [          ] /.github
   67,1 MiB [          ] /.full-disclosure
   66,5 MiB [          ] /.nanog
 Total disk usage:  13,3 GiB  Apparent size:  12,6 GiB  Items: 342465

Interestingly, there are more files, but less disk usage. It's possible the notmuch database there is more efficient. So maybe there's nothing to worry about.

Last night's marcos backup has:

root@marcos:/home/anarcat# find /mnt/home/anarcat/Maildir | pv -l | wc -l
 341k 0:00:16 [20,4k/s] [                             <=>                                                                                                                                     ]
341040

... 341040 files, which seems about right, considering some mail was delivered during the day. An audit can be performed with hashdeep:

borg mount /media/sdb2/borg/::marcos-auto-2021-03-22 /mnt
hashdeep -c sha256 -r /mnt/home/anarcat/Maildir | pv -l -s 341k > Maildir-backup-manifest.txt

And then compared with:

hashdeep -c sha256 -k Maildir-backup-manifest.txt Maildir/

Some extra files should show up in the Maildir, and very few should actually be missing, because I shouldn't have deleted mail from the previous day the next day, or at least very few. The actual summary hashdeep gave me was:

hashdeep: Audit failed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 339080
Files partially matched: 0
            Files moved: 782
        New files found: 107
  Known files not found: 106

So 106 files added, 107 deleted. Seems good enough for me...

Postfix was stopped at Mar 22 21:12:59 to try and stop external events from confusing things even further. I reviewed the delivery log to see if mail that came in during the problem window disappeared:

grep 'dovecot:.*stored mail into mailbox' /var/log/mail.log |
  tail -20 |
  sed 's/.*msgid=<//;s/>.*//' | 
  while read msgid; do 
    notmuch count --exclude=false id:$msgid |
      grep 0 && echo $msgid missing;
  done

And things looked okay. Now of course if we go further back, we find mail I actually deleted (because I do do that sometimes), so it's hard to use this log as an audit trail. We can only hope that the curie spool is sufficiently coherent to be relied on.

Worst case, we'll have to restore from last night's backup, but that's getting far away now: I get hundreds of mails a day in that mail spool, and reseting back to last night does not seem like a good idea.

A dry run of smd-pull on angela seems to agree that it's missing some files:

default: smd-client@localhost: TAGS: stats::new-mails(154914), del-mails(0), bytes-received(0), xdelta-received(0)

... a number of mails somewhere in between the other two, go figure. A "wet" run of this was started, without deletion (-n), which gave us:

default: smd-client@localhost: TAGS: stats::new-mails(154911), del-mails(0), bytes-received(7658160107), xdelta-received(10837609)

Strange that it sync'd three less emails, but that's still better than nothing, and we have a mail spool on angela again:

anarcat@angela:~(main)$ notmuch new
purging with prefix '.': spam moved (0), ham moved (0), deleted (0), done
Note: Ignoring non-mail file: /home/anarcat/Maildir//.uidvalidity
Processed 1779 total files in 26s (66 files/sec.).
Added 1190 new messages to the database. Removed 3 messages. Detected 593 file renames.
tagging with prefix '.': spam, sent, feeds, koumbit, tor, lists, rapports, folders, done.

Notice how only 1190 messages were re-added, that is because I killed notmuch before it had time to remove all those mails from its database.

Possible causes

I am totally at a loss as to why smd started destroying everything like it did. But a few things come to mind:

  1. I rewired my office on that day.

  2. This meant unplugging curie, the workstation.

  3. It has a bad CMOS battery (known problem), so it jumped around the time continuum a few times, sometimes by years.

  4. The smd services are ran from a systemd unit with OnCalendar=*:0/2. I have heard that it's possible that major time jumps "pile up" execution of jobs, and it seems this happened in this case.

  5. It's possible that locking in smd is not as great as it could be, and that it corrupted its internal data structures on curie, which led it to command a destruction of the remote mail spool.

It's also possible that there was a disk failure on the server, marcos. But since it's running on a (software) RAID-1 array, and no errors have been found (according to dmesg), I don't think that's a plausible hypothesis.

Lessons learned

  1. follow what smd says, even if it seems useless or strange.

  2. trust but verify: just backup everything before you do anything, especially the largest data set.

  3. daily backups are not great for email, unless you're ready to lose a day of email (which I'm not).

  4. hashdeep is great. I keep finding new use cases for it. Last time it was to audit my camera SD card to make sure I didn't forget anything, and now this. it's fast and powerful.

  5. borg is great too. the FUSE mount was especially useful, and it was pretty fast to explore the backup, even through that overhead: checksumming 15GB of mail took about 35 minutes, which gives a respectable 8MB/s, probably bottlenecked by the crap external USB drive I use for backups (!).

  6. I really need to finish my backup system so that I have automated offsite backups, although in this case that would actually have been much slower (certainly not 8MB/s!).

Workarounds and solutions

I setup fake-hwclock on curie, so that the next power failure will not upset my clock that badly.

I am thinking of switching to ZFS or BTRFS for most of my filesystems, so that I can use filesystem snapshots (including remotely!) as a backup strategy. This seems so much more powerful than crawling the filesystem for changes, and allows for truly offsite backups protected from an attacker (hopefully). But it's a long way there.

I'm also thinking of rebuilding my mail setup without smd. It's not the first time something like this happens with smd. It's the first time I am more confident it's the root cause of the problem, however, and it makes me really nervous for the future.

I have used offlineimap in the past and it seems it was finally ported to Python 3 so that could be an option again. isync/mbsync is another option, which I tried before but do not remember why I didn't switch. A complete redesign with something like getmail and/or nncp could also be an option. But alas, I lack the time to go crazy with those experiments.

Somehow, doing like everyone else and just going with Google still doesn't seem to be an option for me. Screw big tech. But I am afraid they will win, eventually.

In any case, I'm just happy I got mail again, strangely.

...
@blog March 21, 2021 - 16:27 • 26 days ago
New Release: Tor Browser 10.5a12 (Android Only)
New Release: Tor Browser 10.5a12 (Android Only) sysrqb March 21, 2021

Tor Browser 10.5a12 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release for Android instead.

This release updates Fenix to 87.0.0-beta.2. Additionally, we update NoScript to 11.2.3 and Tor to 0.4.6.1-alpha.

Note: This is the first Tor Browser version that does not support version 2 onion services. Please see the previously published deprecation timeline.

The full changelog since Tor Browser 10.5a11 is:

  • Android
    • Update Fenix to 87.0.0-beta.2
    • Update NoScript to 11.2.3
    • Update Tor to 0.4.6.1-alpha
    • Translations update
    • Bug 40030: DuckDuckGo redirect to html doesn't work
    • Bug 40043: Rebase android-components patches for Fenix 87 beta 2 builds
    • Bug 40045: Add External App Prompt for Sharing Images
    • Bug 40150: Rebase Fenix patches to Fenix 87 beta 2
    • Bug 40361: Rebase tor-browser patches to 87.0b4
  • Build System
    • Android
      • Update Go to 1.15.10
      • Bug 23631: Use rootless containers
      • Bug 40016: getfpaths is not setting origin_project
      • Bug 40172: Move Gradle compilers out of android-toolchain to own gradle project
      • Bug 40241: Update components for mozilla87-based Fenix
...
@anarcat March 19, 2021 - 13:30 • 28 days ago
Securing my IRC (irssi, screen) session with dtach and systemd

A recent vulnerability in GNU screen caused some people to reconsider their commitment to the venerable terminal multiplexing program. GNU screen is probably used by thousands of old sysadmins around the world to run long-standing processes and, particularly, IRC sessions, which are especially vulnerable to arbitrary garbage coming on ... screen, so to speak.

So this vulnerability matters, and you should definitely pay attention to it. If you haven't switched to tmux yet, now might be a good time to get your fingers trained. But don't switch to it just yet for your IRC session, and read on for a better, more secure solution.

After all, it's not because we found this flaw in screen that it doesn't exist in tmux (or your favorite terminal emulator, for that matter, a much scarier thought).

Hardening my bouncer

Back in March 2019, I had already switched away from screen for IRC, but not to tmux like many did, but to dtach. I figured that I didn't actually need multiplexing to run my long-running IRC session: I just needed to be able to reattach to the terminal. That's what dtach does. No windows, no panes, and, especially, no way to start a new shell, which is exactly the kind of hardening I need.

So I came up with this, to start irssi:

dtach -N /run/$USER/dtach-irssi.socket irssi

To attach:

dtach -a /run/$USER/dtach-irssi.socket

Fairly simple no? Already one attack vector gone: evil attacker can't get a new shell through my terminal multiplexer, yay.

Splitting into another user

But why stop there! Why am I running irssi as my main user anyways! Let's take the lessons from good UNIX security, and run this as a separate user altogether. This requires me to create another user, say foo-irc:

adduser foo-irc

... and run it as a systemd service, because how else are you going to start this thing anyways, cron? I came up with something like this unit file:

[Unit]
Description=IRC screen session
After=network.target

[Service]
Type=simple
Environment="TERM=screen.xterm-256color"
User=%i
RuntimeDirectory=%i
ExecStart=-/usr/bin/dtach -N /run/%i/dtach-irssi.socket irssi
ExecStop=-/bin/sh -c 'echo /quit stopping service... | exec /usr/bin/dtach -p /run/%i/dtach-irssi.socket'
ExecReload=-/bin/sh -c 'echo /restart | exec /usr/bin/dtach -p /run/%i/dtach-irssi.socket'

Notice this is a service template, because of the %i stuff. I don't actually remember how to enable this thing, but let's say you drop this in /etc/systemd/system/irssi@.service, then you run:

systemctl daemon-reload

And then, not sure about this bit, instantiate that template:

systemctl enable irssi@foo-irc.service

And then this should start the irssi session:

systemctl start irssi@foo-irc.service

To access the session:

sudo -u foo-irc dtach -a /run/foo-irc/dtach-irssi.socket

Obviously, you will probably need to migrate your irssi configuration over, otherwise you'll end up with a blank, old-school irssi. Take a moment to savor the view though. Nostalgia. Ah.

Hardening irssi

But this is still not enough. That pesky foo-irc user can still launch arbitrary commands, thanks to irssi /exec (and a generous Perl scripting environment). Let's throw the entire kitchen sink at it and see what sticks. At this point, the unit file becomes too long to just maintain in a blog post (which would be silly, but not unheard of), so just look at this git repository instead.

Password-less remote irssi

The neat thing with this hardening is that I now feel comfortable enough with the setup to just add a password-less SSH key to that (basically throwaway) account: worst that can happen if someone gets a hold of that SSH key is they land in a heavily sandboxed irssi session. So yay, no password to jump on chat. Like a real client or something.

Just make sure to secure the SSH key you'll deploy in authorized_keys with:

restrict,pty,command="dtach -a /run/foo-irc/dtach-irssi.socket" [...]

Obviously, make sure the keys are not writable by the user, by placing it somewhere outside their home, which might require hacking at your server's SSH configuration. Because otherwise a compromised user will be able to change his own authorized_keys, which could be bad.

Configuration management

And at this point, you may have noticed that you shouldn't actually followed my instructions to the letter. Instead, just use this neat little Puppet module which does all of the above, but also include some little wrapper so that mosh still works.

It also includes instructions on how to setup your SSH keys.

Enjoy, and let me know if (or rather, how) I messed up.

Updates

  1. it seems that dtach is not very active upstream: the last release (0.9) is from 2016, and the last commit (at the time of writing) is from 2017

  2. dtach is not necessarily safer than screen or tmux from arbitrary input from the outside, in fact there was a vulnerability on dtach CVE-2012-3368 that led to an attacker accessing stack memory (but maybe not code execution)

  3. after writing the Puppet module and publishing this article, I started to get weird behavior from dtach: i would leave the office at night and then return the next morning to find that I was timed out on servers. from my perspective, irssi noticed only when I re-attached the session:

    09:39:51 -!- Irssi: warning Broken pipe
    09:39:51 -!- Irssi: warning SSL write error: Broken pipe
    09:39:51 -!- Irssi: warning SSL write error: Broken pipe
    09:39:51 -!- Irssi: warning SSL write error: Broken pipe
    09:39:51 -!- Irssi: warning SSL write error: Broken pipe
    09:39:51 [bitlbee] -!- Irssi: Connection lost to localhost
    09:39:51 -!- Irssi: warning SSL write error: Broken pipe
    09:39:51 [gitter] -!- Irssi: Connection lost to irc.gitter.im
    09:39:51 [OFTC] -!- Irssi: Connection lost to irc.oftc.net
    09:42:34 [IMC] -!- Irssi: Connection lost to irc.indymedia.org
    09:42:34 -!- Irssi: Connection lost to irc.hackint.org
    09:42:34 -!- Irssi: Connection lost to chat.freenode.net
    09:42:34 -!- dtach_away: Set away
    

    from the outside I actually timed out a few minutes after I detached, which also makes for a weird asymmetry:

    22:31:00 -!- anarcat [~anarcat@ocean] has quit [Ping timeout: 250 seconds]
    

    that is eleven hours before the error I get.

  4. the mosh wrapper script seems to not work as well as it did before. somehow just running mosh $server hangs with a blank screen instead of instantly rejoining the session. I'm not sure it is related to the timeout problem but I did rewrite the wrapper before publication. this is the old version:

    #!/bin/sh
    
    # inspired by https://serverfault.com/questions/749474/ssh-authorized-keys-command-option-multiple-commands
    
    command="dtach -a /run/anarcat-irc/dtach-irssi.socket"
    
    case "$SSH_ORIGINAL_COMMAND" in
        mosh-server*)
        exec mosh-server -- $command
        ;;
        *)
        exec $command
        ;;
    esac
    

    I'm thinking of trying that one out for a while to see if it's related. The weirdest thing is that mosh "un-hangs" if i reattach with plain ssh, so there's definitely something fishy going on here.

...
@blog March 18, 2021 - 17:53 • 29 days ago
New Alpha Release: Tor 0.4.6.1-alpha
New Alpha Release: Tor 0.4.6.1-alpha nickm March 18, 2021

There's a new alpha release available for download. If you build Tor from source, you can download the source code for 0.4.6.1-alpha from the download page on the website. Packages should be available over the coming weeks, with a new alpha Tor Browser release likely next week.

Remember, this is an alpha release: you should only run this if you'd like to find and report more bugs than usual.

Tor 0.4.6.1-alpha is the first alpha release in the 0.4.6.x series. It improves client circuit performance, adds missing features, and improves some of our DoS handling and statistics reporting. It also includes numerous smaller bugfixes.

Below are the changes since 0.4.5.7. (Note that this release DOES include the fixes for the security bugs already fixed in 0.4.5.7.)

Changes in version 0.4.6.1-alpha - 2021-03-18

  • Major features (control port, onion services):
    • Add controller support for creating version 3 onion services with client authorization. Previously, only v2 onion services could be created with client authorization. Closes ticket 40084. Patch by Neel Chauhan.
  • Major features (directory authorityl):
    • When voting on a relay with a Sybil-like appearance, add the Sybil flag when clearing out the other flags. This lets a relay operator know why their relay hasn't been included in the consensus. Closes ticket 40255. Patch by Neel Chauhan.

 

  • Major features (metrics):
    • Relays now report how overloaded they are in their extrainfo documents. This information is controlled with the OverloadStatistics torrc option, and it will be used to improve decisions about the network's load balancing. Implements proposal 328; closes ticket 40222.
  • Major features (relay, denial of service):
    • Add a new DoS subsystem feature to control the rate of client connections for relays. Closes ticket 40253.
  • Major features (statistics):
    • Relays now publish statistics about the number of v3 onion services and volume of v3 onion service traffic, in the same manner they already do for v2 onions. Closes ticket 23126.
  • Major bugfixes (circuit build timeout):
    • Improve the accuracy of our circuit build timeout calculation for 60%, 70%, and 80% build rates for various guard choices. We now use a maximum likelihood estimator for Pareto parameters of the circuit build time distribution, instead of a "right-censored estimator". This causes clients to ignore circuits that never finish building in their timeout calculations. Previously, clients were counting such unfinished circuits as having the highest possible build time value, when in reality these circuits most likely just contain relays that are offline. We also now wait a bit longer to let circuits complete for measurement purposes, lower the minimum possible effective timeout from 1.5 seconds to 10ms, and increase the resolution of the circuit build time histogram from 50ms bin widths to 10ms bin widths. Additionally, we alter our estimate Xm by taking the maximum of the top 10 most common build time values of the 10ms histogram, and compute Xm as the average of these. Fixes bug 40168; bugfix on 0.2.2.14-alpha.
    • Remove max_time calculation and associated warning from circuit build timeout 'alpha' parameter estimation, as this is no longer needed by our new estimator from 40168. Fixes bug 34088; bugfix on 0.2.2.9-alpha.
  • Major bugfixes (signing key):
    • In the tor-gencert utility, give an informative error message if the passphrase given in `--create-identity-key` is too short. Fixes bug 40189; bugfix on 0.2.0.1-alpha. Patch by Neel Chauhan.
  • Minor features (bridge):
  • Minor features (build system):
    • New "make lsp" command to auto generate the compile_commands.json file used by the ccls server. The "bear" program is needed for this. Closes ticket 40227.
  • Minor features (command-line interface):
    • Add build informations to `tor --version` in order to ease reproducible builds. Closes ticket 32102.
    • When parsing command-line flags that take an optional argument, treat the argument as absent if it would start with a '-' character. Arguments in that form are not intelligible for any of our optional-argument flags. Closes ticket 40223.
    • Allow a relay operator to list the ed25519 keys on the command line by adding the `rsa` and `ed25519` arguments to the --list-fingerprint flag to show the respective RSA and ed25519 relay fingerprint. Closes ticket 33632. Patch by Neel Chauhan.
  • Minor features (control port, stream handling):
    • Add the stream ID to the event line in the ADDRMAP control event. Closes ticket 40249. Patch by Neel Chauhan.
  • Minor features (dormant mode):
    • Add a new 'DormantTimeoutEnabled' option for coarse-grained control over whether the client can become dormant from inactivity. Most people won't need this. Closes ticket 40228.
  • Minor features (logging):
    • Change the DoS subsystem heartbeat line format to be more clear on what has been detected/rejected, and which option is disabled (if any). Closes ticket 40308.
    • In src/core/mainloop/mainloop.c and src/core/mainloop/connection.c, put brackets around IPv6 addresses in log messages. Closes ticket 40232. Patch by Neel Chauhan.
  • Minor features (performance, windows):
    • Use SRWLocks to implement locking on Windows. Replaces the "critical section" locking implementation with the faster SRWLocks, available since Windows Vista. Closes ticket 17927. Patch by Daniel Pinto.
  • Minor features (protocol, proxy support, defense in depth):
    • Close HAProxy connections if they somehow manage to send us data before we start reading. Closes another case of ticket 40017.
  • Minor features (tests, portability):
    • Port the hs_build_address.py test script to work with recent versions of python. Closes ticket 40213. Patch from Samanta Navarro.
  • Minor features (vote document):
    • Add a "stats" line to directory authority votes, to report various statistics that authorities compute about the relays. This will help us diagnose the network better. Closes ticket 40314.
  • Minor bugfixes (build):
    • The configure script now shows whether or not lzma and zstd have been used, not just if the enable flag was passed in. Fixes bug 40236; bugfix on 0.4.3.1-alpha.
  • Minor bugfixes (compatibility):
    • Fix a failure in the test cases when running on the "hppa" architecture, along with a related test that might fail on other architectures in the future. Fixes bug 40274; bugfix on 0.2.5.1-alpha.
  • Minor bugfixes (controller):
    • Fix a "BUG" warning that would appear when a controller chooses the first hop for a circuit, and that circuit completes. Fixes bug 40285; bugfix on 0.3.2.1-alpha.
  • Minor bugfixes (directory authorities, voting):
    • Add a new consensus method (31) to support any future changes that authorities decide to make to the value of bwweightscale or maxunmeasuredbw. Previously, there was a bug that prevented the authorities from parsing these consensus parameters correctly under most circumstances. Fixes bug 19011; bugfix on 0.2.2.10-alpha.
  • Minor bugfixes (ipv6):
    • Allow non-SOCKSPorts to disable IPv4, IPv6, and PreferIPv4. Some rare configurations might break, but in this case you can disable NoIPv4Traffic and NoIPv6Traffic as needed. Fixes bug 33607; bugfix on 0.4.1.1-alpha. Patch by Neel Chauhan.
  • Minor bugfixes (key generation):
    • Do not require a valid torrc when using the `--keygen` argument to generate a signing key. This allows us to generate keys on systems or users which may not run Tor. Fixes bug 40235; bugfix on 0.2.7.2-alpha. Patch by Neel Chauhan.
  • Minor bugfixes (onion services, logging):
    • Downgrade the severity of a few rendezvous circuit-related warnings from warning to info. Fixes bug 40207; bugfix on 0.3.2.1-alpha. Patch by Neel Chauhan.
  • Minor bugfixes (relay):
    • Reduce the compression level for data streaming from HIGH to LOW. This should reduce the CPU and memory burden for directory caches. Fixes bug 40301; bugfix on 0.3.5.1-alpha.
  • Code simplification and refactoring:
    • Remove the orconn_ext_or_id_map structure and related functions. (Nothing outside of unit tests used them.) Closes ticket 33383. Patch by Neel Chauhan.
  • Code simplification and refactoring (metrics, DoS):
    • Move the DoS subsystem into the subsys manager, including its configuration options. Closes ticket 40261.
  • Removed features (relay):
    • Because DirPorts are only used on authorities, relays no longer advertise them. Similarly, self-testing for DirPorts has been disabled, since an unreachable DirPort is no reason for a relay not to advertise itself. (Configuring a DirPort will still work, for now.) Closes ticket 40282.
...
@blog March 17, 2021 - 22:09 • 1 months ago
Sign now: European initiative for a ban on biometric mass surveillance
Sign now: European initiative for a ban on biometric mass surveillance Matthias Marx March 17, 2021

The “Reclaim Your Face” coalition has launched a European Citizens’ Initiative for a ban on biometric mass surveillance. European Digital Rights (EDRi) and more than fifty organizations are calling to sign the petition.

The initiative aims to put biometric mass surveillance on the political agenda in Brussels and stop the recent unregulated proliferation. European citizens can sign the petition at https://reclaimyourface.eu.

One million signatures must be collected in at least seven EU countries within one year. If this goal is to be reached, the European Commission must meet the organizers of the initiative. In addition, after a public hearing in the European Parliament, the Commission must explain its further course of action. In this way, the initiative wants to achieve a moratorium on all experiments with this high-risk technology.

Illegal experiments on humans

The time is pressing: we are already being abused as biometric guinea pigs. Shady companies like Clearview AI and pimeyes are building illegal face databases largely unchallenged. Some want to detect political attitudes based on biometric data. Supposedly smart cameras are designed to read nonconforming and suspicious behavior from our movements to improve our “sense of security.” The European push-back agency Frontex would love to deploy a perverse bouquet of surveillance technologies, including high-resolution cameras with facial, pattern and behavior recognition, with satellites and drones.

Internet-style tracking in the offline world

Online, the surveillance industry keeps track of our every move. The resulting data can easily be stored and prepared for analysis. Luckily, we can use tools like Tor to protect our privacy. In the offline world, ubiquitous surveillance is not yet so easy: many camera systems are not networked and the insane amounts of data prevent mass surveillance ‒ but that’s changing fast. Biometric surveillance technology makes the networked, centralized, and all-encompassing recording of our offline everyday life a reality.

Online, we can keep pace with surveillance and meet new surveillance techniques with new defense measures. But the offline world is different: we cannot participate in everyday life through relays run by volunteers. We don’t all want to have to look the same to avoid being observed. We cannot hide from biometric mass surveillance.

The measured body

Biometric data is particularly sensitive data about unchangeable characteristics of our bodies and behavior. They capture our faces, eyes, veins or voices, the way we walk or our typing on a keyboard. These “fingerprints” can often be used to identify people quickly and unambiguously. Like our fingerprints, we cannot simply change these characteristics. The methods used to process biometric data are often prone to error and discriminatory. On top of that, the way they work is often not comprehensible to third parties. Coupled with an unwarranted faith in technology, this poses immense risks and may even put innocent people in jail.

Support

Individuals can sign the petition for a ban on biometric mass surveillance. Organizations can get involved, too.

...
@blog March 16, 2021 - 13:10 • 1 months ago
New releases (with security fixes): Tor 0.3.5.14, 0.4.4.8, and 0.4.5.7
New releases (with security fixes): Tor 0.3.5.14, 0.4.4.8, and 0.4.5.7 nickm March 16, 2021

We have a new stable release today. If you build Tor from source, you can download the source code for 0.4.5.7 on the download page. Packages should be available within the next several weeks, with a new Tor Browser coming next week.

Also today, Tor 0.3.5.14 (changelog) and Tor 0.4.4.8 (changelog) have also been released; you can find them (and source for older Tor releases) at https://dist.torproject.org.

These releases fix a pair of denial-of-service issues, described below. One of these issues is authority-only.  The other issue affects all Tor instances, and is most damaging on directory authorities and relays.  We recommend that everybody should upgrade to one of these versions once packages are available.

Tor 0.4.5.7 fixes two important denial-of-service bugs in earlier versions of Tor.

One of these vulnerabilities (TROVE-2021-001) would allow an attacker who can send directory data to a Tor instance to force that Tor instance to consume huge amounts of CPU. This is easiest to exploit against authorities, since anybody can upload to them, but directory caches could also exploit this vulnerability against relays or clients when they download. The other vulnerability (TROVE-2021-002) only affects directory authorities, and would allow an attacker to remotely crash the authority with an assertion failure. Patches have already been provided to the authority operators, to help ensure network stability.

We recommend that everybody upgrade to one of the releases that fixes these issues (0.3.5.14, 0.4.4.8, or 0.4.5.7) as they become available to you.

This release also updates our GeoIP data source, and fixes a few smaller bugs in earlier releases.

Changes in version 0.4.5.7 - 2021-03-16

  • Major bugfixes (security, denial of service):
    • Disable the dump_desc() function that we used to dump unparseable information to disk. It was called incorrectly in several places, in a way that could lead to excessive CPU usage. Fixes bug 40286; bugfix on 0.2.2.1-alpha. This bug is also tracked as TROVE-2021- 001 and CVE-2021-28089.
    • Fix a bug in appending detached signatures to a pending consensus document that could be used to crash a directory authority. Fixes bug 40316; bugfix on 0.2.2.6-alpha. Tracked as TROVE-2021-002 and CVE-2021-28090.
  • Minor features (geoip data):
    • We have switched geoip data sources. Previously we shipped IP-to- country mappings from Maxmind's GeoLite2, but in 2019 they changed their licensing terms, so we were unable to update them after that point. We now ship geoip files based on the IPFire Location Database instead. (See https://location.ipfire.org/ for more information). This release updates our geoip files to match the IPFire Location Database as retrieved on 2021/03/12. Closes ticket 40224.

 

  • Minor bugfixes (directory authority):
    • Now that exit relays don't allow exit connections to directory authority DirPorts (to prevent network reentry), disable authorities' reachability self test on the DirPort. Fixes bug 40287; bugfix on 0.4.5.5-rc.
  • Minor bugfixes (documentation):
    • Fix a formatting error in the documentation for VirtualAddrNetworkIPv6. Fixes bug 40256; bugfix on 0.2.9.4-alpha.
  • Minor bugfixes (Linux, relay):
    • Fix a bug in determining total available system memory that would have been triggered if the format of Linux's /proc/meminfo file had ever changed to include "MemTotal:" in the middle of a line. Fixes bug 40315; bugfix on 0.2.5.4-alpha.
  • Minor bugfixes (metrics port):
    • Fix a BUG() warning on the MetricsPort for an internal missing handler. Fixes bug 40295; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (onion service):
    • Remove a harmless BUG() warning when reloading tor configured with onion services. Fixes bug 40334; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (portability):
    • Fix a non-portable usage of "==" with "test" in the configure script. Fixes bug 40298; bugfix on 0.4.5.1-alpha.
  • Minor bugfixes (relay):
    • Remove a spammy log notice falsely claiming that the IPv4/v6 address was missing. Fixes bug 40300; bugfix on 0.4.5.1-alpha.
    • Do not query the address cache early in the boot process when deciding if a relay needs to fetch early directory information from an authority. This bug resulted in a relay falsely believing it didn't have an address and thus triggering an authority fetch at each boot. Related to our fix for 40300.
  • Removed features (mallinfo deprecated):
    • Remove mallinfo() usage entirely. Libc 2.33+ now deprecates it. Closes ticket 40309.
...
@blog March 12, 2021 - 23:05 • 1 months ago
How to contribute to the Tor metrics timeline
How to contribute to the Tor metrics timeline dcf March 12, 2021

The metrics timeline is a database of news and events that may affect Tor Metrics graphs. This post is about how you can contribute to the timeline and help keep it up to date.

A timeline of events helps in interpreting graphs. For example, you may look at a graph and ask, "Why did the number of Tor users in Sri Lanka increase for a week in 2018?"

Directly connecting users from Sri Lanka

Checking the timeline, we find that at that time in Sri Lanka there was a block of Facebook and other services. A likely explanation for the increase of users is that people were using Tor to access the blocked services.

start date end date places protocols description links ?
2018-03-07 2018-03-15 lk Sri Lanka blocks Facebook, Instagram, WhatsApp, and Viber. relay graph bridge graph article

Or, you may look at the graph of relays' advertised bandwidth and ask, "What accounts for the bump in August 2018?"

Total relay bandwidth

The timeline has an explanation for this, too. It was the result of a temporary experiment to test the accuracy of relays' bandwidth reports:

start date end date places protocols description links ?
2019-08-06 2019-08-09 01:00:00 Experiment to test the accuracy of relays' advertised bandwidth estimation. description of experiment post announcing start post announcing end advertised bandwidth graph relay flags graph

The metrics timeline is useful but incomplete—for example, it tends to only include events that make international news. Some past events have a start date but are missing an end date. And some events mark unusual graph features, but do not have an explanation. You can help the Tor Project and people trying to understand use of the Tor network by contributing your knowledge to the metrics timeline.

Data format

The timeline is a text file called README.md in the Tor Project's GitLab. Entries are stored in a table under the ## Timeline heading. The table has seven columns:

|start date|end date|places|protocols|description|links|?  |
|----------|--------|------|---------|-----------|-----|---|
start date
The date, in the format 2020-12-31, when the event began (for long-term events) or occurred (for instantaneous events).
end date
The date when a long-term event ended. This can be blank for instantaneous events, or the special value ongoing for an event that has started but not ended yet.
places
A list of country codes, separated by spaces. This field may be blank for events that are not limited to a specific geographic region.
protocols
Some events only affect one or a few of the many protocols used by Tor. For example, this field might contain obfs4, meek, or snowflake if the event affects only one pluggable transport; bridge if bridges but not relays are affected; or onion if the event involves onion services. Leave the field blank if the event is not specific to a protocol.
description
A freeform text description of the event.
links
A list of links to references or other evidence, in Markdown format: [link label](https://example.com).
?
This field indicates whether the event is adequately explained. An X in the ? field means that there is an unusual feature on the graph, but we do not know what caused it. The field is blank when a likely cause is known.

Here are two examples. The first one had to do with Iran (country code ir) and is considered adequately explained, with links to two reports. The second one affected only the meek pluggable transport in Brazil (country code br). Its cause is unknown, so it only has a link to a graph, and an X in the ? field.

|2019-11-16|2019-11-23|ir||Internet blackout in Iran.|[report](https://ooni.org/post/2019-iran-internet-blackout/) [NTC thread](https://ntc.party/t/a-new-kind-of-censoship-in-iran/237)||
|2017-06-07|2017-12-14|br|meek|Sustained increase of meek users in Brazil, similar to the one that took place between 2016-07-21 and 2017-03-03.|[graph](https://metrics.torproject.org/userstats-bridge-combined.html?start=2016-06-01&end=2018-01-15&country=br)|X|

Here are some specific ways you can help:

  • Find an event with ongoing in the end date field, and see if it really is still ongoing. If it is not, fill in the end date.
  • Think of Tor-related or Internet-related events that have happened in the country you live in, and make sure there is an entry for each of them.
  • Find an event with X in the ? field and search for references or online discussion that may explain it.
  • Search the graphs for unusual features and add entries for them, with an X in the ? field.

How to contribute

To add an entry to the metrics timeline, or edit an existing entry, you will need to create a Tor GitLab account, edit the file, and make a merge request.

Merge requests are the most convenient way for us to accept changes, but if you prefer not to create an account, or if you have trouble with the merge request, we still want you to be able to contribute. Contact support and we will try to help. An alternative to making a merge request is to open a new issue with all the necessary information.

  1. Create a GitLab account at https://gitlab.onionize.space/. Where it says "Please explain why you want to collaborate with the Tor community," you can enter "I want to help with the metrics timeline." Then wait until the new account is approved.

  2. Log in with your account and go to README.md file in the metrics timeline repository.

  3. Click the blue "Edit" button. The first time you do this, you will see a message:

    You're not allowed to edit files in this project directly. Please fork this project, make your changes there, and submit a merge request.

    Click the "Fork" button.

  4. Now you may edit the text file. You may want to change the "No wrap" setting to "Soft wrap" to make it easier to read. Scroll down, below the the ## Timeline heading, and make your change. In the "Commit message" box, enter a one-sentence summary of the change, like "March 2021 outage in ." Click the green "Commit changes" button.

  5. At the "New Merge Request" page, leave the default options and click the green "Submit merge request" button.

Now the merge request is created. You will get a notification by email when it is approved, or if the maintainers have questions.

Step-by-step example

Let's go step by step through the process of finding something to improve, making the change, and starting a merge request. We will look for an ongoing event without an end date and find out when it actually ended.

We will need a Tor GitLab account, so we go to https://gitlab.onionize.space/ and request an account.

Account request form

After the account is created, we can go to https://gitlab.torproject.org/ and log in. We go to README.md in the metrics timeline repository and do a Ctrl+F search for "ongoing". This finds a shutdown event in Ethiopia in mid-2020, which was ongoing at the time the event was added:

start date end date places protocols description links ?
2020-06-30 05:00:00 ongoing et Internet shutdown in Ethiopia. article IODA relay users graph

Looking at the graph of directly connecting users from that time, the effect of the shutdown on June 30 is obvious, but it looks like the number of users recovers a few weeks later:

Directly connecting users from Ethiopia

Then we do a web search for terms like "Ethiopia Internet shutdown 2020". We find an article saying that Internet access was partially restored on July 15 and the shutdown ended on July 23, which is consistent with the Tor Metrics graph. We will make three changes to the entry:

  1. Change the end date field to 2020-07-23.
  2. Note the 2020-07-15 partial restoration of access in the description field.
  3. Add a link to the article to the links field.

Still at the README.md page, we click the blue "Edit" button, then the green "Fork" button.

Screenshot of clicking the

We scroll down to find the entry we are interested in:

Text of the Ethiopia entry, before editing

And then make these changes:

  1. Set end date to 2020-07-23.
  2. Add to description: Restrictions were partially lifted 2020-07-15.
  3. Add to links: [article](https://qz.com/africa/1884387/ethiopia-internet-is-back-on-but-oromo-tensions-remain/).

Text of the Ethiopia entry, after editing

In the "Commit message" box, we enter "June 2020 shutdown in Ethiopia ended in July," then click the green "Commit changes" button.

Screenshot of clicking the

We arrive at a page titled "New Merge Request." We leave all the options unchanged and click the green "Submit merge request" button.

Screenshot of clicking the

Now the merge request is created and you are done. Just wait for the maintainer to merge the change or leave a comment. You will get an email notification when something in the merge request changes.

You can see the merge request that was created and merged for this example at tpo/metrics/timeline!2.

...
@ooni March 9, 2021 - 00:00 • 1 months ago
Myanmar: Data on internet blocks and internet outages following military coup
On 1st February 2021, the military in Myanmar carried out a coup d’etat, seizing power and detaining the country’s State Counsellor (equivalent to a prime minister) and other democratically elected leaders. A few days after the coup, ISPs in Myanmar started blocking access to Facebook services. On 5th February 2021, they started blocking access to Twitter and Instagram as well. On 6th February 2021, access to the internet was shut down entirely for nearly 30 hours. ...
@pastly February 22, 2021 - 21:15 • 2 months ago
Enough about Hacker Factor's '0days'

Last summer Dr. Neal Krawetz AKA "Hacker Factor" made a series of posts on his blog about Tor "0days." This post is a summary of Tor Project's response to one of his posts. Neither this post nor Tor Project's tweet serve as a perfect point-by-point rebuttal of everything he claims in the post, nor all of his "0day" posts. The things that he says that are skipped over here are not automatically valid just because they are skipped. The theme of the responses hold for just about everything he ever says about Tor. As they say, it's easier to spread bullshit than it is to refute it.

Okay wait. Many of the things he says aren't bullshit. He has some valid points. He just can't express those points in a productive manner anymore. His Tor posts are riddled with phrases that instantly put Tor people on the defensive, so it's a masochistic exercise to review them again every time someone asks "hey, what's your thoughts on this HF guy's post from last year?"

The Tor Project tweet is a level headed response (that I helped write); again, this is just a summary of that response, and I'm taking the opportunity to vent while writing it. I will take no questions or comments, nor read emails about this post. I'm freely using inflammatory, emotionally charged language because-- unlike HF--I do not expect, or want, a conversation to come out of this. This is a crass cathartic exercise for me.

The title of the HF blog post this post deals with is "Tor 0day: Burning Bridges." You can find it with your favorite search engine; I'm not going to help drive traffic to his site.

Here we go. The actual content of this "short" post I'm writing for my own reference.

Use of the word 0day

HF knows exactly what he's doing when he uses the term "0day." He's not stupid. He knows what people immediately think when they here that term. He knows 0day sounds scary and gets people excited about a dangerous new discovery. He knows he'd get media attention.

He hides behind "well technically I'm correct because one of the little-used definitions of 0day includes things that aren't fixed and exist in the wild." You're technically and pointlessly correct, HF. And every time someone calls you out on this inflammatory word choice, you get the free rebuttal of "you're not even addressing the real issues! You just don't like my (perfectly valid!!!1!) word choice. You clearly have nothing." No. Fuck you. Use this excuse again if/when you see this, then fuck right off.

Scrollbar width

This is (and was) a publicly documented information leak. There are many ways the user's OS can be leaked (one of them is even on purpose!), and fixing just one of them without fixing many of the others is pointless.

People should report bugs like this so they can be documented and fixed in batches. People should not throw a hissy fit when the bug isn't fixed right away in order to validate their sense of self-importance.

Tor's TLS fingerprint

The way Tor uses TLS between relays and between a client and their relay is (and always has been) fingerprintable. This has been publicly known since 2007. Before HF, it was brought up in 2018 in a much more slanderous and make-a-name-for-yourself-at-the-cost-of-others tone (archive copy).

HF's proposed solution is the wrong one. Tor Project has decided on a better one: bridges with pluggable transports.

Obfs4 is identifiable

Perhaps surprisingly, this is known. It's also an important problem. It's being worked on at a pace slower than HF finds acceptable.

But HF presents variations on known attacks without evidence that they work at a large scale. Two possible issues: too much state to keep track of, or too many false positives such that the adversary is unwilling to deploy it. Luckily for HF, the bar for publishing "science" in a blog post is on the ground. He can say things confidentially and non-experts believe him. Shame on you, HF.

He further shows that he barely looked into this before putting pen to paper (or fingers to keyboard?) by

  • admitting to not knowing of any prior work (in response Tor Project points him to some),

  • citing a paper to support the claim that the Great Firewall can detect obfs4 when the paper say the opposite,

  • citing a blog post about obfs4 bridges being blocked in China, then ignoring that the issue discussed therein is about bridge distribution. Remember HF, in this section you were talking about fingerprintable network activity.

...
@blog March 4, 2021 - 00:08 • 1 months ago
New Release: Tor Browser 10.0.13 (Linux Only)
New Release: Tor Browser 10.0.13 (Linux Only) sysrqb March 03, 2021

Tor Browser 10.0.13 for Linux is now available from the Tor Browser download page and also from our distribution directory.

This version fixes instability on some Linux distributions.

The full changelog since Tor Browser 10.0.12 is:

  • Linux
    • Bug 40328: Fix instability after upgrading to glibc 2.33
...
@kushal March 3, 2021 - 04:34 • 1 months ago
Get a TLS certificate for your onion service

For a long time, I wanted to have a certificate for the onion address of my blog. Digicert was the only CA who was providing those certificates with an Extended Validation. Those are costly and suitable for an organization to get, but not for me personally, especially due to the cost.

TLS certificate working

A few days ago, on IRC, I found out that Harica is providing Domain validation for the onion sites for around €30 per year. I jumped in to get one. At the same time, ahf was also getting his certificate. He helped me with the configuration for nginx.

How to get your own certificate?

  • Make sure you have your site running as Tor v3 onion service
  • Create an account at https://cm.harica.gr/
  • Goto server certificates on the left bar, and make a new request for your domain, provide the onion address as requested in the form.
  • It will give you the option to upload a CSR Certificate Signing Request. You can generate one by openssl req -newkey rsa:4096 -keyout kushaldas.in.onion.key -out csr.csr. For the common name, provide the same onion address.
  • After the click on the website, it will ask you to download a file and put it in your web root inside of .well-known/pki-validation/ directory. Make sure that you can access the file over Tor Browser.
  • When you click the final submission button, the system will take some time to verify the domain. After payment, you should be able to download the certificate with the full chain (the file ending with .p7b). There are 3 options on the webpage, so please remember to download the correct file :)
  • You will have to convert it into PEM format, I used the command ahf showed me: openssl pkcs7 -inform pem -in kushaldas.in.p7b -print_certs -out kushaldas.in.onion.chain.pem -outform pem

Setting up nginx

This part will be the same as any other standard nginx configuration. The following is what I use. Please uncomment the Strict-Transport-Security header line only after you are sure everything is working fine.

server {
	listen unix:/var/run/tor-hs-kushal.sock;

    server_name kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion;
    access_log /var/log/nginx/kushal_onion-access.log;

    location / {
	return 301 https://$host$request_uri;
    }

}

server {
    listen unix:/var/run/tor-hs-kushal-https.sock ssl http2;

    server_name kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion;
    access_log /var/log/nginx/kushal_onion-access.log;

    ssl_certificate /etc/pki/kushaldas.in.onion.chain.pem;
	ssl_certificate_key /etc/pki/kushaldas.in.onion.open.key;

    #add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
	add_header X-Frame-Options DENY;
	add_header X-Content-Type-Options nosniff;
    # Turn on OCSP stapling as recommended at
    # https://community.letsencrypt.org/t/integration-guide/13123
    # requires nginx version >= 1.3.7
    ssl_stapling on;
    ssl_stapling_verify on;

    # modern configuration. tweak to your needs.
    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;

	index index.html;
	root /var/www/kushaldas.in;

	location / {
		try_files $uri $uri/ =404;
	}
}

I also have the following configuration in the /etc/tor/torrc file to use the unix socket files.

HiddenServiceDir /var/lib/tor/hs-kushal/
HiddenServiceVersion 3
HiddenServicePort 80 unix:/var/run/tor-hs-kushal-me.sock
HiddenServicePort 443 unix:/var/run/tor-hs-kushal-https.sock

In case you want to know more about why do you need the certificate for your onion address, the Tor Project has a very nice explanation.

...
@blog February 26, 2021 - 01:10 • 2 months ago
New Release: OnionShare 2.3
New Release: OnionShare 2.3 micah February 25, 2021

This post was originally published on Micah Lee's blog.

After a ridiculously long sixteen months (or roughly ten years in pandemic time) I'm excited to announce that OnionShare 2.3 is out! Download it from onionshare.org.

This version includes loads of new and exciting features which you can read about in much more detail on the brand new OnionShare documentation website, docs.onionshare.org. For now though I'm just going to go over the major ones: tabs, anonymous chat, and better command line support.

Doing all the things at once

In the olden days, OnionShare only did one thing: let you securely and anonymously share files over the Tor network. With time we added new features. You could use it as an anonymous dropbox, and then later to host an onion site.

But what if you wanted to, for example, run your own anonymous dropbox as well as share files with someone? If your OnionShare was busy running a service, you couldn't run a second service without stopping the first service. This is all fixed now thanks to tabs.

onionshare's new layout

Now when you open OnionShare you are presented with a blank tab that lets you choose between sharing files, receiving files, hosting a website, or chatting anonymous. You can have as many tabs open as you want at a time, and you can easily save tabs (that's what the purple thumbtack in the tab bar means) so that if you quit OnionShare and open it again later, these services can start back up with the same OnionShare addresses.

So with OnionShare 2.3 you can host a few websites, have your own personal anonymous dropbox, and securely send files to people whenever you want, all at the same time. Under the hood, the addition of tabs also makes OnionShare connect to the Tor network faster, especially if you're using a bridge.

Secure, anonymous, ephemeral chat rooms that don't log anything

Another major new feature is chat. You start a chat service, it gives you an OnionShare address, and then you send this address to everyone who is invited to the chat room (using an encrypted messaging app like Signal, for example). Then everyone loads this address in a Tor Browser, makes up a name to go by, and can have a completely private conversation.

onionshare chat

If you're already using an encrypted messaging app, what’s the point of an OnionShare chat room? It leaves fewer traces.

If, for example, you send a message to a Signal group, a copy of your message ends up on each device (the devices, and computers if they set up Signal Desktop of each member of the group). Even if disappearing messages is turned on it’s hard to confirm all copies of the messages are actually deleted from all devices, and from any other places (like notifications databases) they may have been saved to. OnionShare chat rooms don’t store any messages anywhere, so the problem is reduced to a minimum.

OnionShare chat rooms can also be useful for people wanting to chat anonymously and securely with someone without needing to create any accounts. For example, a whistleblower can send an OnionShare address to a journalist using a disposable e-mail address, and then wait for the journalist to join the chat room, all without compromising their anonymity.

Because OnionShare relies on Tor onion services, connections between the Tor Browser and OnionShare are all end-to-end encrypted (E2EE). When someone posts a message to an OnionShare chat room, they send it to the server through their E2EE onion connection. The OnionShare server then forwards the message to all other members of the chat room through the other members' E2EE onion connections, using WebSockets. OnionShare doesn’t implement any chat encryption on its own. It relies on the Tor onion service’s encryption instead.

Huge thanks to Saptak Sengupta for developing the anonymous chat feature (doing the bulk of the work in like a single day (!), in the midst of a hacker con in Goa, India last March).

OnionShare from the command line

onionshare command line

OnionShare 2.3 finally de-couples the command line and the graphical versions. You can install onionshare-cli on any platform, including headless Linux servers, using pip:

pip3 install --user onionshare-cli

You also need to have tor installed to use it from your package manager, or Homebrew if you're using macOS.

It's simple to use. For example, here's how you start a chat server:

onionshare command line

I hope you enjoy the new version of OnionShare!

Note February 21, 2021: OnionShare 2.3 for Linux will be available in Flathub after this pull request is reviewed and merged, so hang tight. In the meantime, it's already available in Snapcraft (though it logs analytics), or you can install the .flatpak file directly from onionshare.org/dist/2.3.

Update February 22, 2022: Version 2.3 had a bug where chat was broken :( but we just released version 2.3.1 which fixes it! :).

Update February 23, 2020: The Flatpak package is live! Linux users get it from Flathub.


 

If you'd like to leave a comment, you can do so on Micah Lee's original blog post.

 


 

...
@blog February 24, 2021 - 18:54 • 2 months ago
Learning more about our users
Learning more about our users duncan February 24, 2021

At the Tor Project we practice user-centered design. This means we put our users at the heart of our development process, making a conscious effort to understand the contexts in which people use our tools and paying particular attention to the bumps they encounter along the way.

Many digital product companies rely heavily on data gathered from invasive tracking scripts to better understand their users’ behavior, further fueling the surveillance economy. However that’s not how we do things at Tor – instead, we aim to conduct research that respects the basic principles of privacy and consent.

Prior to 2020, Tor Project contributors and local volunteers traveled to meet our users in their own communities across the world, with a special emphasis on the global south. Now, due to the challenges of the COVID-19 pandemic, much of this research has moved online – and we’re hoping to augment our qualitative interviews and training sessions with larger-scale quantitative research in a way that continues to respect our users’ privacy.

The Tor Project team and friends of Tor in Stockholm, 2019

Tor Browser User Survey

At our Stockholm All Hands meeting in July 2019 we circulated a pilot paper survey among attendees (as in the kind you fill out with a pen and drop into a box) to learn more about their experience using one of our tools in particular: Tor Browser. Although the sample size was intentionally small and heavily skewed towards internal contributors and friends of Tor, it nonetheless gave us a clear indication of the pain points (i.e. frustrations) many of our users encounter while browsing.

Now we’re ready to open up this research to all Tor Browser users in the form of the Tor Browser User Survey – and would like to invite you to give us your feedback:

Launch the Tor Browser User Survey

This survey is also available as a .onion, which can be opened directly in Tor Browser, providing a greater level of security and privacy for survey participants:

Open the Tor Browser survey as a .onion

Snowflake Client User Survey

For those who aren’t familiar, Snowflake is an in-development Pluggable Transport that’s designed to defeat internet censorship, available for testing in Tor Browser Alpha. You can find out more about how to use Snowflake to bypass censorship and how to install the Snowflake extension to help users in censored networks on our Support portal.

We’ve been steadily increasing the number of users testing Snowflake over the past 6 months as we continue to scale up for a future stable release, and alongside the Tor Browser User Survey we’d like to invite Snowflake users to feed back on their browsing experience and connection quality while using it as a client:

Launch the Snowflake Client User Survey

As with the Tor Browser User Survey, the Snowflake Client User Survey is also available as a .onion to help safeguard your privacy while responding:

Open the Snowflake survey as a .onion

Please note: for your personal safety we recommend high-risk individuals do not include any personally identifiable information. As such, certain questions have been marked as optional or include the option to respond “Prefer not to disclose” where appropriate.

Get involved in open user research for Tor

If you’d like to take a step further and volunteer to run an online research session in your own community we host an open call for user research at our monthly User Experience (UX) Team meetings. To get advance notice of the next UX Team meeting subscribe to the UX mailing list and we’ll send you a reminder a few days before it happens. If you want to be extra-prepared, we recommend visiting the Tor Project’s Community portal to get up to speed with our previous work and research methods too.

...
@blog February 24, 2021 - 12:26 • 2 months ago
New Release: Tor Browser 10.5a11
New Release: Tor Browser 10.5a11 gk February 24, 2021

Tor Browser 10.5a11 is now available from the Tor Browser Alpha download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This release updates Firefox to 78.8.0esr for desktop and Firefox for Android to 86.1.0. Additionally, we update Tor to 0.4.5.6 and OpenSSL to 1.1.1j. This release includes important security updates to Firefox for Desktop, and similar important security updates to Firefox for Android.

Note: Tor Browser 10.5 does not support CentOS 6.

The full changelog since Tor Browser 10.5a8 (Linux, MacOS), 10.5a9 (Android), and 10.5a10 (Windows) is:

  • All Platforms
    • Update NoScript to 11.2.2
    • Update OpenSSL to 1.1.1j
    • Update Tor to 0.4.5.6
  • Windows + OS X + Linux
    • Update Firefox to 78.8.0esr
    • Bug 40029: Create survey banner on about:tor for snowflake
  • OS X + Linux
    • Update HTTPS Everywhere to 2021.1.27
    • Bug 40212: Bump version of snowflake and webrtc
  • Android
    • Update Firefox to 86.1.0
    • Bug 40144: Hide Download Manager
    • Bug 40148: Create survey banner on about:tor for Snowflake
    • Bug 40344: Set privacy.window.name.update.enabled=false
  • Build System
    • Linux
...
@blog February 23, 2021 - 20:09 • 2 months ago
New Release: Tor Browser 10.0.12
New Release: Tor Browser 10.0.12 sysrqb February 23, 2021

Tor Browser 10.0.12 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Desktop Firefox to 78.8.0esr and Android Firefox to 86.1.0. In addition, Tor Browser 10.0.12 updates NoScript to 11.2.2, Openssl to 1.1.1j, and Tor to 0.4.5.6. This version includes important security updates to Firefox for Desktop, and similar important security updates to Firefox for Android.

The full changelog since Desktop and Android Tor Browser 10.0.11 is:

  • All Platforms
    • Update NoScript to 11.2.2
    • Update Openssl to 1.1.1j
    • Update Tor to 0.4.5.6
  • Windows + OS X + Linux
    • Update Firefox to 78.8.0esr
    • Bug 40026: Create survey banner on about:tor for desktop
    • Bug 40287: Switch DDG search from POST to GET
  • Android
    • Update Firefox to 86.1.0
    • Bug 40138: Create survey banner on about:tor for Android
    • Bug 40144: Hide Download Manager
    • Bug 40171: Make WebRequest and GeckoWebExecutor First-Party aware
    • Bug 40188: Build and ship snowflake only if it is enabled
    • Bug 40309: Avoid using regional OS locales
    • Bug 40344: Set privacy.window.name.update.enabled=false
  • Build System
    • Android
      • Bug 40214: Update AMO Collection URL
      • Bug 40217: Update components for switch to mozilla86-based Fenix
...