Planet Tor

@kushal July 21, 2021 - 04:10 • 7 days ago
Trouble of zoom and participant name

Last night I was in a panel along with Juan Andrés Guerrero-Saade organized by Aveek Sen, the topic was "Tips on how journalists can avoid getting snooped". You can watch the recording at Youtube.

But this post is not about that. It is about Zoom. Just before logging into the call, I made sure that the name is changed while joining the call, generally my daughter uses the Zoom and her name was mentioned before. I personally have almost zero zoom usage (except 2-3 times in last 1 year). But, after logging into the call, zoom again went back to the older name, and did not allow me to change it during the session. I kept trying during the session without any luck. I don't know why did they do this or why I could not find a way to change my name, but I feel this is really stupid.

...
@anarcat July 21, 2021 - 01:44 • 8 days ago
Hacking my Kobo Clara HD

I just got a new Kobo ebook reader, a Kobo Clara HD. It's pretty similar to the Glo HD I had but which has unfortunately died after 5 years, even after trying to replace the battery.

Quick hardware review

This is a neat little device. It's very similar to the Glo HD, which is a bit disappointing: you'd think they would have improved on the design in the 5+ years since the Glo HD has come out.. It does have an "amber" night light which is nice, but the bezel is still not level with the display, and the device is still kind of on the thick side. A USB-C (instead of micro-USB) port would have been nice too.

But otherwise, it's pretty slick, and just works. And because the hardware design didn't change, I can still hack at it like a madman, which is really why I bought this thing in the first place.

Hopefully it will last longer than 5 years. Ebook readers should really last for decades, not years, but I guess that's too much to expect from our consumerist, suicidal, extinctionist society.

Configuration hacks

Here are the hacks I done on the device. I had done many more hacks on the Kobo Glo HD, but I decided to take a more streamlined, minimalist and, hopefully, easier for new users than the pile of hacks I was doing before (which I expand on at the end of the article).

SD card replacement

I replaced the SD card. The original card shipped with the Clara HD was 8GB which meant all my books actually fitted on the original, but just barely. The new card is 16GB.

Unfortunately, I did this procedure almost at the end of this guide (right before writing the syncthing scripts, below). Next time, that should be the first thing done so the original SD card acts as a pristine copy of the upstream firmware. So even though this seems like an invasive and difficult procedure, I actually do recommend you do it first.

The process is basically to:

  1. crack open the Kobo case (don't worry, it sounds awful but I've done it often)
  2. take the SD card out
  3. copy it over to a new, larger card (say on your computer)
  4. put the larger card in

This guide has all the details.

Registration bypass hack

This guide (from the same author!) has this awesome trick to bypass the annoying registration step. Basically:

  1. pretend you do not have wifi
  2. mount the device
  3. sqlite3 /media/.../KOBOeReader/.kobo/KoboReader.sqlite
  4. INSERT INTO user(UserID,UserKey) VALUES('1','');
  5. unmount the device

More details in the above guide, again.

Install koreader

My e-reader of choise is Koreader. It's just that great. I still don't find the general user interface (ie. the "file browswer") as intuitive as the builtin one, but the book reading just feels better. And anyways it's the easier way to get a shell on the device.

Follow those instructions, particularly the NickelMenu instructions (see also the NickelMenu home page). Yes, you need to install some other thing to start koreader, which doesn't start on its own. NickelMenu is the simplest and better integrated I have found.

You might also want to install some dictionnaries and configure SSH:

  1. mount USB
  2. drop your SSH public key in .../KOBOeReader/.adds/koreader/settings/SSH/authorized_keys
  3. unmount USB
  4. enable SSH in koreader (Gear -> Network -> SSH -> start SSH)

Note that ed25519 keys do not work: try an RSA key. This might be because koreader ships with dropbear (or an older version), but I haven't verified this.

Install syncthing

I use Syncthing to copy all my books into the device now. I was previously using Koreader's OPDS support with Calibre's web interface, but that was clunky and annoying, and I'd constantly have to copy books around. Now the entire collection is synchronized.

As a bonus, I can actually synchronise (and backup!) the koreader metadata, since it's stored next to the files. So in theory, this means I could use koreader from multiple devices and have my reading progress sync'd, but I haven't tested that feature just yet.

I chose Syncthing because it's simple, lightweight, supported on Linux and Android, and statically compiles by default which means it's easy to deploy on the Kobo.

Here is how I installed and started Syncthing at first:

  1. Download the latest version for ARM
  2. extract the archive
  3. copy the syncthing binary into .../KOBOeReader/.adds/
  4. login over SSH (see above on how to enable) with -p 2222 -l root
  5. create the following directory: ~/.config/syncthing/
  6. create the following configuration file, named config.xml:

    <configuration version="18">
        <gui enabled="true" tls="false" debugging="false">
            <address>0.0.0.0:8384</address>
        </gui>
    </configuration>
    
  7. copy a valid ca-certificates.crt file (say from your Linux desktop) into /etc/ssl/certs/ on the Kobo (otherwise syncthing cannot bootstrap discovery servers)
  8. launch syncthing over SSH: /mnt/onboard/.adds/syncthing

You should now be able to connect to the syncthing GUI through your web browser.

Immediately change the GUI admin user and password on the Settings: GUI tab.

Then, figure out how to start it. Here are your options:

  1. on boot (inittab or whatever). downside: power usage.
  2. on wifi (udev hacks). downside: unreliable (see wallabako).
  3. on demand (e.g. nickel menu, koreader terminal shortcuts). downside: kind of clunky in koreader, did not work in nickel menu.
  4. manually, through shell. downside: requires a shell, but then again we already have one through koreader?

What I have done is to write trivial shell scripts (in .../KOBOeReader/scripts) to start syncthing. The first is syncthing-start.sh:

#!/bin/sh

/mnt/onboard/.adds/syncthing serve &

Then syncthing-stop.sh:

#!/bin/sh

/usr/bin/pkill syncthing

This makes those scripts usable from the koreader file browser. Then the folder can be added to the folder shortcuts and a long-hold on the script will allow you to execute it.

Still have to figure out why the Nickel Menu script is not working, but it could simply reuse the above to simplify debugging. This is the script I ended up with, in .../KOBOeReader/.adds/nm/syncthing:

menu_item :main    :Syncthing (toggle)    :cmd_spawn         :exec /mnt/onboard/scripts/syncthing-stop.sh
  chain_success:skip:4
    chain_success                      :cmd_spawn          :exec /mnt/onboard/scripts/syncthing-start.sh
    chain_success                      :dbg_toast          :Started Syncthing server
    chain_failure                      :dbg_toast          :Error starting Syncthing server
    chain_always:skip:-1
  chain_success                        :dbg_toast          :Stopped Syncthing server
menu_item :main    :Syncthing (start)    :cmd_output         :exec /mnt/onboard/scripts/syncthing-start.sh
menu_item :main    :Syncthing (stop)    :cmd_output         :exec /mnt/onboard/scripts/syncthing-stop.sh

It's unclear why this doesn't work: I only get "Error starting Syncthing server" for the toggle, and no output for the (start) action. In either case, syncthing doesn't actually start.

Avoided tasks

This list wouldn't be complete without listing more explicitly the stuff I have done before on the Kobo Glo HD and which I have deliberately decided not to do here because my time is precious:

  • plato install: beautiful project, but koreader is good enough
  • wallabako setup: too much work to maintain, Wallabag articles are too distracting and available on my phone anyways
  • using calibre to transfer books: not working half the time, different file layout than the source, one less Calibre dependency
  • using calibre to generate e-books based on RSS feeds (yes, I did that, and yes, it was pretty bad and almost useless)
  • SSH support: builtin to koreader

Now maybe I'll have time to actually read a book...

...
@blog July 20, 2021 - 19:38 • 8 days ago
New Release: Tor Browser 11.0a2
New Release: Tor Browser 11.0a2 sysrqb July 20, 2021

Tor Browser 11.0a2 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable Windows/macOS/Linux or Android release instead.

This version updates Firefox to version 78.12.0esr on Windows, macOS, and Linux, and Firefox to version 90.1.1 on Android. This version includes important security updates to Firefox on Windows, macOS, and Linux, and Android.

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services later this year. Please see the deprecation F.A.Q. entry regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 11.0a1:

  • All Platforms
    • Update HTTPS Everywhere to 2021.7.13
  • Windows + OS X + Linux
    • Update Firefox to 78.12.0esr
    • Bug 40497: Cannot set multiple pages as home pages in 10.5a17
    • Bug 40506: Saved Logins not available in 10.5
    • Bug 40507: Full update is not downloaded after applying partial update fails
    • Bug 40510: open tabs get redirected to about:torconnect on restart
    • Bug 40524: Update DuckDuckGo onion site URL in search preferences and onboarding
  • Android
    • Update Fenix to 90.1.1
    • Bug 40062: Update DuckDuckGo onion search plugin
    • Bug 40177: Hide Tor icons in settings
  • Build System
    • All Platforms
      • Update Go to 1.16.6
...
@blog July 20, 2021 - 19:31 • 8 days ago
New Release: Tor Browser 10.5.3 (Android)
New Release: Tor Browser 10.5.3 (Android) sysrqb July 20, 2021

Tor Browser 10.5.3 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Firefox to 90.1.1. This version includes important security updates to Firefox.

Warning:
Tor Browser will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 10.5.1:

  • Android
    • Update HTTPS Everywhere to 2021.7.13
    • Update Fenix to 90.1.1
    • Bug 40172: Find the Quit button
    • Bug 40173: Rebase fenix patches to fenix v90.0.0-beta.6
    • Bug 40177: Hide Tor icons in settings
    • Bug 40179: Show Snowflake bridge option on Release
    • Bug 40180: Rebase fenix patches to fenix v90.1.1
  • Build System
    • Android
...
@kushal July 18, 2021 - 06:20 • 10 days ago
A few bytes of curl

curl is most probably the highest used software in the world. I generally use it daily (directly) in the various scripts at the SecureDrop, starting from inside of Dockerfiles, to Ansible roles or in CI. I never read much about various options available other than a few very basic ones. So I decided to look more into the available options. Here are a few interesting points from that reading:

Protocols supported

This is a very long list, I had no clue!!!!

  • dict
  • file
  • ftp
  • ftps
  • gopher
  • gophers
  • http
  • https
  • imap
  • imaps
  • ldap
  • ldaps
  • mqtt
  • pop3
  • pop3s
  • rtsp
  • scp
  • sftp
  • smb
  • smbs
  • smtp
  • smtps
  • telnet
  • tftp

The --version flag will tell you this details and many things more.

$ curl --version
curl 7.76.1 (x86_64-redhat-linux-gnu) libcurl/7.76.1 OpenSSL/1.1.1k-fips zlib/1.2.11 brotli/1.0.9 libidn2/2.3.1 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.9.5/openssl/zlib nghttp2/1.43.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

User agent string

We can use -A to provide any given string as User Agent.

$ curl -A "browser/kd 1.0.0" http://httpbin.org/get
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "browser/kd 1.0.0", 
    "X-Amzn-Trace-Id": "Root=1-60ec8985-05acbf60769f18855db925f1"
  }, 
  "origin": "xxx.xxx.xxx.xxx", 
  "url": "http://httpbin.org/get"
}

Download via sequence

We can specify the range in our command. The first example downloads different index files from my blog.

curl -O "https://kushaldas.in/index-[1-10].html"

A few more examples:

curl -O "http://example.com/[1-100].png"

curl -O "http://example.com/[001-100].png"

curl -O "http://example.com/[0-100:2].png"

curl -O "http://example.com/section[a-z].html"

curl -O "http://example.com/{one,two,three,alpha,beta}.html"

Parallel downloads

You can pass -Z to the above command to download in parallel. Try it out yourself.

Configuration file

We can create a configuration file to hold all the command line parameters and pass it to the curl command via -K flag.

curl -K configs.txt https://kushaldas.in

One flag each line, comments via #.

# Change user agent
--user-agent "ACAB/1.0"

Curl checks for a default configuration file at ~/.curlrc unless called with -q.

To view only the headers

-I to check the headers returned by the server.

$ curl -I https://kushaldas.in
HTTP/2 200 
server: nginx/1.14.2
date: Sun, 18 Jul 2021 06:40:37 GMT
content-type: text/html; charset=utf-8
content-length: 25977
last-modified: Mon, 05 Jul 2021 15:24:32 GMT
etag: "60e32430-6579"
strict-transport-security: max-age=31536000
onion-location: https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion
permissions-policy: interest-cohort=()
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: strict-origin
accept-ranges: bytes

DNS over HTTPS

We can force curl to use a DoH server for the DNS query, pass --doh-url flag to the command along with the server address.

curl -I --doh-url https://adblock.doh.mullvad.net/dns-query https://securedrop.org

...
@blog July 13, 2021 - 21:50 • 15 days ago
Bug Smash Fund, Year 2: Progress Since February 2021
Bug Smash Fund, Year 2: Progress Since February 2021 Al Smith July 13, 2021

Last August, we asked you to help us fundraise during our second annual Bug Smash Fund campaign. This fund is designed to grow a healthy reserve earmarked for maintenance work, finding bugs, and smashing them—all tasks necessary to keep Tor Browser, the Tor network, and the many tools that rely on Tor strong, safe, and running smoothly. In 2020, despite the challenges of COVID-19 and event cancellations, you helped us to raise $106,709!

We want to share an update on some of the work that the second year of the Bug Smash Fund has made possible.

Since 2019, we’ve marked 410 tickets with BugSmashFund. As of today, 373 of those tickets have been closed, and 37 of them are still in progress. This year, we've used the Bug Smash Fund to work on continuous integration tooling, Tor Browser improvements, helping onion services providers defend against DDoS by migrating to v3 onion services, fixing bugs on GetTor, moving forward with Arti, and security fixes. We have also used the Bug Smash Fund to create a new status.torproject.org page, which will act as a landing place for network and service status updates.

Thanks for supporting this work!

Below is a list of some of the tickets we’ve closed since our last update in February 2021.

Website

We fixed several bugs on our main website (https://torproject.org). 

Censorship Analysis

We tracked several censorship events in Iran, Venezuela and other countries.

BridgeDB

We have a sponsored project to improve bridgeDB, but some bugs are not covered and were fixed with the Bug Smash Fund.

GetTor

We are back into maintaining GetTor. It will be integrated into rdsys this year, but for now, we have GetTor running on its own.

Onionoo

We fixed a few bugs in the service that runs Relays Search.

Tor

We fixed a variety of bugs on core tor with the Bug Smash Fund over the last several months.

Tor Browser

We’ve fixed bugs in the Tor Browser building process, as well as closed two tickets related to Tor Browser itself: onion alias url rewrite is broken and document first party isolation for Tor researchers. We are also in the process of changing the repositories branch from “master” to “main,” and some of that work was done during this period.

Thank you to everybody who made a contribution to the Bug Smash Fund. This work is critical in helping us to provide safer tools for millions of people around the world exercising their human rights to privacy and freedom online.

If you’d like to make a contribution to the Bug Smash Fund, you can do so by making a gift at donate.torproject.org: just add “Bug Smash Fund” into the comment field, and we’ll make sure it’s directed to the right place.

...
@blog July 13, 2021 - 19:22 • 15 days ago
New Release: Tor Browser 10.5.2 (Windows, macOS, Linux)
New Release: Tor Browser 10.5.2 (Windows, macOS, Linux) sysrqb July 13, 2021

Tor Browser 10.5.2 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Firefox to 78.12.0esr. This version includes important security updates to Firefox.

Warning:
Tor Browser will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 10.5:

  • Windows + OS X + Linux
    • Update Firefox to 78.12.0esr
    • Bug 40497: Cannot set multiple pages as home pages in 10.5a17
    • Bug 40507: Full update is not downloaded after applying partial update fails
    • Bug 40510: open tabs get redirected to about:torconnect on restart
...
@blog July 13, 2021 - 12:16 • 15 days ago
New Release: Tails 4.20
New Release: Tails 4.20 Tails July 13, 2021

Tor Connection assistant

Tails 4.20 completely changes how to connect to the Tor network from Tails.

After connecting to a local network, a Tor Connection assistant helps you connect to the Tor network.

This new assistant is most useful for users who are at high risk of physical surveillance, under heavy network censorship, or on a poor Internet connection:

  • It protects better the users who need to go unnoticed if using Tor could look suspicious to someone who monitors their Internet connection (parental control, abusive partner, school or work network, etc.).

  • It allows people who need to connect to Tor using bridges to configure them without having to change the default configuration in the Welcome Screen.

  • It helps first-time users understand how to connect to a local Wi-Fi network.

  • It provides feedback while connecting to Tor and helps troubleshoot network problems.

We know that this assistant is still far from being perfect, even if we have been working on this assistant since February. If anything is unclear, confusing, or not working as you would expect, please send your feedback to tails-dev@boum.org (public mailing list).

This first release of the Tor Connection assistant is only a first step. We will add more improvements to it in the coming months to:

  • Save Tor bridges to the Persistent Storage (#5461)

  • Help detect when Wi-Fi is not working (#14534)

  • Detect if you have to sign in to the local network using a captive portal (#5785)

  • Synchronize the clock to make it easier to use Tor bridges in Asia (#15548)

  • Make it easier to learn about new Tor bridges (#18219, #15331)

Changes and updates

  • Update OnionShare from 1.3.2 to 2.2.

    This major update adds a feature to host a website accessible from a Tor onion service.

  • Update KeePassXC from 2.5.4 to 2.6.2.

    This major update comes with a redesign of the interface.

  • Update Tor Browser to 10.5.2.

  • Update Thunderbird to 78.11.0.

  • Update Tor to 0.4.5.9.

  • Update the Linux kernel to 5.10.46. This should improve the support for newer hardware (graphics, Wi-Fi, and so on).

  • Rename MAC address spoofing as MAC address anonymization in the Welcome Screen.

Fixed problems

Automatic upgrades

  • Made the download of upgrades and the handling of errors more robust. (#18162)

  • Display an error message when failing to check for available upgrades. (#18238)

Tails Installer

  • Made the display of the Reinstall button more robust. (#18300)

  • Make the Install and Upgrade unavailable after a USB stick is removed. (#18346)

For more details, read our changelog.

Known issues

  • Automatic upgrades are broken from Tails 4.14 and earlier.

    To upgrade from Tails 4.14 or earlier, you can either:

    • Do a manual upgrade.

    • Fix the automatic upgrade from a terminal. To do so:

      1. Start Tails and set up an administration password.

      2. In a terminal, execute the following command:

        torsocks curl --silent https://tails.boum.org/isrg-root-x1-cross-signed.pem | sudo tee --append /usr/local/etc/ssl/certs/tails.boum.org-CA.pem && systemctl --user restart tails-upgrade-frontend 

        This command is a single command that wraps across several lines. Copy and paste the entire block at once and make sure that it executes as a single command.

      3. Approximately 30 seconds later, you should be prompted to upgrade to the latest version of Tails. If no prompt appears, you might already be running the latest version of Tails.

See the list of long-standing issues.

Get Tails 4.20

To upgrade your Tails USB stick and keep your persistent storage

  • Automatic upgrades are broken from Tails 4.14 and earlier. See the known issue above.

  • Automatic upgrades are available from Tails 4.14 or later to 4.20.

    You can reduce the size of the download of future automatic upgrades by doing a manual upgrade to the latest version.

  • If you cannot do an automatic upgrade or if Tails fails to start after an automatic upgrade, please try to do a manual upgrade.

To install Tails on a new USB stick

Follow our installation instructions:

The Persistent Storage on the USB stick will be lost if you install instead of upgrading.

To download only

If you don't need installation or upgrade instructions, you can download Tails 4.20 directly:

What's coming up?

Tails 4.21 is scheduled for August 10.

Have a look at our roadmap to see where we are heading to.

Support and feedback

For support and feedback, visit the Support section on the Tails website.

...
@blog July 12, 2021 - 19:26 • 16 days ago
New Release: Tor Browser 11.0a1 (Android Only)
New Release: Tor Browser 11.0a1 (Android Only) sysrqb July 12, 2021

Tor Browser 11.0a1 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This version updates Fenix to 90.0.0-beta.6.

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services later this year. Please see the deprecation F.A.Q. entry regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 10.5a17:

  • Android
    • Bug 40172: Find the Quit button
    • Bug 40173: Rebase fenix patches to fenix v90.0.0-beta.6
    • Bug 40179: Show Snowflake bridge option on Release
  • Build System
    • Android
...
@blog July 9, 2021 - 17:00 • 19 days ago
Transparency, Openness, and Our 2020 Financials
Transparency, Openness, and Our 2020 Financials Al Smith July 09, 2021

Every year, as required by U.S. federal law for 501(c)(3) nonprofits, the Tor Project completes a Form 990, and as required by contractual obligations and state regulations, an independent audit of our financial statements. After completing standard audits for 2019-2020,* our federal tax filings and audit are both now available. We upload all of our tax documents and publish a blog post about these documents in order to be transparent.

Transparency for a privacy project is not a contradiction: privacy is about choice, and we choose to be transparent in order to build trust and a stronger community. In this post, we aim to be very clear about where the Tor Project’s money comes from and what we do with it. This is how we operate in all aspects of our work: we show you all of our projects, in source code, and in periodic project and team reports, and in collaborations with researchers who help assess and improve Tor. Transparency also means being clear about our values, promises, and priorities as laid out in our social contract.

This year’s version of the financial transparency post is a bit different than past iterations: we hope that by expanding each subsection of this post and adding more detail, you will have an even better understanding of the Tor Project’s funding, and that you will be able to read the audit and Form 990 documents on your own should you have more questions.

*Reminder: We no longer tie our tax filings and our audits to the calendar year; instead our fiscal years run from July through June. We made this change in 2017 because following the calendar year meant that our fiscal years were ending right in the middle of fundraising season (December), making it harder to plan budgets.

Fiscal Year 2019-2020 Summary

The last few years have been bumpy financially for the Tor Project, but at the end of 2019 and beginning of 2020, we began stabilizing. You helped make this possible by contributing to an incredibly successful 2019 year-end campaign, raising more funds than we ever had before during a year-end campaign up to that point. We were looking towards rebuilding our reserves after having to use them to cover expenses in 2017 and 2018. 

Then, of course, the COVID-19 pandemic changed the world. Individual donations took a drastic downturn, events (where we raise donations at booths) were canceled for the rest of the year, and foundations shifted their funding strategies to stabilize their current grantees. Some foundations even completely halted their giving for the year. 

Despite all of these challenges, we ended the financial year (June 2020) in a stable place.

Revenue & Support

Tor’s total revenue and support in fiscal year 2019-2020 was a bit under $4.9 million. Take a look at page two of the audit:

Screenshot showing the Tor Project Revenue and Support for fiscal year 2019-2020

You can see that “Revenue and Support” is broken into five different categories in the audit documents: most categories are more or less self-explanatory, but let’s talk about in-kind contributions. In-kind contributions are donated services or goods--like translation completed by volunteers, website hosting, donated hardware, and contributed patches. This year, we counted $450,705 in donated services: that’s 2,490 hours of software development, 1,203,719 words translated, and roughly $96K in cloud hosting services. Clearly, Tor would not be possible without you. Thank you!

Because in-kind contributions don’t equal actual money in the bank--but instead equate to value assigned to donated services and goods--think of the $450,705 in-kind contributions as the “Support” portion of the Revenue and Support category. Consider the remaining $4,400,782 in this category as the “Revenue.”

In the 2019-2020 fiscal year, our “Revenue and Support” was less than the previous fiscal year by about $700,000. On the surface this might seem scary, but this reduction in revenue and support also comes with a reduction in expenses and a reversal of unsustainable spending of our reserves. Here’s a comparison, where you can see that in FY 2018-2019, we had to spend a significant amount of our reserves. In FY 2019-2020, we actually increased our assets slightly, despite all of the challenges related to COVID-19.

Financial Year

Income

Expenses

Change in Net Assets

July 2018 - June 2019

$5,606,013

$6,188,913

($582,900)

July 2019 - June 2020

$4,851,487

$4,811,399

$40,088

Government Support

We get a lot of questions (and see a lot of FUD) about how the U.S. government funds the Tor Project, so we want to make this as clear as possible, and show you where to find this information in the future (or for previous years) in these publicly-available documents.

Let’s talk specifically about which parts of the U.S. government support Tor, and what kind of projects they fund. Below, you can see a screenshot from the Tor Project’s FY 2019-2020 Form 990 on page 42, where we’ve listed all of our U.S. government funders. You will find text like this in all of our Form 990s.

Screenshot of government support sources for the Tor Project in our 990

Now, we’ll break down which projects are funded by each entity and link you to places in GitLab where we organize the work associated with this funding.

U.S. State Department Bureau of Democracy, Human Rights, and Labor ($752,154)

  • Project: Empowering Communities in the Global South to Bypass Censorship
    • Description: This ongoing project’s goal is to empower human rights defenders in the Global South by improving censorship event detection and reporting, ensuring users have the best options for their needs to bypass censorship, and informing human rights defenders when censorship is happening and how to bypass it.

National Science Foundation + Georgetown University ($98,727)

Open Technology Fund ($908,744)

Institute of Museum and Library Science + New York University ($101,549)

  • Project: This funding passed through the Tor Project to Library Freedom Project to deliver the Library Freedom Institute. 

Defense Advanced Research Projects Agency + Georgetown University

For even more about how government funding works for the Tor Project, consider reading our previous financial transparency posts, as well as Roger’s thorough comments on these posts.

Other Grants & Contracts

Of the remaining 47% of our revenue, about 26% comes from non-U.S. governments, foundations, other nonprofits, and corporations.

Many of these contributions are in the form of restricted grants, which means we propose a project that is on our roadmap to a funder, they agree that this project is important, and we are funded to complete these projects. Some examples in this category include DIAL Open Source Center’s support of Tor Browser ESR migration work, Zcash Foundation’s support of our project to write the specs for Walking Onions, and RIPE NCC’s support of our work to improve IPv6 support on the Tor network.

Also in this category are unrestricted funds, like support from Media Democracy Fund, Craig Newmark Fund, and FOSS Responders. These unrestricted funds are not tied to a specific project, which means we can use this funding to respond and develop our tools in a more agile way.

Unrestricted funds also include contributions from corporations, and is where you will find membership dues from our members. In the 2019-2020 fiscal year, you’ll see contributions from our first two members, Avast and Mullvad! We haven’t listed every single entity you will see in our Form 990 in this blog post, but we hope you have a better understanding of what you might find in the Form 990. Please explore these documents to learn even more!

Individual Contributions

Individual contributions come in many forms: some people donate $5 to the Tor Project one time, some donate $100 every month, and some make large gifts annually. The common thread is that individual donations are unrestricted funds, and are the most important kind of support we receive. Unrestricted funds allow us to respond to censorship events, develop our tools in a more agile way, and ensure we have reserves to keep Tor strong in case of emergencies (like what happened in 2020, with COVID-19.)

In the 2019-2020 fiscal year, you contributed $890,353 to the Tor Project in the form of one-time gifts and monthly donations. These gifts came in all different forms (including ten different cryptocurrencies, which are then converted to USD), and come together to equal our greatest individual fundraising year in our history. Thank you!

Expenses

OK, we’ve told you how we get funding (and which documents to look at to learn more). Now what do we do with that money? You can find that information on page four of this year’s audit.

Screenshot showing the Tor Project's expenses

We break our expenses into three main categories: 

  • Administration: costs associated with organizational administration, like salary for our Executive Director, office supplies, business insurance costs;
  • Fundraising: costs associated with the fundraising program, like salary for fundraising staff, tools we use for fundraising, bank fees, postal mail supplies, swag; and 
  • Program services: costs associated with making Tor and supporting the people who use it, including application, network, UX, metrics, and community staff salaries; contractor salaries; and IT costs.

In the 2019-2020 fiscal year, 90.4% of our expenses were associated with program services. That means that a very significant portion of our budget goes directly into building Tor and making it better. Next comes fundraising at 6.4% and administration at 3.2%. 

Chart showing the percentage of expenses are associated with program services

According to Charity Navigator, technology nonprofits for which program services make up more than 82.5% of their expenses receive the highest “financial health” score in their ranking system. This means that we’re meeting and significantly exceeding the “industry best” for tech nonprofits. We’re proud to show that our work is both efficient and effective.

Ultimately, like Roger has written in many past versions of this blog post, it’s very important to remember the big picture: Tor's budget is modest considering our small staff and global impact. And it’s also critical to remember that our annual revenue is utterly dwarfed by what our adversaries spend to make the world a more dangerous and less free place.

In closing, we are extremely grateful for all of our donors and supporters. You make this work possible, and we hope this expanded version of our financial transparency post sheds more light on how the Tor Project raises money and how we spend it. Remember, that beyond making a donation, there are other ways to get involved, including volunteering and running a Tor relay!

...
@atagar July 9, 2021 - 00:13 • 20 days ago
Status Report for June 2021

Happy summer everyone! Washington sizzled this month amid a historic heat dome. Willis Carrier, you are a national treasure.

I love AC


Pywikibot Scripts

To parallelize rampup with development I sunk this month into our scripts. I plan to refactor every script and expand their test coverage so this will be an ongoing project. This month included…

add_text.py

movepages.py

listpages.py


Deprecation Policy

In consultation with Xqt and JJMC89 I wrote a deprecation policy for pywikibot. From version 6.4.0 onward pywikibot will use symantic versioning, and assure backward compatibility for minor version bumps.

I also dropped our use of DeprecationWarnings since invisible notifications… aren’t particularly helpful.


Verbosity Reduction

Last, but probably my favorite work this month, I greatly reduced the verbosity of our test and sphinx build output.

My test reduction cropped our unit tests from 536 lines to ~20 (96% reduction). Documentation compilation dropped from 120 lines to 48 (60% reduction).

...
@blog July 8, 2021 - 15:48 • 20 days ago
Announcing Arti, a pure-Rust Tor implementation
Announcing Arti, a pure-Rust Tor implementation nickm July 08, 2021

Greetings!

Today I'm happy to announce a new era in Tor implementation.

Over the past year or so, we've been working on "Arti", a project to rewrite Tor in Rust. Thanks to funding from Zcash Open Major Grants (ZOMG), we can finally put the Arti project up in our priorities list, and devote more time to it.

Below I'll talk about why we're doing this project, what it means for Tor users and operators, where it's going in the future, and how people can help.

A little background

Tor is a set of protocols to provide anonymity, privacy, and censorship resistance on the Internet. Tor is also a program (in C) that provides client-side and server-side implementations of those protocols. We started Tor back around 2002, based on earlier Onion Routing designs from the mid-1990s. In 2006, we incorporated the Tor Project as a nonprofit charity. Since then, Tor has grown to handle millions of users around the world.

Why write Tor in Rust?

Today's Tor is written in the C programming language. Although C is venerable and ubiquitous, it's notoriously error-prone to use, and its lack of high-level features make many programming tasks more complex than they'd be in a more modern language.

For us, these problems mean that programming in C is a slow and painstaking process. Everything we write takes more code than we'd like it to, and we need to double-check even the safest-looking code to make sure it doesn't fall prey to any of C's list of enormous gotchas. This slows us down seriously, and increases the cost of adding new features.

Rust seems like the clearest way out of our bind. It's a high-level language, and significantly more expressive than C. What's more, it's got some really innovative features that let the language enforce certain safety properties at compile-time. To a first approximation, if the code compiles, and it isn't explicitly marked as "unsafe", then large categories of bugs are supposed to be impossible.

That's a huge win for us in programming and debugging time, and a huge win for users in security and reliability. Since 2016, we've been tracking all the security bugs that we've found in Tor, and it turns out that at least half of them were specifically due to mistakes that should be impossible in safe Rust code.

Example: multithreaded crypto

Here's a case where Rust's safety can really help us.

For years now, we've wanted to split Tor's relay cryptography across multiple CPU cores, but we've run into trouble. C's support for thread-safety is quite fragile, and it is very easy to write a program that looks safe to run across multiple threads, but which introduces subtle bugs or security holes. If one thread accesses a piece of state at the same time that another thread is changing it, then your whole program can exhibit some truly confusing and bizarre bugs.

But in Rust, this kind of bug is easy to avoid: the same type system that keeps us from writing memory unsafety prevents us from writing dangerous concurrent access patterns. Because of that, Arti's circuit cryptography has been multicore from day 1, at very little additional programming effort.

Why a full rewrite?

At one point, we had hoped to slowly replace Tor's C code with Rust, one piece at a time. That hasn't worked out for us, however.

Our problem here is that the modules in our existing C code are not terribly well separated from one another: most modules are reachable from most other modules. That makes it hard for us to rewrite our code one module at a time, without first untangling it to be more modular. And untangling the code is risky, for all the same reasons that working in C is typically risky.

With a rewrite, we figured that we can keep our existing C code stable and make only minimal changes to it, while building up a working base of Rust code to serve as a basis for future development.

(And while we're writing a new implementation, we can clean up design issues that have been hard to fix in C. For example, the complicated structure of the C code has made it hard to adopt for embedding into other applications. But with our Arti rewrite, we can take embedding into account from the start, to help support applications down the road.)

What can Arti do today? What features are missing?

First off: Don't use Arti for real privacy yet.

Arti doesn't yet run as a relay at all. It doesn't support Tor's anti-censorship features yet, and it can't connect to onion services yet.

Finally, note that today's Arti is missing several key security features for privacy: you shouldn't use it for browsing if you have actual privacy needs at all.

So what can Arti do? Right now, Arti can successfully bootstrap, run as a SOCKS proxy, and connect over the Tor network. It has an (unstable) API that you can use to embed it in other Rust programs, and give them support for connections over the Tor network.

What are the next steps for Arti?

Thanks to funding from ZOMG, we're going to try bring Arti to a production-quality client implementation over the next year and a half.

In our first phase, we're focusing on the missing security features that we need in order to get Arti as secure as Tor. We estimate we'll be done with this in October of this year. This phase will probably move the most slowly, since we're ramping up our Rust development capacity (and getting better at Rust!), and as we're finishing up some existing commitments.

In our second phase, we'll focus on all the features needed for seamless embedding. We'll add missing features for efficiency and responsiveness, and add APIs for bootstrap reporting and other functionality that applications need to give a good user experience. We estimate we'll finish this around March of 2022.

In our third phase, we'll work to get Arti ready for production client use. We also expect that this will involve a lot of fine-tuning, experimentation, and fixing issues in response to early experimentation and user experience. We expect we'll finish this around September of 2022.

And in our fourth phase, we'll be working on anti-censorship features (including bridges and pluggable transports). We think we can do that in a single additional month, wrapping up around October of 2022.

Beyond that, the plans are unwritten (and so far, unfunded). The next priority will probably be programming support for v3 onion services, and after that, all the other missing client-side features that users need.

And then? We also want support for running a Tor Relay in Rust. That will require a great deal of additional effort, but it should help significantly with the network's performance, reliability, security, and pace of development.

What does this mean for the existing C Tor implementation?

Depending on whether you're an optimist, you might say that that the C Tor code isn't going anywhere soon. Or you might say that its days are numbered.

In order to make the time to work on Arti, we need to devote our resources in that direction. We expect that in the coming years, we will spend more and more time programming in Rust, and less in C. Eventually, once our Rust implementation of Tor is a good replacement for our C implementation, we will stop adding new features to the C implementation, and eventually drop support for it entirely.

But that's far off, for now. At present, we're going to continue supporting and developing our C Tor as a client and relay. We expect that the pace of new features in C will slow, but we will continue fixing issues, shipping bugfixes, and solving important problems in our C code, until Rust is ready to replace it entirely. We will work to keep C Tor users secure, safe, and private, until it is finally ready to be replaced.

How can I try out Arti?

Remember, don't try it yet if you want security or privacy, since it won't give you those.

If you'd like to try Arti with TorBrowser, you can read the instructions in our CONTRIBUTING.md file. They assume that you have cargo installed, and that you know how to build rust programs.

If you'd like to write a Rust program using Arti, have a look at the arti-tor-client crate. For now, you're probably better off using ours instead of the one on crates.io. (Remember, these APIs aren't stable, and are subject to change without warning.)

How can I follow along with development?

We're going to be posting updates in a bunch of different ways, to see what works best for everybody.

First off, we'll be posting regular updates (more technically and more frequently than you'd really want from this blog) on the tor-dev mailing list, and eventually on a dedicated website.

Second, we're going to be recording some of our regular meetings and posting them on the Tor Project's YouTube channel in a dedicated playlist. There we'll be talking about what to work on next, how to organize and schedule, and generally keeping track of pace and priorities.

Third, we hope to be hosting regular public hackathons and programming sessions for interested developers. More information as it develops!

How can I help?

For now, we most need developers and documentation -- especially you're already familiar with Rust or Tor, but even if you're not.

The CONTRIBUTING.md document has a few suggestions of where to start, but it's still pretty new. (Are you any good at writing "How can I help" documents?)

...
@blog July 7, 2021 - 17:27 • 21 days ago
New Release: Tor Browser 10.5.1 (Android Only)
New Release: Tor Browser 10.5.1 (Android Only) sysrqb July 07, 2021

Tor Browser 10.5.1 is now available from the Tor Browser download page and also from our distribution directory.

This version is a bugfix for Android Tor Browser 10.5.

Warning:
Tor Browser will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 10.5:

  • Android
...
@ooni July 7, 2021 - 00:00 • 22 days ago
OONI Partner Training 2021
Image: 1st OONI Partner Training, 28th-30th June 2021 Image: 2nd OONI Partner Training, 5th-7th July 2021 Over the last week, we had the pleasure to host two 3-day OONI Partner Training events for our partners in Africa, Latin America, the Middle East, and Asia. As part of these events, our goal was to share OONI-specific knowledge and skills, and to collect feedback to better serve community needs. ...
@blog July 6, 2021 - 16:56 • 22 days ago
New Release: Tor Browser 10.5
New Release: Tor Browser 10.5 Antonela July 06, 2021

Tor Browser 10.5 is now available from the Tor Browser download page and also from our distribution directory.

This new Tor Browser release is focused on improving the internet access of users connecting through Tor in censored contexts.

What's new?

V2 Onion Services Deprecation

Tor Browser 10.5 V2 onion addresses warning

As we announced last year, v2 onion services will be completely unreachable once Tor Browser moves to Tor 0.4.6.x in October 2021. From now until then, Tor Browser will warn you when visiting a v2 onion site of its upcoming deprecation.

Snowflake is now available as a bridge

Tor Browser 10.5 Snowflake bridge

With Snowflake, censored users can rely on proxies run by volunteers to connect to the internet.

During Q1 this year, the UX team ran a survey on Tor Browser Alpha to better understand Snowflake’s user experience. The survey received 1,795 complete responses, of which 726 participants confirmed they use Snowflake as a pluggable transport. The majority of Snowflake users who completed the survey began using Tor Browser several times a week within the past year. 75% of users had a positive view of Snowflake, although many experienced connection troubles and slow speeds while browsing. These facts and the stable network of volunteers allow us to make it available on this release.

› Read more about Snowflake's stable release

Improving the user experience of connecting to Tor

Tor Browser 10.5 about:connect

Tor Launcher has acted as the options panel for advanced Tor network configurations over the years. It also serves as a control point for users who are in censored networks. The UX and the Anti-Censorship teams joined efforts to improve the connecting flow for Tor Browser users. This release is the first in the upcoming series of helping censored users seamlessly access the open internet by simplifying the connection flow, detecting censorship and providing bridges.

› Read more about the new connection experience

Known Issues

Tor Browser 10.5 comes with a number of known issues:

 

Changelog

The full changelog since Tor Browser 10.0.18

  • All Platforms
    • Update NoScript to 11.2.9
    • Update Tor Launcher to 0.2.30
    • Translations update
    • Bug 25483: Provide Snowflake based on Pion for Windows, macOS, and Linux
    • Bug 33761: Remove unnecessary snowflake dependencies
    • Bug 40064: Bump libevent to 2.1.12
    • Bug 40137: Migrate https-everywhere storage to idb
    • Bug 40261: Bump versions of snowflake and webrtc
    • Bug 40263: Update domain front for Snowflake
    • Bug 40302: Update version of snowflake
    • Bug 40030: DuckDuckGo redirect to html doesn't work
  • Windows + OS X + Linux
    • Bug 27476: Implement about:torconnect captive portal within Tor Browser [tor-browser]
    • Bug 32228: Bookmark TPO support domains in Tor Browser
    • Bug 33803: Add a secondary nightly MAR signing key [tor-browser]
    • Bug 33954: Consider different approach for Bug 2176
    • Bug 34345: "Don't Bootstrap" Startup Mode
    • Bug 40011: Rename tor-browser-brand.ftl to brand.ftl
    • Bug 40012: Fix about:tor not loading some images in 82
    • Bug 40138: Move our primary nightly MAR signing key to tor-browser
    • Bug 40209: Implement Basic Crypto Safety
    • Bug 40428: Correct minor Cryptocurrency warning string typo
    • Bug 40429: Update Onboarding for 10.5
    • Bug 40455: Block or recover background requests after bootstrap
    • Bug 40456: Update the SecureDrop HTTPS-Everywhere update channel
    • Bug 40475: Include clearing CORS preflight cache
    • Bug 40478: Onion alias url rewrite is broken
    • Bug 40484: Bootstrapping page show Quickstart text
    • Bug 40490: BridgeDB bridge captcha selection is broken in alpha
    • Bug 40495: Onion pattern is focusable by click on about:torconnect
    • Bug 40499: Onion Alias doesn't work with TOR_SKIP_LAUNCH
  • Android
    • Bug 30318: Integrate snowflake into mobile Tor Browser
    • Bug 40206: Disable the /etc/hosts parser
  • Linux
    • Bug 40089: Remove CentOS 6 support for Tor Browser 10.5
  • Build System
    • All Platforms
      • Update Go to 1.15.13
      • Bug 23631: Use rootless containers [tor-browser-build]
      • Bug 33693: Change snowflake and meek dummy address [tor-browser]
      • Bug 40016: getfpaths is not setting origin_project
      • Bug 40169: Update apt package cache after calling pre_pkginst, too
      • Bug 40194: Remove osname part in cbindgen filename
    • Windows + OS X + Linux
      • Bug 40081: Build Mozilla code with --enable-rust-simd
      • Bug 40104: Use our TMPDIR when creating our .mar files
      • Bug 40133: Bump Rust version for ESR 78 to 1.43.0
      • Bug 40166: Update apt cache before calling pre_pkginst in container-image config
    • Android
      • Bug 28672: Android reproducible build of Snowflake
      • Bug 40313: Use apt-get to install openjdk-8 .deb files with their dependencies
    • Windows
    • Linux
      • Bug 26238: Move to Debian Jessie for our Linux builds
      • Bug 31729: Support Wayland
      • Bug 40041: Remove CentOS 6 support for 10.5 series
      • Bug 40103: Add i386 pkg-config path for linux-i686
      • Bug 40112: Strip libstdc++ we ship
      • Bug 40118: Add missing libdrm dev package to firefox container
      • Bug 40235: Bump apt for Jessie containers

     

    Give Feedback

    If you find a bug or have a suggestion for how we could improve this release, please let us know. Thanks to all of the teams across Tor, and the many volunteers, who contributed to this release.

Updates:

2021-07-06 1900 UTC: Due to Bug 40324, Android Tor Browser 10.5 should not be installed.

2021-07-06 20:50 UTC: We are aware of issues related to saved passwords becoming unavailable after upgrading to Tor Browser 10.5. Please see tpo/applications/tor-browser#40506 for more information about the investigation.

2021-07-07 18:00 UTC: Tor Browser 10.5.1 is now available for Android as a fix for Bug 40324.

...
@kushal July 5, 2021 - 15:00 • 23 days ago
Reproducible wheel buidling failure on CircleCI container

At SecureDrop project we have Python wheels built for Python 3.7 on Buster in a reproducible way. We use the same wheels inside of the Debian packages. The whole process has checks to verify the sha256sums based on gpg signatures, and at the end we point pip to a local directory to find all the dependencies.

Now, I am working to update the scripts so that we can build wheels for Ubuntu Focal, and in future we can use those wheels in the SecureDrop server side packages. While working on this, in the CI I suddenly noticed that all wheels started failing on the reproducibility test on Buster. I did not make any change other than the final directory path, so was wondering what is going on. Diffoscope helped to find out the difference:

│ -6 files, 34110 bytes uncompressed, 9395 bytes compressed:  72.5%
│ +6 files, 34132 bytes uncompressed, 9404 bytes compressed:  72.4%
├── six-1.11.0.dist-info/METADATA
│ @@ -9,14 +9,15 @@
│  Platform: UNKNOWN
│  Classifier: Programming Language :: Python :: 2
│  Classifier: Programming Language :: Python :: 3
│  Classifier: Intended Audience :: Developers
│  Classifier: License :: OSI Approved :: MIT License
│  Classifier: Topic :: Software Development :: Libraries
│  Classifier: Topic :: Utilities
│ +License-File: LICENSE
│  
│  .. image:: http://img.shields.io/pypi/v/six.svg
│     :target: https://pypi.python.org/pypi/six
│  
│  .. image:: https://travis-ci.org/benjaminp/six.svg?branch=master
│      :target: https://travis-ci.org/benjaminp/six
├── six-1.11.0.dist-info/RECORD
│ @@ -1,6 +1,6 @@
│  six.py,sha256=A08MPb-Gi9FfInI3IW7HimXFmEH2T2IPzHgDvdhZPRA,30888
│  six-1.11.0.dist-info/LICENSE,sha256=Y0eGguhOjJj0xGMImV8fUhpohpduJUIYJ9KivgNYEyg,1066
│ -six-1.11.0.dist-info/METADATA,sha256=vfvF0GW2vCjz99oMyLbw15XSkmo1IxC-G_339_ED4h8,1607
│ +six-1.11.0.dist-info/METADATA,sha256=Beq9GTDD6nYVwLYrN3oOcts0HSPHotfRWQ_Zn8_9a7g,1629
│  six-1.11.0.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│  six-1.11.0.dist-info/top_level.txt,sha256=_iVH_iYEtEXnD8nYGQYpYFUvkUW9sEO1GYbkeKSAais,4
│  six-1.11.0.dist-info/RECORD,,

It seems somehow the packages were picking up metadata about the LICENSE file. After looking into the environment, I found virtualenv is pulling latest setuptools into the virtualenv. Thus breaking the reproducibility. It was a quick fix. Yes, we do pin setuptools and pip in our wheels repository.

Next, the extension based wheels started failing, and I was going totally crazy to find out why there is some ABI change. I still don't know the correct reason, but noticed that the circleci/python:3.7-buster container image (and the upstream Python container) is using different flags to build the extensions than Debian Buster on vm/bare metal. For example, below on the top we have the command line from the container and then inside of the normal Debian Buster.

creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so



creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel

If you want to see the difference in the flags, you can try out the following command:

python3 -m sysconfig

Look for PY_CFLAGS and related flags in the command output. Reproducibility depends on the environment, and we should not expect the latest container image (with the latest Python 3.7.x release) creating the same thing like in Debian Buster, but the failure started recently, we did not notice this a few months. Also, this means in future we should enable reproducibility tests on nightlies in CI.

...
@ooni July 2, 2021 - 00:00 • 27 days ago
Media censorship in Azerbaijan through the lens of network measurement
Key Findings Introduction Methods OONI network measurement Acknowledgement of limitations Background Network landscape and internet penetration Legal environment Internet censorship and media freedom environment Findings Blocked news media websites Blocking of social media amid 2020 Nagorno-Karabakh war Blocking of social media websites Blocking of instant messaging apps WhatsApp Telegram Blocked circumvention tool sites Tor and Psiphon Conclusion Acknowledgements Key Findings As part of our analysis of OONI measurements collected from Azerbaijan between January 2020 to May 2021, we found: ...
@blog June 30, 2021 - 16:14 • 28 days ago
New stable release: Tor 0.4.6.6
New stable release: Tor 0.4.6.6 nickm June 30, 2021

We have a new stable release today. If you build Tor from source, you can download the source code for 0.4.6.6 on the download page.

Tor 0.4.6.6 makes several small fixes on 0.4.6.5, including one that allows Tor to build correctly on older versions of GCC. You should upgrade to this version if you were having trouble building Tor 0.4.6.5; otherwise, there is probably no need.

Changes in version 0.4.6.6 - 2021-06-30

  • Minor bugfixes (compilation):
    • Fix a compilation error when trying to build Tor with a compiler that does not support const variables in static initializers. Fixes bug 40410; bugfix on 0.4.6.5.
    • Suppress a strict-prototype warning when building with some versions of NSS. Fixes bug 40409; bugfix on 0.3.5.1-alpha.
  • Minor bugfixes (testing):
    • Enable the deterministic RNG for unit tests that covers the address set bloomfilter-based API's. Fixes bug 40419; bugfix on 0.3.3.2-alpha.
...
@anarcat June 29, 2021 - 15:10 • 29 days ago
Another syncmaildir crash

So I had another major email crash with my syncmaildir setup. This time I was at least able to confirm the issue, and I still haven't lost mail thanks to backups and sheer luck (again).

The crash

It is not really worth going over the crash in details, it's fairly similar to the last one: something bad happened and smd started destroying everything. The hint is that it takes a long time to do what usually takes seconds. It helps that I now have a second monitor showing logs.

I still lost much more mail than the last time. I used to have "301 723 messages", according to notmuch. But then when I ran smd-pull by hand, it was telling me:

95K emails scanned

Oops. You can see notmuch happily noticing the destroyed files on the server:

jun 28 16:33:40 marcos notmuch[28532]: No new mail. Removed 65498 messages. Detected 1699 file renames.
jun 28 16:36:05 marcos notmuch[29746]: No new mail. Removed 68883 messages. Detected 2488 file renames.
jun 28 16:41:40 marcos notmuch[31972]: No new mail. Removed 118295 messages. Detected 3657 file renames.

The final count ended up being 81 042 messages, according to notmuch. A whopping 220 000 mails deleted.

The interesting bit, this time around, is that I caught smd in the act of running two processes in parallel:

jun 28 16:30:09 curie systemd[2845]: Finished pull emails with syncmaildir. 
jun 28 16:30:09 curie systemd[2845]: Starting push emails with syncmaildir... 
jun 28 16:30:09 curie systemd[2845]: Starting pull emails with syncmaildir... 

So clearly that is the source of the bug.

Recovery

Emergency stop on curie:

notmuch dump > notmuch.dump
systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer

On marcos (the server), guessed the number of messages delivered since the last backup to be 71, just looking at timestamps in the mail log. Made a list:

grep postfix/local /var/log/mail.log | tail -71 > lost-mail

Found postfix queue IDs:

sed 's/.*\]://;s/:.*//' lost-mail > qids

Turn those into message IDs, find those that are missing from the disk (had previously ran notmuch new just to be sure it's up to date):

while read qid ; do 
    grep "$qid: message-id" /var/log/mail.log
done < qids  | sed 's/.*message-id=<//;s/>//' | while read msgid; do
    sudo -u anarcat notmuch count --exclude=false id:$msgid | grep -q 0 && echo $msgid
done

Copy this back on curie as missing-msgids and:

$ wc -l missing-msgids 
48 missing-msgids
$ while read msgid ; do notmuch count --exclude=false id:$msgid | grep -q 0 && echo $msgid ; done < missing-msgids
mailman.189.1624881611.23397.nodes-reseaulibre.ca@reseaulibre.ca
AnwMy7rdSpK-N-vt4AiOag@ismtpd0148p1mdw1.sendgrid.net

only two mails missing! whoohoo!

Copy those back onto marcos as really-missing-msgids, and look at the full mail logs to see what they are:

~anarcat/src/koumbit-scripts/mail/postfix-trace --from-file really-missing-msgids2

I actually remembered deleting those, so no mail lost!

Rebuild the list of msgids that were lost, on marcos:

while read qid ; do grep "$qid: message-id" /var/log/mail.log; done < qids  | sed 's/.*message-id=<//;s/>//'

Copy that on curie as lost-mail-msgids, then copy the files over in a test dir:

while read msgid ; do
    notmuch search --output=files --exclude=false "id:$msgid"
done < lost-mail-msgids | sed 's#/home/anarcat/Maildir/##' | rsync -v  --files-from=- /home/anarcat/Maildir/ shell.anarc.at:restore/Maildir-angela/

If that looks about right, on marcos:

find restore/Maildir-angela/ -type f | wc -l

... should match the number of missing mails, roughly.

Copy if in the real spool:

while read msgid ; do
    notmuch search --output=files --exclude=false "id:$msgid"
done < lost-mail-msgids | sed 's#/home/anarcat/Maildir/##' | rsync -v  --files-from=- /home/anarcat/Maildir/ shell.anarc.at:Maildir/

Then on the server, notmuch new should find the new emails, and we shouldn't have any lost mail anymore:

while read qid ; do grep "$qid: message-id" /var/log/mail.log; done < qids  | sed 's/.*message-id=<//;s/>//' | while read msgid; do sudo -u anarcat notmuch count --exclude=false id:$msgid | grep -q 0 && echo $msgid ; done

Then, crucial moment, try to pull the new mails from the backups on curie:

anarcat@curie:~(main)$ smd-pull  -n  --show-tags -v
Found lockfile of a dead instance. Ignored.
Phase 0: handshake
Phase 1: changes detection
    5K emails scanned
   10K emails scanned
   15K emails scanned
   20K emails scanned
   25K emails scanned
   30K emails scanned
   35K emails scanned
   40K emails scanned
   45K emails scanned
   50K emails scanned
Phase 2: synchronization
Phase 3: agreement
default: smd-client@localhost: TAGS: stats::new-mails(49687), del-mails(0), bytes-received(215752279), xdelta-received(3703852)
"smd-pull  -n  --show-tags -v" took 3 mins 39 secs

This brought me back to the state after the backup plus the mails delivered during the day, which means I had to catchup with all my holiday's read emails (1440 mails!) but thankfully I made a dump of the notmuch database on curie at the start of the procedure, so this actually restored a sane state:

pv notmuch.dump | notmuch restore

Phew!

Workaround

I have filed this as a bug in upstream issue 18. Considering I filed 11 issues and only 3 of those were closed, I'm not holding my breath. I nevertheless filed PR 19 in the hope that this will fix my particular issue, but I'm not even sure this is the right fix...

Fix

At this point, I'm really ready to give up on SMD. It's really, really nice to be able to sync mail over SSH because I don't need to store my IMAP password on disk. But surely there are more reliable syncing mechanisms. I do not remember ever losing that much mail before. At worst, offlineimap would duplicate emails like mad, but never destroy my entire mail spool that way.

As mentioned before, there are other programs that sync mail. I'm looking at:

  • offlineimap3: requires IMAP, used the py2 version in the past, might just still work, first sync painful (IIRC), ways to tunnel over SSH, see comment below
  • isync/mbsync: might be faster, I remember having trouble switching from offlineimap to this, has support for TLS client certs, running over SSH, and generally has good words from multiple Debian and notmuch people
  • getmail: just downloads email, might not be enough
  • nncp: treat the local spool as another mail server, might not be compatible with my "multiple clients" setup
  • doveadm-sync: requires dovecot on both ends, but supports using SSH to sync, will try this next, may have performance problems, see comment below
  • interimap: syncs two IMAP servers, apparently faster than doveadm and offlineimap
  • mail-sync: notify support, happens over any piped transport (e.g. ssh), diff/patch system, requires binary on both ends, mentions UUCP in the manpage, seems awfully complicated to setup, mentions rsmtp which is a nice name for rsendmail
...
@blog June 28, 2021 - 22:33 • 30 days ago
New Release: Tor Browser 10.5a17
New Release: Tor Browser 10.5a17 sysrqb June 28, 2021

Tor Browser 10.5a17 is now available from the Tor Browser download page and also from our distribution directory.

Note: This is an alpha release, an experimental version for users who want to help us test new features. For everyone else, we recommend downloading the latest stable release instead.

This version updates Tor to 0.4.6.5. This version is the last planned version before Tor Browser 10.5 is considered stable. Please report any bugs as soon as possible. In particular, any bugs in the new initial connect screen

Warning:
Tor Browser Alpha does not support version 2 onion services. Tor Browser (Stable) will stop supporting version 2 onion services later this year. Please see the deprecation F.A.Q. entry regarding Tor version 0.4.6. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Tor Browser 10.5a16:

  • All Platforms
    • Update NoScript to 11.2.9
    • Update Tor to 0.4.6.5
    • Update Tor Launcher to 0.2.29
  • Windows + OS X + Linux
    • Bug 34345: "Don't Bootstrap" Startup Mode
    • Bug 40302: Update version of snowflake
    • Bug 40455: Block or recover background requests after bootstrap
    • Bug 40456: Update the SecureDrop HTTPS-Everywhere update channel
    • Bug 40475: Include clearing CORS preflight cache
    • Bug 40478: Onion alias url rewrite is broken
  • Build System
    • All Platforms
      • Update Go to 1.15.13
    • Android
      • Bug 40313: Use apt-get to install openjdk-8 .deb files with their dependencies
...
@blog June 24, 2021 - 13:53 • 1 months ago
Improving the user experience of connecting to Tor in Tor Browser 10.5
Improving the user experience of connecting to Tor in Tor Browser 10.5 Antonela June 24, 2021

During the past few years, the UX team has been working on qualitatively improving the entire Tor Browser user journey: from discovering to finding, downloading, installing, starting, and browsing; we released a seamless and familiar experience for our largest user base. Tor Browser 9.5 was an entire reshaping of the experience for users reaching onion sites, Tor Browser 9.0 and 8.0 shipped an improved experience for core legacy issues, and Tor Browser 8.5 was a rebranded release. However, users continue to report that launching Tor Browser for the first time is an abrasive experience.

Our user research program in the Global South has shed light on how users experiment when confronted with pain points while connecting to Tor. Censored users have also explicitly mentioned how confusing they find the process of copying a bridge address from a webpage and then pasting that address into the browser interface. During our interviews in late 2020, a journalist living in Hong Kong said, “Using bridges is a very manual process.”

Additionally, users have specifically reported that they find Tor Launcher confusing. Research exposed these pain points and has demonstrated how confusion caused by cognitive overload delays the user’s decision-making flow. Known issues with Tor Launcher, like the time gap between Tor Launcher and the main browser window opening after first-time installation, has left some users disappointed. In response, the growing ecosystem of Tor apps have opted to embed the connection experience in their UI instead, like you can see in Brave’s Private Windows with Tor or OnionShare.

Our findings have informed the development of a new iteration of Tor Browser that focuses on improving the flow users follow when connecting to Tor for the first time and on subsequent uses by removing the Tor Launcher UI, making Tor bootstrapping automatic, and relying on a better UI embedded in the main browser screen to provide visual feedback. We are also working on improving the censored user's experience by designing a user flow that will detect censorship and suggest and provide bridges to connect to the Tor network successfully.

 

Quickstart in Tor Browser 10.5

 

The journey to improve the user experience of connecting to the Tor network starts with Tor Browser 10.5, which will be released later this month. This release is the first in an upcoming series of releases helping censored users seamlessly access the open internet.

 

Quickstart preferences in Tor Browser 10.5

 

We would love to hear your thoughts about this release. Feel free to open a ticket anonymously or email the tor-ux mailing list.

 

...
@blog June 21, 2021 - 14:00 • 1 months ago
New Release: Tor Browser 10.0.18
New Release: Tor Browser 10.0.18 sysrqb June 21, 2021

Tor Browser 10.0.18 is now available from the Tor Browser download page and also from our distribution directory.

This version updates Tor to 0.4.5.9, including important security fixes. In addition, on Android, this version updates Firefox to 89.1.1, and NoScript to 11.2.8 This version includes important security updates to Firefox for Android.

Warning:
Tor Browser will stop supporting version 2 onion services later this year. Please see the previously published deprecation timeline. Migrate your services and update your bookmarks to version 3 onion services as soon as possible.

The full changelog since Desktop Tor Browser 10.0.17 and Android Tor Browser 10.0.16:

  • All Platforms
    • Update Tor to 0.4.5.9
  • Android
    • Update Fenix to 89.1.1
    • Update NoScript to 11.2.8
    • Bug 40055: Rebase android-componets patches on 75.0.22 for Fenix 89
    • Bug 40165: Announce v2 onion service deprecation on about:tor
    • Bug 40166: Hide "Normal" tab (again) and Sync tab in TabTray
    • Bug 40167: Hide "Save to Collection" in menu
    • Bug 40169: Rebase fenix patches to fenix v89.1.1
    • Bug 40170: Error building tor-browser-89.1.1-10.5-1
    • Bug 40432: Prevent probing installed applications
    • Bug 40470: Rebase 10.0 patches onto 89.0
  • Build System
    • Android
      • Bug 40290: Update components for mozilla89-based Fenix
...
@pastly February 22, 2021 - 21:15 • 5 months ago
Enough about Hacker Factor's '0days'

Last summer Dr. Neal Krawetz AKA "Hacker Factor" made a series of posts on his blog about Tor "0days." This post is a summary of Tor Project's response to one of his posts. Neither this post nor Tor Project's tweet serve as a perfect point-by-point rebuttal of everything he claims in the post, nor all of his "0day" posts. The things that he says that are skipped over here are not automatically valid just because they are skipped. The theme of the responses hold for just about everything he ever says about Tor. As they say, it's easier to spread bullshit than it is to refute it.

Okay wait. Many of the things he says aren't bullshit. He has some valid points. He just can't express those points in a productive manner anymore. His Tor posts are riddled with phrases that instantly put Tor people on the defensive, so it's a masochistic exercise to review them again every time someone asks "hey, what's your thoughts on this HF guy's post from last year?"

The Tor Project tweet is a level headed response (that I helped write); again, this is just a summary of that response, and I'm taking the opportunity to vent while writing it. I will take no questions or comments, nor read emails about this post. I'm freely using inflammatory, emotionally charged language because-- unlike HF--I do not expect, or want, a conversation to come out of this. This is a crass cathartic exercise for me.

The title of the HF blog post this post deals with is "Tor 0day: Burning Bridges." You can find it with your favorite search engine; I'm not going to help drive traffic to his site.

Here we go. The actual content of this "short" post I'm writing for my own reference. Links to additional anti-HF texts on the Internet are at the end of this post.

Use of the word 0day

HF knows exactly what he's doing when he uses the term "0day." He's not stupid. He knows what people immediately think when they here that term. He knows 0day sounds scary and gets people excited about a dangerous new discovery. He knows he'd get media attention.

He hides behind "well technically I'm correct because one of the little-used definitions of 0day includes things that aren't fixed and exist in the wild." You're technically and pointlessly correct, HF. And every time someone calls you out on this inflammatory word choice, you get the free rebuttal of "you're not even addressing the real issues! You just don't like my (perfectly valid!!!1!) word choice. You clearly have nothing." No. Fuck you. Use this excuse again if/when you see this, then fuck right off.

Scrollbar width

This is (and was) a publicly documented information leak. There are many ways the user's OS can be leaked (one of them is even on purpose!), and fixing just one of them without fixing many of the others is pointless.

People should report bugs like this so they can be documented and fixed in batches. People should not throw a hissy fit when the bug isn't fixed right away in order to validate their sense of self-importance.

Tor's TLS fingerprint

The way Tor uses TLS between relays and between a client and their relay is (and always has been) fingerprintable. This has been publicly known since 2007. Before HF, it was brought up in 2018 in a much more slanderous and make-a-name-for-yourself-at-the-cost-of-others tone (archive copy).

HF's proposed solution is the wrong one. Tor Project has decided on a better one: bridges with pluggable transports.

Obfs4 is identifiable

Perhaps surprisingly, this is known. It's also an important problem. It's being worked on at a pace slower than HF finds acceptable.

But HF presents variations on known attacks without evidence that they work at a large scale. Two possible issues: too much state to keep track of, or too many false positives such that the adversary is unwilling to deploy it. Luckily for HF, the bar for publishing "science" in a blog post is on the ground. He can say things confidentially and non-experts believe him. Shame on you, HF.

He further shows that he barely looked into this before putting pen to paper (or fingers to keyboard?) by

  • admitting to not knowing of any prior work (in response Tor Project points him to some),

  • citing a paper to support the claim that the Great Firewall can detect obfs4 when the paper say the opposite,

  • citing a blog post about obfs4 bridges being blocked in China, then ignoring that the issue discussed therein is about bridge distribution. Remember HF, in this section you were talking about fingerprintable network activity.

Additional links

...
@blog June 14, 2021 - 17:14 • 1 months ago
Internet Freedom, Privacy, & LGBTQIA+ Human Rights
Internet Freedom, Privacy, & LGBTQIA+ Human Rights Al Smith June 14, 2021

Every June, we recognize Pride month because internet freedom and the human rights of LGBTQIA+ people go hand in hand.

LGBTQIA+ people have the right to access information, resources, and community relevant to their identities without being tracked, surveilled, censored, or persecuted—but in many parts of the world, governments block LGBTQIA+ content and punish people who engage with it.

Globally, many LGBTQIA+ people need privacy and censorship circumvention tools like Tor to communicate with their peers, find important resources, or fight for their rights without facing violence.

This is one of the many reasons why we do what we do at the Tor Project, and it’s the reason we monitor the availability of LGBTQ+ sites in OONI tests, so we can better understand which countries are censoring these sites and who needs circumvention technology. (You can help monitor internet censorship, including against LGBTQIA+ sites, by running OONI Probe.) That’s why we travel to countries where governments outlaw or punish being LGBTQIA+ and lead workshops for community organizations about how to protect their privacy online. That’s why we partner with LGBTQIA+ groups to ensure that we’re learning about how they use privacy tech, and what they need from Tor in order to stay safe when using the internet.

Pride and privacy go hand in hand. This June and year round, the Tor Project stands in solidarity with the LGBTQIA+ community. Happy Pride! 

...