The freenode IRC network has been hijacked.

TL;DR: move to libera.chat or OFTC.net, as did countless free software projects including Gentoo, CentOS, KDE, Wikipedia, FOSDEM, and more. Debian and the Tor project were already on OFTC and are not affected by this.

What is freenode and why should I care?

freenode is the largest remaining IRC network. Before this incident, it had close to 80,000 users, which is small in terms of modern internet history -- even small social networks are larger by multiple orders of magnitude -- but is large in IRC history. The IRC network is also extensively used by the free software community, being the default IRC network on many programs, and used by hundreds if not thousands of free software projects.

I have been using freenode since at least 2006.

This matters if you care about IRC, the internet, open protocols, decentralisation, and, to a certain extent, federation as well. It also touches on who has the right on network resources: the people who "own" it (through money) or the people who make it work (through their labor). I am biased towards open protocols, the internet, federation, and worker power, and this might taint this analysis.

What happened?

It's a long story, but basically:

1. back in 2017, the former head of staff sold the freenode.net domain (and its related company) to Andrew Lee, "American entrepreneur, software developer and writer", and, rather weirdly, supposedly "crown prince of Korea" although that part is kind of complex (see House of Yi, Yi Won, and Yi Seok). It should be noted the Korean Empire hasn't existed for over a century at this point (even though its flag, also weirdly, remains)

2. back then, this was only known to the public as this strange PIA and freenode joining forces gimmick. it was suspicious at first, but since the network kept running, no one paid much attention to it. opers of the network were similarly reassured that Lee would have no say in the management of the network

3. this all changed recently when Lee asserted ownership of the freenode.net domain and started meddling in the operations of the network, according to this summary. this part is disputed, but it is corroborated by almost a dozen former staff which collectively resigned from the network in protest, after legal threats, when it was obvious freenode was lost.

4. the departing freenode staff founded a new network, irc.libera.chat, based on the new ircd they were working on with OFTC, solanum

5. meanwhile, bot armies started attacking all IRC networks: both libera and freenode, but also OFTC and unrelated networks like a small one I help operate. those attacks have mostly stopped as of this writing (2021-05-24 17:30UTC)

6. on freenode, however, things are going for the worse: Lee has been accused of taking over a channel, in a grotesque abuse of power; then changing freenode policy to not only justify the abuse, but also remove rules against hateful speech, effectively allowing nazis on the network (update: the change was reverted, but not by Lee)

Update: even though the policy change was reverted, the actual conversations allowed on freenode have already degenerated into toxic garbage. There are also massive channel takeovers (presumably over 700), mostly on channels that were redirecting to libera, but also channels that were still live. Channels that were taken over include #fosdem, #wikipedia, #haskell...

Instead of working on the network, the new "so-called freenode" staff is spending effort writing bots and patches to basically automate taking over channels. I run an IRC network and this bot is obviously not standard "services" stuff... This is just grotesque.

At this point I agree with this HN comment:

We should stop implicitly legitimizing Andrew Lee's power grab by referring to his dominion as "Freenode". Freenode is a quarter-century-old community that has changed its name to libera.chat; the thing being referred to here as "Freenode" is something else that has illegitimately acquired control of Freenode's old servers and user database, causing enormous inconvenience to the real Freenode.

I don't agree with the suggested name there, let's instead call it "so called freenode" as suggested later in the thread.

What now?

I recommend people and organisations move away from freenode as soon as possible. This is a major change: documentation needs to be fixed, and the migration needs to be coordinated. But I do not believe we can trust the new freenode "owners" to operate the network reliably and in good faith.

It's also important to use the current momentum to build a critical mass elsewhere so that people don't end up on freenode again by default and find an even more toxic community than your typical run-of-the-mill free software project (which is already not a high bar to meet).

Update: people are moving to libera in droves. It's now reaching 18,000 users, which is bigger than OFTC and getting close to the largest, traditionnal, IRC networks (EFnet, Undernet, IRCnet are in the 10-20k users range). so-called freenode is still larger, currently clocking 68,000 users, but that's a huge drop from the previous count which was 78,000 before the exodus began. We're even starting to see the effects of the migration on netsplit.de.

Update 2: the isfreenodedeadyet.com site is updated more frequently than netsplit and shows tons more information. It shows 25k online users for libera and 61k for so-called freenode (down from ~78k), and the trend doesn't seem to be stopping for so-called freenode. There's also a list of 400+ channels that have moved out. Keep in mind that such migrations take effect over long periods of time.

Where do I move to?

The first thing you should do is to figure out which tool to use for interactive user support. There are multiple alternatives, of course -- this is the internet after all -- but here is a short list of suggestions, in preferred priority order:

1. irc.libera.chat
2. irc.OFTC.net
3. Matrix.org, which bridges with OFTC and (hopefully soon) with libera as well, modern IRC alternative
4. XMPP/Jabber also still exists, if you're into that kind of stuff, but I don't think the "chat room" story is great there, at least not as good as Matrix

Basically, the decision tree is this:

• if you want to stay on IRC:
  • if you are already on many OFTC channels and few freenode channels: move to OFTC
  • if you are more inclined to support the previous freenode staff: move to libera
  • if you care about matrix users (in the short term): move to OFTC
• if you are ready to leave IRC:
  • if you want the latest and greatest: move to Matrix
  • if you like XML and already use XMPP: move to XMPP

Frankly, at this point, everyone should seriously consider moving to Matrix. The user story is great, the web is a first class user, it supports E2EE (although XMPP as well), and has a lot of momentum behind it. It even bridges with IRC well (which is not the case for XMPP) so if you're worried about problems like this happening again.

(Indeed, I wouldn't be surprised if similar drama happens on OFTC or libera in the future. The history of IRC is full of such epic controversies, takeovers, sabotage, attacks, technical flamewars, and other silly things. I am not sure, but I suspect a federated model like Matrix might be more resilient to conflicts like this one.)

Changing protocols might mean losing a bunch of users however: not everyone is ready to move to Matrix, for example. Graybeards like me have been using irssi for years, if not decades, and would take quite a bit of convincing to move elsewhere.

I have mostly kept my channels on IRC, and moved either to OFTC or libera. In retrospect, I think I might have moved everything to OFTC if I would have thought about it more, because almost all of my channels are there. But I kind of expect a lot of the freenode community to move to libera, so I am keeping a socket open there anyways.

How do I move?

The first thing you should do is to update documentation, websites, and source code to stop pointing at freenode altogether. This is what I did for feed2exec, for example. You need to let people know in the current channel as well, and possibly shutdown the channel on freenode.

Since my channels are either small or empty, I took the radical approach of:

• redirecting the channel to ##unavailable which is historically the way we show channels have moved to another network
• make the channel invite-only (which effectively enforces the redirection)
• kicking everyone out of the channel
• kickban people who rejoin
• set the topic to announce the change

In IRC speak, the following commands should do all this:

/msg ChanServ set #anarcat mlock +if ##unavailable
/msg ChanServ clear #anarcat users moving to irc.libera.chat
/msg ChanServ set #anarcat restricted on
/topic #anarcat this channel has moved to irc.libera.chat

If the channel is not registered, the following might work

/mode #anarcat +if ##unavailable

Then you can leave freenode altogether:

/disconnect Freenode unacceptable hijack, policy changes and takeovers. so long and thanks for all the fish.

Keep in mind that some people have been unable to setup such redirections, because the new freenode staff have taken over their channel, in which case you're out of luck...

Some people have expressed concern about their private data hosted at freenode as well. If you care about this, you can always talk to NickServ and DROP your nick. Be warned, however, that this assumes good faith of the network operators, which, at this point, is kind of futile. I would assume any data you have registered on there (typically: your NickServ password and email address) to be compromised and leaked. If your password is used elsewhere (tsk, tsk), change it everywhere.

Update: there's also another procedure, similar to the above, but with a different approach. Keep in mind that so-called freenode staff are actively hijacking channels for the mere act of mentioning libera in the channel topic, so thread carefully there.

Last words

This is a sad time for IRC in general, and freenode in particular. It's a real shame that the previous freenode staff have been kicked out, and it's especially horrible that if the new policies of the network are basically making the network open to nazis. I wish things would have gone out differently: now we have yet another fork in the IRC history. While it's not the first time freenode changes name (it was called OPN before), now the old freenode is still around and this will bring much confusion to the world, especially since the new freenode staff is still claiming to support FOSS.

I understand there are many sides to this story, and some people were deeply hurt by all this. But for me, it's completely unacceptable to keep pushing your staff so hard that they basically all (except one?) resign in protest. For me, that's leadership failure at the utmost, and a complete disgrace. And of course, I can't in good conscience support or join a network that allows hate speech.

Regardless of the fate of whatever we'll call what's left of freenode, maybe it's time for this old IRC thing to die already. It's still a sad day in internet history, but then again, maybe IRC will never die...

We recently launched the Token Zoo!

The Token Zoo is a knowledge base of various published (and unpublished) anonymous credential schemes and their properties. It might prove useful to you, if - like me - you’ve spent months going through the vast anonymous credential literature and you feel like you need some hand holding!

Hi all! This month I moved on from Tor to begin volunteering with Wikipedia. Covid taught me the importance of face to face contact, and Wikipedia has local Seattle meetups that could scratch an itch Tor didn’t.

Something I desperately look forward to now that I have…

To get my feet wet I invested this month toward standardizing and making minor adjustments to pywikibot…

• Suffix all exceptions with ‘Error’.
• Consistently use string’s format().
• Remove usage of the deprecated distutils module.
• Drop unnecessary else/elif used after raise.
• Context manager to suppress and assert a bot’s logs.
• Renamed config2 and is_IP.
• Deprecated DotReadableDict, SelfCall, SizedKeyCollection, empty_iterator, frozenmap, textlib.unescape, and regex classes.
• Fixed minor test failures (1, 2, 3).

I was recently asked how I setup my websites to:

1. Redirect HTTP to HTTPS when not accessed via an onion service.
2. Serve the website over HTTPS when not accessed via an onion service.
3. Serve the website over HTTP when accessed via an onion service.

I will further explain:

• How the .onion available button is obtained in my setup.
• How to add an onion Alt-Svc that works.

I have a very simple setup. I have a tor daemon running on the same machine as nginx. As most of my websites are static, nginx serves their files directly in most cases. There is no software between tor and nginx; if there is for you, that drastically changes things and this post may be of little use to you. If you have extra software "behind" nginx (e.g. a python app generating a dynamic website), most likely this post will still be useful to you. For example, instead of telling nginx this like I do:

location / {
        try_files $uri $uri/ =404;
}

You might be telling nginx this:

location / {
        include proxy_params;
        proxy_pass http://unix:/path/to/some/app.sock;
}

I use Certbot (Let's Encrypt) for my CA, and it automatically generates some of the nginx config you will see below.

All of the nginx config blocks are in one file, /etc/nginx/sites-available/flashflow.pastly.xyz. As is standard with nginx on Debian, there's a symlink to that file in /etc/nginx/sites-enabled/ and /etc/nginx/nginx.conf was already set to load files in /etc/nginx/sites-enabled/.

This post uses flashflow.pastly.xyz and its onion address as an example. Whenever you see flashflow.pastly.xyz or its onion address, mentally replace the domains with your own.

Redirect HTTP to HTTPS when not accessed via an onion service.

This is entirely handled by nginx and uses a server {} block automatically generated by Certbot. It is this:

server {
    if ($host = flashflow.pastly.xyz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
    listen 80;
    listen [::]:80;
    server_name flashflow.pastly.xyz;
    return 404; # managed by Certbot
}

All this block does is redirect to HTTPS. It is used when the user is visiting flashflow.pastly.xyz on port 80, as indicated by the server_name and listen lines.

Serve the website over HTTPS when not accessed via an onion service.

This is entirely handled by nginx. Again as the server_name and listen lines indicate, this block is used when the user is visiting flashflow.pastly.xyz on port 443 (using TLS). This is overwhelmingly automatically generated by Certbot too.

I slightly simplified this block as presented here. We will edit this block later in this post to add Onion-Location and Alt-Svc headers.

server {
    server_name flashflow.pastly.xyz;
    root /var/www/flashflow.pastly.xyz;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/flashflow.pastly.xyz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/flashflow.pastly.xyz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Serve the website over HTTP when accessed via an onion service.

This is the nginx config block. It is a simplified version of the previous one, as it is also actually serving the website, but with plain HTTP and when the user is visiting the onion service, not flashflow.pastly.xyz.

server {
    listen 80;
    listen [::]:80;
    server_name jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion;
    root /var/www/flashflow.pastly.xyz;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

These are the relevant lines from the tor's torrc. We will edit this block later in this post to add Alt-Svc support.

HiddenServiceDir /var/lib/tor/flashflow.pastly.xyz_service
HiddenServicePort 80

In this post I've shared two server {} blocks that tell nginx to listen on port 80. Nginx knows to use this block for onion service connections because the server_name (the hostname that the user's browser is telling nginx it wants to visit) is the onion service. Nginx uses the other server {} block with port 80 when the user's browser tells nginx that it wants to visit flashflow.pastly.xyz.

After adding those lines to the torrc, I reloaded tor (restart not required). Then I could learn what the onion address is:

$ cat /var/lib/tor/flashflow.pastly.xyz_service/hostname 
jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion

And from there knew what to put on the server_name line.

Whenever I edited nginx's config, I reloaded nginx when done (systemctl reload nginx) and verified it didn't say there was an error.

Whenever I edited tor's config, I reloaded tor when done (systemctl reload tor@default) and verified by checking tor's logs that there was no error (journalctl -eu tor@default) and that tor is still running (systemctl status tor@default).

How the .onion available button is obtained in my setup.

Verify that the preceeding steps are working. Verify that:

1. Visiting http://flashflow.pastly.xyz redirects to https://flashflow.pastly.xyz to serve the website.
2. Visiting http://jsd33qlp6[...]d.onion serves the website.

This button advertises the fact that the website is also available at an onion service, which improves users' security and may even improve their performance. Further, if they've configured Tor Browser to do so, Tor Browser can automatically redirect to the onion service instead of presenting a button for the user to maybe click.

Find the 2nd server {} block you added, the one that listens on port 443. We are now going to add a single line to it that instructs nginx to add an HTTP header in its responses.

server {
    server_name flashflow.pastly.xyz;
    [... lines omitted ...]
    location / {
        try_files $uri $uri/ =404;
        # ADD THE BELOW LINE
        add_header Onion-Location http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion$request_uri;
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    [... lines omitted ...]
}

Reload nginx and verify it didn't say there was an error.

Visiting https://flashflow.pastly.xyz should now result in a purple .onion available button appearing in the URL bar when the page is done loading. Clicking it will take the user from https://flashflow.pastly.xyz/foo/bar to http://jsd33qlp6[...]d.onion/foo/bar.

How to add an onion Alt-Svc that works.

Verify that the preceeding steps are working. Verify that:

1. Visiting http://flashflow.pastly.xyz redirects to https://flashflow.pastly.xyz to serve the website.
2. Visiting http://jsd33qlp6[...]d.onion serves the website.
3. (Optional) visiting https://flashflow.pastly.xyz results in a purple .onion available button in the URL bar.

This is another HTTP header that tells the browser there is another way to fetch the given resource that it should consider using in the future instead. The Alt-Svc header is used in contexts entirely outside of Tor, but it can also be used to tell Tor Browser to consider secretly fetching content from this host from an onion service in the future.

Common gotcha: The onion service must also support HTTPS. The onion service does not need a TLS certificate that is valid for the onion address: it should just use the same certificate as the regular web service, even though it is invalid for the onion service. The browser verifies that the certificate it gets from jsd33qlp6[...]d.onion is valid for flashflow.pastly.xyz when using the .onion as an Alt-Svc for the .xyz.

Add to the torrc the following line:

HiddenServiceDir /var/lib/tor/flashflow.pastly.xyz_service
HiddenServicePort 80
# ADD THE BELOW LINE
HiddenServicePort 443

Reload tor when done (systemctl reload tor@default) and verify by checking tor's logs that there was no error (journalctl -eu tor@default) and that tor is still running (systemctl status tor@default).

Find the 2nd server {} block you added, the one that listens on port 443. We are now going to add a single line to it that instructs nginx to add an HTTP header in its responses, and edit the server_name line to list the onion service.

server {
    # EDIT THE BELOW LINE
    server_name flashflow.pastly.xyz jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion;
    [... lines omitted ...]
    location / {
        try_files $uri $uri/ =404;
        # ADD THE BELOW LINE
        add_header Alt-Svc 'h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;';
    }
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    [... lines omitted ...]
}

Reload nginx and verify it didn't say there was an error.

You can verify the Alt-Svc header is being sent by, well, inspecting the headers that nginx sends when you request either https://flashflow.pastly.xyz or https://jsd33qlp6[...]d.onion.

$ curl --head https://flashflow.pastly.xyz
HTTP/2 200 
server: nginx/1.14.2
[... lines omitted ...]
onion-location: http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion/
alt-svc: h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;
[... lines omitted ...]


# the --insecure flag tells curl to keep going even though it will see a
# cert that isn't valid for the onion service. This is expected, as
# explained previously.
$ torsocks curl --insecure --head https://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion
HTTP/2 200 
server: nginx/1.14.2
[... lines omitted ...]
onion-location: http://jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion/
alt-svc: h2="jsd33qlp6p2t3snyw4prmwdh2sukssefbpjy6katca5imn4zz4pepdid.onion:443"; ma=86400;
[... lines omitted ...]

Verifying that Tor Browser actually uses the headers is harder and beyond the scope of this post. The basic idea is to abuse Alt-Svc to serve something different up via the onion service and check that you get the different content after a couple of page refreshes.

The Tor Project now has a status page which shows the state of our major services.

You can check status.torprojet.org for news about major outages in Tor services, including v3 and v2 onion services, directory authorities, our website (torproject.org), and the check.torproject.org tool. The status page also displays outages related to Tor internal services, like our GitLab instance.

This post documents why we launched status.torproject.org, how the service was built, and how it works.

Why a status page

The first step in setting up a service page was to realize we needed one in the first place. I surveyed internal users at the end of 2020 to see what could be improved, and one of the suggestions that came up was to "document downtimes of one hour or longer" and generally improve communications around monitoring. The latter is still on the sysadmin roadmap, but a status page seemed like a good solution for the former.

We already have two monitoring tools in the sysadmin team: Icinga (a fork of Nagios) and Prometheus, with Grafana dashboards. But those are hard to understand for users. Worse, they also tend to generate false positives, and don't clearly show users which issues are critical.

In the end, a manually curated dashboard provides important usability benefits over an automated system, and all major organisations have one.

Picking the right tool

It wasn't my first foray in status page design. In another life, I had setup a status page using a tool called Cachet. That was already a great improvement over the previous solutions, which were to use first a wiki and then a blog to post updates. But Cachet is a complex Laravel app, which also requires a web browser to update. It generally requires more maintenance than what we'd like, needing silly things like a SQL database and PHP web server.

So when I found cstate, I was pretty excited. It's basically a theme for the Hugo static site generator, which means that it's a set of HTML, CSS, and a sprinkle of Javascript. And being based on Hugo means that the site is generated from a set of Markdown files and the result is just plain HTML that can be hosted on any web server on the planet.

Deployment

At first, I wanted to deploy the site through GitLab CI, but at that time we didn't have GitLab pages set up. Even though we do have GitLab pages set up now, it's not (yet) integrated with our mirroring infrastructure. So, for now, the source is hosted and built in our legacy git and Jenkins services.

It is nice to have the content hosted in a git repository: sysadmins can just edit Markdown in the git repository and push to deploy changes, no web browser required. And it's trivial to setup a local environment to preview changes:

hugo serve --baseUrl=http://localhost/
firefox https://localhost:1313/

Only the sysadmin team and gitolite administrators have access to the repository, at this stage, but that could be improved if necessary. Merge requests can also be issued on the GitLab repository and then pushed by authorized personnel later on, naturally.

Availability

One of the concerns I have is that the site is hosted inside our normal mirror infrastructure. Naturally, if an outage occurs there, the site goes down. But I figured it's a bridge we'll cross when we get there. Because it's so easy to build the site from scratch, it's actually trivial to host a copy of the site on any GitLab server, thanks to the .gitlab-ci.yml file shipped (but not currently used) in the repository. If push comes to shove, we can just publish the site elsewhere and point DNS there.

And, of course, if DNS fails us, then we're in trouble, but that's the situation anyway: we can always register a new domain name for the status page when we need to. It doesn't seem like a priority at the moment.

Comments and feedback are welcome!

This article was first published on the Tor Project Blog.

Rust doesn't yet support asynchronous functions in traits, but several important async-related traits (like AsyncRead and Stream ) define their interface using functions that return Poll. So, what can you do when you have a function that is async, and you need to use it to implement one of these traits?

(I'll be assuming that you already know a little about pinning, futures, and async programming in Rust. That's not because they're easy “everybody-should-know-it” topics, but because I'm still learning them myself, and I don't understand them well enough be a good teacher. You can probably keep reading if you don't understand them well . )

A little background

Here's a situation I ran into earlier this year. In the end, I only solved it with help from Daniel Franke, so I decided that I should write up the solution here in case it can help somebody else.

I've been working on Arti, an implementation of the Tor protocols in Rust. After a bunch of hacking, I finally got to the point where I had a DataStream type that provides an anonymous connection over the Tor network:

impl DataStream {
    pub async fn read(&mut self, buf: &mut[u8]) -> io::Result<usize>
    { ... }
    pub async fn write(&mut self, buf: &[u8]) -> io::Result<usize>
    { ... }
}

Now, there's a lot of complexity hiding in those ellipses. Tor isn't a simple protocol: when it's trying to read, it may need to wait for data to arrive. It may also need to send messages in response to arriving data. It might need to update internal state, or even tear down an entire Tor circuit because of a protocol error. Async functions made it possible to implement all of this stuff in a more-or-less comprehensible way, so rewriting those functions to explicitly return a typed future was not an option.

But I wanted DataStream to implement AsyncRead and AsyncWrite, so I could use it with other code in the Rust async ecosystem. So let's look at AsyncRead (because it's simpler than AsyncWrite). The only required method in AsyncRead is:

pub fn poll_read(
    self: Pin<&mut Self>,
    cx: &mut Context<'_>,
    buf: &mut [u8]
) -> Poll<io::Result<usize>>

This read() has to check whether there's data can be read into buf immediately, without blocking. If there is, we read the data and return the number of bytes we read. Otherwise, we have to schedule ourself on cx, and return Poll::Pending.¹

Moving forward, getting stuck

Compare poll_read to the read function in DataStream. First off, there's a mismatch between how these two functions use their output buffers. Because DataStream::read is async, it returns a future that will hang on to its buffer until the future is finally ready. But poll_read has to return right away, and it can't store a reference to its buffer at all. So I started by defining a wrapper variant of DataStream to implements the behavior that poll_read would need:²

pub struct DataReaderImpl {
    s: DataStream,
    pending: Vec<u8>
    offset: usize,
    len: usize,
}
pub struct DataReaderImpl {
    s: DataStream,
    pending: Vec<u8>
}
impl DataReaderImpl {
    fn new(s: DataStream) -> DataReaderImpl {
        DataReaderImpl {
            s,
            pending: Vec::new(),
        }
    }
    // Load up to 1k into the pending buffer.
    async fn fill_buf(&mut self) -> io::Result<usize> {
        let mut data = vec![0;1024];
        let len = self.s.read(&mut data[..]).await?;
        data.truncate(len);
        self.pending.extend(data);
        Ok(len)
    }
    // pull bytes from the pending buffer into `buf`.
    fn extract_bytes(&mut self, buf: &mut [u8]) -> usize {
        let n = cmp::min(buf.len(), self.pending.len());
        buf[..n].copy_from_slice(&self.pending[..n]);
        self.pending.drain(0..n);
        n
    }
}

Then, I thought, it ought to be easy to write AsyncRead! Here was my first try:

// This won't work...
impl AsyncRead for DataReaderImpl {
    fn poll_read(mut self: Pin<&mut Self>,
                 cx: &mut Context<'_>,
                 buf: &mut [u8]) -> Poll<io::Result<usize>> {
       if self.pending.is_empty() {
            // looks like we need more bytes.
            let fut = self.fill_buf();
            futures::pin_mut!(fut);
            match fut.poll(cx) {
                Poll::Ready(Err(e)) =>
                    return Poll::Ready(Err(e)),
                Poll::Ready(Ok(n)) =>
                    if n == 0 {
                        return Poll::Ready(Ok(0)); // EOF
                    }
                Poll::Pending =>
                    todo!("crud, where do i put the future?"), // XXXX
            }
        }

        // We have some data; move it out to the caller.
        let n = self.extract_bytes(buf);
        Poll::Ready(Ok(n))
    }
}

Almost there! But what do I do if the future says it's pending? I need to store it and poll it again later the next time I call this function. But to do that, I won't be able to pin the future to the stack! I'll have to store it in the structure instead. And since the future comes from an async function, it won't have a type that I can name; I'll have to store it as a Box<dyn Future>.

Oh hang on, it'll need to be pinned. And sometimes there won't be a read in progress, so I won't have a future at all. Maybe I store it in an Option<Pin<Box<dyn Future>>>?

(This is the point where I had to take a break and figure out pin-projection³.)

But after I played around with that for a while, I hit the final snag: ultimately, I was trying to create a self-referential structure⁴, which you can't do in safe Rust. You see, the future returned by DataReaderImpl::fill_buf needs to hold a reference to the DataReaderImpl, and so the future needs to outlive the DataReaderImpl. That means you can't store it in the DataReaderImpl. You can't even store it and the DataReaderImpl in the same struct: that creates self-reference.

So what could I do? Was I supposed to use unsafe code or some tricky crate to make a self-referential struct anyway? Was my solution fundamentally flawed? Was I even trying to do something possible‽

I asked for help on Twitter. Fortunately, Daniel Franke got back to me, looked at my busted code, and walked me through the actual answer.

Hold the future or the reader: not both!

Here's the trick: We define an enum that holds the DataReaderImpl or the future that its fill_buf function returns, but not both at once. That way, we never have a self-referential structure!

First we had to define a new variation on fill_buf that will take ownership of the reader when it's called, and return ownership once it's done:

impl DataReaderImpl {
    async fn owning_fill_buf(mut self) -> (Self, io::Result<usize>) {
        let r = self.fill_buf().await;
        (self, r)
    }
}

Then we had to define an enum that could hold either the future or the DataReaderImpl object, along with a wrapper struct to hold the enum.

type OwnedResult = (DataReaderImpl, io::Result<usize>);
enum State {
    Closed,
    Ready(DataReaderImpl),
    Reading(Pin<Box<dyn Future<Output=OwnedResult>>>),
}
struct DataReader {
    state: Option<State>
}

Note that the DataReader struct holds an Option<State>—we'll want to modify the state object destructively, so we'll need to take ownership of the state in poll_read and then replace it with something else.⁵

With this groundwork in place we could finally give an implementation of AsyncRead that works:

impl AsyncRead for DataReader {
    fn poll_read(mut self: Pin<&mut Self>,
                 cx: &mut Context<'_>,
                 buf: &mut [u8]) -> Poll<io::Result<usize>> {
        // We're taking this temporarily. We have to put
        // something back before we return.
        let state = self.state.take().unwrap();

        // We own the state, so we can destructure it.
        let mut future = match state {
            State::Closed => {
                self.state = Some(State::Closed);
                return Poll::Ready(Ok(0));
            }
            State::Ready(mut imp) => {
                let n = imp.extract_bytes(buf);
                if n > 0 {
                    self.state = Some(State::Ready(imp));
                    // We have data, so we can give it back now.
                    return Poll::Ready(Ok(n));
                }
                // Nothing available; launch a read and poll it.
                Box::pin(imp.owning_fill_buf())
            }
            // If we have a future, we need to poll it.
            State::Reading(fut) => fut,
        };

        // Now we have a future for an in-progress read.
        // Can it make any progress?
        match future.as_mut().poll(cx) {
            Poll::Ready((_imp, Err(e))) => { // Error
                self.state = Some(State::Closed);
                Poll::Ready(Err(e))
            }
            Poll::Ready((_imp, Ok(0))) => { // EOF
                self.state = Some(State::Closed);
                Poll::Ready(Ok(0))
            }
            Poll::Ready((mut imp, Ok(_))) => {
                // We have some data!
                let n = imp.extract_bytes(buf);
                self.state = Some(State::Ready(imp));
                debug_assert!(n > 0);
                Poll::Ready(Ok(n))
            }
            Poll::Pending => {
                // We're pending; remember the future
                // and tell the caller.
                self.state = Some(State::Reading(future));
                Poll::Pending
            }
        }
    }
}

Now when poll_read() takes ownership of the previous state, it either owns a DataReaderImpl or a future returned by owning_fill_buf()—but never both at once, so we don't have any self-reference problems. When poll_read() done, it has to put a new valid state back before it returns.

Conclusions

For the current version of all this code, have a look at tor_proto::stream::data in Arti. Note that the code in Arti is more complex than what I have in this post, and some of that complexity is probably unnecessary: I've been learning more about Rust as I go along.

I hope that some day there's an easier way to do all of this (with real asynchronous traits, maybe?) but in the meantime, I hope that this write-up will be useful to somebody else.

I wrote for LWN for about two years. During that time, I wrote (what seems to me an impressive) 34 articles, but I always had a pile of ideas in the back of my mind. Those are ideas, notes, and scribbles lying around. Some were just completely abandoned because they didn't seem a good fit for LWN.

Concretely, I stored those in branches in a git repository, and used the branch name (and, naively, the last commit log) as indicators of the topic.

This was the state of affairs when I left:

remotes/private/attic/novena                    822ca2bb add letter i sent to novena, never published
remotes/private/attic/secureboot                de09d82b quick review, add note and graph
remotes/private/attic/wireguard                 5c5340d1 wireguard review, tutorial and comparison with alternatives
remotes/private/backlog/dat                     914c5edf Merge branch 'master' into backlog/dat
remotes/private/backlog/packet                  9b2c6d1a ham radio packet innovations and primer
remotes/private/backlog/performance-tweaks      dcf02676 config notes for http2
remotes/private/backlog/serverless              9fce6484 postponed until kubecon europe
remotes/private/fin/cost-of-hosting             00d8e499 cost-of-hosting article online
remotes/private/fin/kubecon                     f4fd7df2 remove published or spun off articles
remotes/private/fin/kubecon-overview            21fae984 publish kubecon overview article
remotes/private/fin/kubecon2018                 1edc5ec8 add series
remotes/private/fin/netconf                     3f4b7ece publish the netconf articles
remotes/private/fin/netdev                      6ee66559 publish articles from netdev 2.2
remotes/private/fin/pgp-offline                 f841deed pgp offline branch ready for publication
remotes/private/fin/primes                      c7e5b912 publish the ROCA paper
remotes/private/fin/runtimes                    4bee1d70 prepare publication of runtimes articles
remotes/private/fin/token-benchmarks            5a363992 regenerate timestamp automatically
remotes/private/ideas/astropy                   95d53152 astropy or python in astronomy
remotes/private/ideas/avaneya                   20a6d149 crowdfunded blade-runner-themed GPLv3 simcity-like simulator
remotes/private/ideas/backups-benchmarks        fe2f1f13 review of backup software through performance and features
remotes/private/ideas/cumin                     7bed3945 review of the cumin automation tool from WM foundation
remotes/private/ideas/future-of-distros         d086ca0d modern packaging problems and complex apps
remotes/private/ideas/on-dying                  a92ad23f another dying thing
remotes/private/ideas/openpgp-discovery         8f2782f0 openpgp discovery mechanisms (WKD, etc), thanks to jonas meurer
remotes/private/ideas/password-bench            451602c0 bruteforce estimates for various password patterns compared with RSA key sizes
remotes/private/ideas/prometheus-openmetrics    2568dbd6 openmetrics standardizing prom metrics enpoints
remotes/private/ideas/telling-time              f3c24a53 another way of telling time
remotes/private/ideas/wallabako                 4f44c5da talk about wallabako, read-it-later + kobo hacking
remotes/private/stalled/bench-bench-bench       8cef0504 benchmarking http benchmarking tools
remotes/private/stalled/debian-survey-democracy 909bdc98 free software surveys and debian democracy, volunteer vs paid work

Wow, what a mess! Let's see if I can make sense of this:

Attic

Those are articles that I thought about, then finally rejected, either because it didn't seem worth it, or my editors rejected it, or I just moved on:

• novena: the project is ooold now, didn't seem to fit a LWN article. it was basically "how can i build my novena now" and "you guys rock!" it seems like the MNT Reform is the brain child of the Novena now, and I dare say it's even cooler!
• secureboot: my LWN editors were critical of my approach, and probably rightly so - it's a really complex subject and I was probably out of my depth... it's also out of date now, we did manage secureboot in Debian
• wireguard: LWN ended up writing extensive coverage, and I was biased against Donenfeld because of conflicts in a previous project

Backlog

Those were articles I was planning to write about next.

• dat: I already had written Sharing and archiving data sets with Dat, but it seems I had more to say... mostly performance issues, beaker, no streaming, limited adoption... to be investigated, I guess?
• packet: a primer on data communications over ham radio, and the cool new tech that has emerged in the free software world. those are mainly notes about Pat, Direwolf, APRS and so on... just never got around to making sense of it or really using the tech...
• performance-tweaks: "optimizing websites at the age of http2", the unwritten story of the optimization of this website with HTTP/2 and friends
• serverless: god. one of the leftover topics at Kubecon, my notes on this were thin, and the actual subject, possibly even thinner... the only lie worse than the cloud is that there's no server at all! concretely, that's a pile of notes about Kubecon which I wanted to sort through. Probably belongs in the attic now.

Fin

Those are finished articles, they were published on my website and LWN, but the branches were kept because previous drafts had private notes that should not be published.

Ideas

• astropy: "Python in astronomy" - had a chat with saimn while writing about sigal, and it turns out he actually works on free software in astronomy, in Python... I actually expect LWN to cover this sooner than later, after Lee Phillips's introduction to SciPy
• avaneya: crowdfunded blade-runner-themed GPLv3 simcity-like simulator, i just have that link so far
• backups-benchmarks: review of backup software through performance and features, possibly based on those benchmarks, maybe based on this list from restic although they refused casync. benchmark articles are hard though, especially when you want to "cover them all"... I did write a silly Attic vs Bup back when those programs existed (2014), in a related note...
• ideas/cumin: review of the Cumin automation tool from WikiMedia Foundation... I ended up using the tool at work and writing service documentation for it
• ideas/future-of-distros: modern packaging problems and complex apps, starting from this discussion about the removal of Dolibarr from Debian, a summary of the thread from liw, and ideas from joeyh (now from the outside of Debian), then debates over the power of FTP masters - ugh, glad I didn't step in that rat's nest
• ideas/on-dying: "what happens when a hacker dies?" rather grim subject, but a more and more important one... joeyh has ideas again, phk as well, then there's a protocol for dying (really grim)... then there are site policies like GitHub, Facebook, etc... more in the branch, but that one I can't help but think about now that family has taken a bigger place in my life...
• ideas/openpgp-discovery: OpenPGP discovery mechanisms (WKD, etc), suggested by Jonas Meurer (somewhere?), only links to Mailveloppe, LEAP, WKD (or is it WKS?), another standard, probably would need to talk about OpenPGP CA now and how Debian and Tor manage their keyrings... pain in the back.
• ideas/password-bench: bruteforce estimates for various password patterns compared with RSA key sizes, spinoff of my smartcard article, in the crypto-bench, look at this shiny graph, surely that must mean an article, right?
• ideas/prometheus-openmetrics: "Evolving the Prometheus exposition format into a standard", seems like this happened
• ideas/telling-time: telling time to users is hard. xclock vs ttyclock, etc. maybe gameclock and undertime as well? syncing time is hard, but it turns out showing it is non trivial as well... basically turning this bug report into an article. for some reason I linked to this meme, derived from this meme, presumably a premonition of my stupid idea of writing undertime TIMEZONES!
• ideas/wallabako: "talk about wallabako, read-it-later + kobo hacking", that's it, not even a link to the project!

A lot of those branches were actually just an empty commit, with the commitlog being the "pitch", more or less. I'd send that list to my editors, sometimes with a few more links (basically the above), and they would nudge me one way or the other.

Sometimes they would actively discourage me to write about something, and I would do it anyways, send them a draft, and they would patiently make me rewrite it until it was a decent article. This was especially hard with the terminal emulator series, which took forever to write and even got my editors upset when they realized I had never installed Fedora (I ended up installing it, and I was proven wrong!)

Stalled

Oh, and then there's those: those are either "ideas" or "backlog" that got so far behind that I just moved them out of the way because I was tired of seeing them in my list.

• stalled/bench-bench-bench benchmarking http benchmarking tools, a horrible mess of links, copy-paste from terminals, and ideas about benchmarking... some of this trickled out into this benchmarking guide at Tor, but not much more than the list of tools
• stalled/debian-survey-democracy: "free software surveys and Debian democracy, volunteer vs paid work"... A long standing concern of mine is that all Debian work is supposed to be volunteer, and paying explicitly for work inside Debian has traditionally been frowned upon, even leading to serious drama and dissent (remember Dunc-Tank)? back when I was writing for LWN, I was also doing paid work for Debian LTS. I also learned that a lot (most?) Debian Developers were actually being paid by their job to work on Debian. So I was confused by this apparent contradiction, especially given how the LTS project has been mostly accepted, while Dunc-Tank was not... See also this talk at Debconf 16. I had hopes that this study would show the "hunch" people have offered (that most DDs are paid to work on Debian) but it seems to show the reverse (only 36% of DDs, and 18% of all respondents paid). So I am still confused and worried about the sustainability of Debian.

What do you think?

So that's all I got. As people might have noticed here, I have much less time to write these days, but if there's any subject in there I should pick, what is the one that you would find most interesting?

Oh! and I should mention that you can write to LWN! If you think people should know more about some Linux thing, you can get paid to write for it! Pitch it to the editors, they won't bite. The worst that can happen is that they say "yes" and there goes two years of your life learning to write. Because no, you don't know how to write, no one does. You need an editor to write.

That's why this article looks like crap and has a smiley.

Time flies. Back in 2008, I wrote a game clock. Since then, what was first called "chess clock" was renamed to pychessclock and then Gameclock (2008). It shipped with Debian 6 squeeze (2011), 7 wheezy (4.0, 2013, with a new UI), 8 jessie (5.0, 2015, with a code cleanup, translation, go timers), 9 stretch (2017), and 10 buster (2019), phew! Eight years in Debian over 4 releases, not bad!

But alas, Debian 11 bullseye (2021) won't ship with Gameclock because both Python 2 and GTK 2 were removed from Debian. I lack the time, interest, and energy to port this program. Even if I could find the time, everyone is on their phone nowadays.

So finding the right toolkit would require some serious thinking about how to make a portable program that can run on Linux and Android. I care less about Mac, iOS, and Windows, but, interestingly, it feels it wouldn't be much harder to cover those as well if I hit both Linux and Android (which is already hard enough, paradoxically).

(And before you ask, no, Java is not an option for me thanks. If I switch to anything else than Python, it would be Golang or Rust. And I did look at some toolkit options a few years ago, was excited by none.)

So there you have it: that is how software dies, I guess. Alternatives include:

• Chessclock - really old Ruby which made Gameclock rename
• Ghronos - also really old Java app
• Lichess - has a chess clock built into the app
• Otter - if you squint a little

PS: Monkeysign also suffered the same fate, for what that's worth. Alternatives include caff, GNOME Keysign, and pius. Note that this does not affect the larger Monkeysphere project, which will ship with Debian bullseye.

Last week I did two rounds of my Creating Python modules in Rust workshop. During the second session on Sunday, someone asked if we can create standard dunder methods, say __str__ or __repr__. I never did that before, and during the session I tried to read the docs and implement it. And I failed :)

Later I realized that I should have read the docs carefully. To add those methods, we will have to implement PyObjectProtocol for the Rust structure.

#[pyproto]
impl PyObjectProtocol for Ros {
    fn __repr__(&self) -> PyResult<String> {
        let cpus = self.sys.get_processors().len();
        let repr = format!("<Ros(CPUS: {})>", cpus);
        Ok(repr)
    }

    fn __str__(&self) -> PyResult<String> {
        let cpus = self.sys.get_processors().len();
        let repr = format!("<Ros(CPUS: {})>", cpus);
        Ok(repr)
    }
}

>>> from randomos import Ros
>>> r = Ros()
>>> r
<Ros (CPUS: 8)>
>>> str(r)
'<Ros (CPUS: 8)>'

This code example is in the ros-more branch of the code.