December 02, 2022

hackergotchi for Junichi Uekawa

Junichi Uekawa

Already December.

Already December. Things changed a lot. Six months ago I was at home most of the time. I hope I can keep on going out for a while.

02 December, 2022 10:13AM by Junichi Uekawa

Paul Wise

FLOSS Activities November 2022


This month I didn't have any particular focus. I just worked on issues in my info bubble.





  • Debian IRC: setup topic/entrymsg to redirect OFTC #debian-jp to the Debian Japan IRC server, cleaned up #debianppc ChanServ metadata
  • Debian wiki: unblock IP addresses, approve domains, approve accounts



The libemail-outlook-message-perl, purple-discord, sptag, celery work was sponsored. All other work was done on a volunteer basis.

02 December, 2022 02:58AM

November 30, 2022

Enrico Zini

Things I learnt in November 2022


  • You can Build-Depend on debhelper-compat (=version) and get rid of debhelper as a build-dependency, and of debian/compat (details)
  • You can Build-Depend on dh-sequence-foo and get rid of the corresponding dh-foo build-dependency, and of the need to add --with foo in debian/rules (details)
  • You can (and should) get rid of dh-buildinfo, which is now handled automatically
  • In salsa.debian.org there is a default CI pipeline for Debian packages that works beautifully without needing to add any .gitlab-ci.yml to a repository
  • Add Testsuite: autopkgtest-pkg-python to debian/control, and you get a free autopkgtest that verifies that your packaged Python module can be imported. The default CI pipeline in salsa will automatically run the tests. (specification, details)


  • From Python 3.8, you can use = in format strings to make it easier to debug variables and expressions (details):
>>> name="test"
>>> print(f"{name=}")
>>> print(f"{3*8=}")


  • [abc].tile.openstreetmap.org links need to be replaced with tile.openstreetmap.org (details)

30 November, 2022 11:00PM

hackergotchi for Matthew Garrett

Matthew Garrett

Making unphishable 2FA phishable

One of the huge benefits of WebAuthn is that it makes traditional phishing attacks impossible. An attacker sends you a link to a site that looks legitimate but isn't, and you type in your credentials. With SMS or TOTP-based 2FA, you type in your second factor as well, and the attacker now has both your credentials and a legitimate (if time-limited) second factor token to log in with. WebAuthn prevents this by verifying that the site it's sending the secret to is the one that issued it in the first place - visit an attacker-controlled site and said attacker may get your username and password, but they won't be able to obtain a valid WebAuthn response.

But what if there was a mechanism for an attacker to direct a user to a legitimate login page, resulting in a happy WebAuthn flow, and obtain valid credentials for that user anyway? This seems like the lead-in to someone saying "The Aristocrats", but unfortunately it's (a) real, (b) RFC-defined, and (c) implemented in a whole bunch of places that handle sensitive credentials. The villain of this piece is RFC 8628, and while it exists for good reasons it can be used in a whole bunch of ways that have unfortunate security consequences.

What is the RFC 8628-defined Device Authorization Grant, and why does it exist? Imagine a device that you don't want to type a password into - either it has no input devices at all (eg, some IoT thing) or it's awkward to type a complicated password (eg, a TV with an on-screen keyboard). You want that device to be able to access resources on behalf of a user, so you want to ensure that that user authenticates the device. RFC 8628 describes an approach where the device requests the credentials, and then presents a code to the user (either on screen or over Bluetooth or something), and starts polling an endpoint for a result. The user visits a URL and types in that code (or is given a URL that has the code pre-populated) and is then guided through a standard auth process. The key distinction is that if the user authenticates correctly, the issued credentials are passed back to the device rather than the user - on successful auth, the endpoint the device is polling will return an oauth token.

But what happens if it's not a device that requests the credentials, but an attacker? What if said attacker obfuscates the URL in some way and tricks a user into clicking it? The user will be presented with their legitimate ID provider login screen, and if they're using a WebAuthn token for second factor it'll work correctly (because it's genuinely talking to the real ID provider!). The user will then typically be prompted to approve the request, but in every example I've seen the language used here is very generic and doesn't describe what's going on or ask the user. AWS simply says "An application or device requested authorization using your AWS sign-in" and has a big "Allow" button, giving the user no indication at all that hitting "Allow" may give a third party their credentials.

This isn't novel! Christoph Tafani-Dereeper has an excellent writeup on this topic from last year, which builds on Nestori Syynimaa's earlier work. But whenever I've talked about this, people seem surprised at the consequences. WebAuthn is supposed to protect against phishing attacks, but this approach subverts that protection by presenting the user with a legitimate login page and then handing their credentials to someone else.

RFC 8628 actually recognises this vector and presents a set of mitigations. Unfortunately nobody actually seems to implement these, and most of the mitigations are based around the idea that this flow will only be used for physical devices. Sadly, AWS uses this for initial authentication for the aws-cli tool, so there's no device in that scenario. Another mitigation is that there's a relatively short window where the code is valid, and so sending a link via email is likely to result in it expiring before the user clicks it. An attacker could avoid this by directing the user to a domain under their control that triggers the flow and then redirects the user to the login page, ensuring that the code is only generated after the user has clicked the link.

Can this be avoided? The best way to do so is to ensure that you don't support this token issuance flow anywhere, or if you do then ensure that any tokens issued that way are extremely narrowly scoped. Unfortunately if you're an AWS user, that's probably not viable - this flow is required for the cli tool to perform SSO login, and users are going to end up with broadly scoped tokens as a result. The logs are also not terribly useful.

The infuriating thing is that this isn't necessary for CLI tooling. The reason this approach is taken is that you need a way to get the token to a local process even if the user is doing authentication in a browser. This can be avoided by having the process listen on localhost, and then have the login flow redirect to localhost (including the token) on successful completion. In this scenario the attacker can't get access to the token without having access to the user's machine, and if they have that they probably have access to the token anyway.

There's no real moral here other than "Security is hard". Sorry.

comment count unavailable comments

30 November, 2022 09:53PM

hackergotchi for Daniel Pocock

Daniel Pocock

Shaya Potter & Debian WaReZ expulsion

In recent years, Debian leaders have abused their positions to leak rumors and gossip about some developers while hiding far more serious scandals.

One of those is the shadow expulsion of Shaya Potter (blog, CV) in 1998.

Potter was a child prodigy who began a US Navy internship when he was fifteen or sixteen. At the same time, Novare, Inc was hosting some Debian infrastructure on their company servers.

Potter claims his software was being deployed to the USS Theodore Roosevelt (Secure Tactical Access Terminal) while at the same time, Debian records show that he was stashing WaReZ on master.debian.org, a server operated by Novare.

The case of a navy intern committing piracy is interesting for a wide range of reasons that have a lot more to do with Debian than the navy.

Potter began co-authoring Debian as a minor. It appears that he was 19 when he was caught with WaReZ but it is not clear what age he was when he actually downloaded these illegal copies of software. The age of majority varies between different countries and different states of the US. Therefore, there was some confusion about whether it should be handled as a juvenile offence. I recently wrote about the concerns with FSFE encouraging minors to do unpaid work on open source projects.

Earlier this year I wrote about the fact that only 86 of the original Debian co-authors agreed to form an association with a constitution. If we look at the names of the people who consented to form an association, Shaya Potter is not in the list. If Potter did not consent to the constitution then he wasn't a member at all. Morever, the association only came into existence at the end of the vote in December 1998. Potter was supposedly expelled in November 1998, one month before the vote. Therefore, as the organization didn't really exist yet and as he hadn't consented to be a member, we can't really say he was expelled at all.

Nonetheless, it looks like his position with the US Navy also ended the year after the Debian drama. Barely five years later, Potter went on to win the LISA'05 award for the best student paper.

The great irony in this case is that people were angry with Potter for violating copyright law. Yet this was the first time the developers forcefully removed the name of a Debian co-author from the software. Therefore, by removing his name and failing to give him equal credit as a joint author of the Debian software, they were violating his copyright interests. They set a precedent for violating the copyright interests of other authors, for example, the violent suppression of Ted Walther in DebConf6.

To put this in perspective, consider the case of Rolf Harris, convicted of harassment and abuse in the UK. His copyright interests from a long career in show business still continued to receive royalties even during his time in prison. If a sex offender can continue receiving royalties while in prison, why can't all Debian Developers continue receiving recognition even after minor disputes within the project?

At the time, developers expressed some concern that the scandal would be leaked and gain significant publicity due to the connection with the US Navy.

It is interesting to see how this scandal regarding a minor and the US military was covered up while gossip about other developers, like the assault on Ted Walther was deliberately leaked. The accusations against Potter were far more serious than the gossip about Walther.

In 2018, Debian leader Chris Lamb started making attacks on the privacy of volunteers and our families.

Despite supposedly being "expelled" from an organization that didn't exist, Potter has continued maintaining the hebcal package, the Perpetual Jewish Calendar, for more than twenty years. Potter moved to Israel in August 2019, four months after the decision to send DebConf20 to Haifa, Israel. It looks like the expulsion was in some ways a charade so that Debian can pretend they take a hard line on piracy while in reality, there were no consequences for Potter's career. He walked straight out of the US Navy and into IBM TJ Watson research laboratory.

One of Potter's earlier emails to the debian-private gossip network

Subject: Re: Debian release strategy (Was: Re: XFree86 3.2 is out, should we , , delay the release , of Debian 1.2)
Date: Fri, 1 Nov 1996 14:17:42 -0500 (EST)
From: Shaya Potter <spotter@itd.nrl.navy.mil>
To: Klee Dienes <klee@sedona.com>
CC: Bruce Perens <Bruce@pixar.com>, debian-private@lists.debian.org

On 1 Nov 1996, Klee Dienes wrote:

> > I think the key is getting LIBC 6 and the new X available _early_ in
> > the time frame.
> I've got a preliminary packaging of glibc-1.96 ready (as part of the
> testing/development of the POSIX/FIPS-152 conformance testing
> package).  I've been holding off on uploading it until after rex was
> frozen to avoid adding too much confusion to the pot, as well as to
> see what comes of our new-found relationship with Lasermoon.  I'll
> upload a copy on Monday after the code freeze takes place.

Oh, about the posix testing, if you need help I am still here.


Shaya Potter

Please respect the confidentiality of material on the debian-private list.
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-private-REQUEST@lists.debian.org . Trouble? e-mail to Bruce@Pixar.com

Potter asked the Debian leader to hand over copies of debian-private discussion about underage developers.

Subject: Re: why I want the archives on me (was Re: spotter@debian.org)
Date: Tue, 17 Nov 1998 12:56:41 -0500
From: Shaya Potter <spotter@ymail.yu.edu>
To: joost@pc47.mpn.cp.philips.com
CC: debian-private@lists.debian.org

----- Original Message -----
From: <joost@pc47.mpn.cp.philips.com>

>On Tue, 17 Nov 1998, Shaya Potter wrote:
>> Now that this is out of the way, I'd like to publicly ask if I can have
>> archive of all the communication that went on in regard to me.
>Strictly speaking I tend to disagree that you or anybody has an a-priori
>right to know what is being said and told on debian-private.  It is simply
>a private list.  Things would be different if you were mentioned in a
>public list without being able to respond.  But that is in all aspects
>clearly not the current situation.

First, I never said I have a right.  In many ways I think i don't have a
right, or even if I did, I don't deserve it.  I don't think my statements
have implied that I believe I have a right to demand that it be given to me.

I do have a right to ask that it be done.  Debian has a right to say yes or

>(Nevertheless, I think that it would be considerate to cc: you in
>any discussion that involves you in a very personal manner - this has
>IMHO until now hardly been the case though.)

It hasn't?  Than how did the decision to expell me come about?  Who told
people who made the decision what happened?  Was this all done in private

>If a non-subscriber of debian-private must share in the conversation on
>debian-private, then this should IMHO be done by adding that person to the
>clearly visible cc: line of the header of any messages to be "published."
>That way, it will be adequately clear that the correspondence leaves the
>realm of debian-private and thus everybody can conclude that normal
>confidentiality can not be expected.  AFAIK respect for the confidential
>nature of debian-private is a prerequisite for subscription to this list.

I would have respected the confidentiality, as I have made it known that I
don't want this to spread, as I am embarrased by my actions.

>Practically speaking, I disagree that the underlying case generally
>concerns you. What matters here is not who Shaya Potter personally is or
>what particularly Shaya Potter did. The discussion is about how issues
>like the one involving you relate to Debian.  This discussion does not
>involve you personally.

I don't want the entire discussion, I just want to see the parts that touch
on me personally.  I don't care for the rest, of what about underage
developers and the like....

>> I was told that it would not be a star chamber, and that I'd be cc'd in
>> on all the corrospondace.  That didn't occur.
>There was no "star chamber."  You have already been generously cc:'-ed.

I was?  The only cc:'s I ever got were in response to me starting a thread.
That implies to me, that acc. to what you were saying, that no discussion
on -private occured that I didn't start.  However, I know this not to be the
case, as before I was unsubscribed from -private, I saw a thread or 2
started that dealt with me.

>IMHO you do not have a right to be cc:-'ed on the _general_ discussion
>which does not particularly (personally) involve you.

never said I did.

>> Also, I really have no idea of what discussion went on, if mistruthes
>> were spread about the incident (as in reality, I'm the only one that
>> knows completely what happened, and no one really ever asked me for the
>> full story).
>If this worries you so much, then I seriously wonder why you did not
>immediately relate it to debian-private when the issue arose in the first

I did apologize on -private right away, however, I didn't want to spread
what I did.  I specifically told people that I would rather this not be
discussed on -private and have me showed the door quietly, and told never to
come back.  That didn't happen, it was discussed on -private.  I don't know
what was discussed in relation to me, so I want to be informed.

>Again, the discussion is not yours.  Again, you are not personally
>involved.  Your only "role" in the discussion is that you have created a
>precedent.  I thinks we can all agree that we would rather have had you
>not be a precedent case, but it happened.  I'm very sorry, but you'll
>have to blame yourself for that.

Trust me, I've blamed myself a lot for this.  If you seen any of my
corrospondance you would know this.  I don't blame anyone for my
predicament, but myself.

>Discussion on debian-private does not count as a statement from Debian.
>So there simply were no statements.  I'm not really in favor of making any
>strong or overly verbose statements either.  If there ever is to be a
>statement from Debian about an issue such as the current one involving
>Shaya, I think that person should be briefed thoroghly beforehand.

I'm not talking about a debian statement.  I don't want a public statement,
and I know a lot of people from debian don't want one either (though some
might).  What I meant by statements, was statements that individuals made,
that might be incorrect, or inacurate.

>Shaya, can you please just put this to a rest?  IMHO it is not very
>productive for anybody.  And please take it from me that you have no
>reason to be concerned that you have been in a "star chamber."

I am not worried about a star chamber, I would have prefered it in many
ways.  However, at least with a star chamber you usually get to see the case
presented against you, even though you don't have the ability to defend
yourself.  As I said many times, my case is indefensable, so that wouldn't
bother me.


Potter actually resigned on 2 November 1998 but they still spent two weeks deciding to expel him

He was not a member anyway as the organization didn't exist yet. He resigned. Therefore, how obscene it is to retrospectively have an expulsion.

This practice of expelling a non-member, which is obviously unethical and impossible, laid the foundation for many future problems in Debian. For example, Frans Pop had tried to resign before committing suicide but people pulled him back in.

Subject: Re: Novare and master
Date: Mon, 2 Nov 1998 15:56:06 -0600
From: Ean R . Schuessler 
To: Shaya Potter 
CC: debian-private@lists.debian.org

On Mon, Nov 02, 1998 at 12:59:22AM -0500, Shaya Potter wrote:
> I have heard through the grapevine that you are considering pulling
> novare's support of debian b/c of my actions.  is this true?  if it is, I
> would hope that you reconsider and view me as an aberation.  My behavior
> was wrong, stupid, illegal, puts novare and debian in a sticky situation
> and above all that, broke a trust that was given to me.  I would like to
> apologize for these things, but in my mind an apology doesn't fix the
> past, it can only set the foundation for the future.  If in your mind, you
> would be more open to allowing debian to continue on as is, if I am gone,
> let this be my letter of resignation.  If other debian developers think I
> should be gone,let it too be a letter of resignation.  I don't want this
> to turn into a public argument and the more it's argued even in private,
> the more public it can become.  I can't take the embarasment and rather be
> kicked out and punished swiftly than have the chance of staying.

Say what? You're on smack. Kick your bootie for being a naughty little man, yes. Stop supporting Debian because you wanted some 3l1t3 w4r3$,
not friggin' likely.

I think that you should take on 10 new packages and maintain them in an
exemplary manner, you should also break some raw eggs on your face, take
a picture and we'll post it on devel. I might even send my girlfriend to
infiltrate your Navy base and spank you bloody. What you should stop doing,
however, is whining and pirating software.


Ean Schuessler                Director of New Products and Technologies
Novare International Inc.        The Unstoppable Fist of Digital Action
*** WARNING: This signature may contain jokes.

30 November, 2022 09:30PM

hackergotchi for Bits from Debian

Bits from Debian

New Debian Developers and Maintainers (September and October 2022)

The following contributors got their Debian Developer accounts in the last two months:

  • Abraham Raji (abraham)
  • Phil Morrell (emorrp1)
  • Anupa Ann Joseph (anupa)
  • Mathias Gibbens (gibmat)
  • Arun Kumar Pariyar (arun)
  • Tino Didriksen (tinodidriksen)

The following contributors were added as Debian Maintainers in the last two months:

  • Gavin Lai
  • Martin Dosch
  • Taavi Väänänen
  • Daichi Fukui
  • Daniel Gröber
  • Vivek K J
  • William Wilson
  • Ruben Pollan


30 November, 2022 03:00PM by Jean-Pierre Giraud

Russell Coker

Russ Allbery

Review: The Fed Unbound

Review: The Fed Unbound, by Lev Menand

Publisher: Columbia Global Reports
Copyright: 2022
ISBN: 1-7359137-1-5
Format: Kindle
Pages: 156

The Fed Unbound is a short non-fiction exploration of US Federal Reserve actions to reducing systemic risk caused by shadow banking. Its particular focus is the role of the Fed from the 2008 financial crisis to the present, including the COVID shock, but it includes a history of what Menand calls the "American Monetary Settlement," the political compromise that gave rise to the Federal Reserve.

In Menand's view, a central cause of instability in the US financial system (and, given the influence of the dollar system, the world financial system as well) is shadow banking: institutions that act as banks without being insured like banks or subject to bank regulations. A bank, in this definition, is an organization that takes deposits. I'm simplifying somewhat, but what distinguishes a deposit from a security or an investment is that deposits can be withdrawn at any time, or at least on very short notice. When you want to spend the money in your checking account, you don't have to wait for a three-month maturity period or pay an early withdrawal penalty. You simply withdraw the money, with immediate effect. This property is what makes deposits "money," rather than something that you can (but possibly cannot) sell for money, such as stocks or bonds.

Most people are familiar with the basic story of how banks work. Essentially no bank simply takes people's money and puts it in a vault until the person wants it again. If that were the case, you would need to pay the bank to store your money. Instead, a bank takes in deposits and then lends some portion of that money out to others. Those loans, for things like cars or houses or credit card spending, come due over time, with interest. The interest rate the bank charges on the loans is much higher than the rate it has to pay on its deposits, and it pockets the difference.

The problem with this model, of course, is that the bank doesn't have your money, so if all the depositors go to the bank at the same time and ask for their money, the bank won't be able to repay them and will collapse. (See, for example, the movie It's a Wonderful Life, or Mary Poppins, or any number of other movies or books.) Retail banks are therefore subject to stringent regulations designed to promote public trust and to ensure that traditional banking is a boring (if still lucrative) business. Banks are also normally insured, which in the US means that if they do experience a run, federal regulators will step in, shut down the bank in an orderly fashion, and ensure every depositor gets their money back (at least up to the insurance limit).

Alas, if you thought people would settle for boring work that makes a comfortable profit, you don't know the financial industry. Highly-regulated insured deposits are less lucrative than acting like a bank without all of those restrictions and rules and deposit insurance payments. As Menand relates in his brief history of US banking, financial institutions constantly invent new forms of deposits with similar properties but without all the pesky rules: eurodollars (which have nothing to do with the European currency), commercial paper, repo, and many others. These forms of deposits are primarily used by large institutions like corporations. The details vary, but they tend to be prone to the same fundamental instability as bank deposits: if there's a run on the market, there may not be enough liquidity for everyone to withdraw their money at once. Unlike bank deposits, though, there is no insurance, no federal regulator to step in and make depositors whole, and much less regulation to ensure that runs are unlikely.

Instead, there's the Federal Reserve, which has increasingly become the bulwark against liquidity crises among shadow banks. This happened in 2008 during the financial crisis (which Menand argues can be seen as a shadow bank run sparked by losses on mortgage securities), and again at a larger scale in 2020 during the initial COVID crisis.

Menand is clear that these interventions from the Federal Reserve were necessary. The alternative would have been an uncontrolled collapse of large sections of the financial system, with unknown consequences. But the Fed was not intended to perform those types of interventions. It has no regulatory authority to reform the underlying financial systems to make them safer, remove executives who failed to maintain sufficient liquidity for a crisis, or (as is standard for all traditional US banks) prohibit combining banking and more speculative investment on the same balance sheet. What the Federal Reserve can do, and did, is function as a buyer of last resort, bailing out shadow banks by purchasing assets with newly-created money. This works, in the sense that it averts the immediate crisis, but it creates other distortions. Most importantly, constant Fed intervention doesn't create an incentive to avoid situations that require that intervention; if anything, it encourages more dangerous risk-taking.

The above, plus an all-too-brief history of the politics of US banking, is the meat of this book. It's a useful summary, as far as it goes, and I learned a few new things. But I think The Fed Unbound is confused about its audience.

This type of high-level summary and capsule history seems most useful for people without an economics background and who haven't been following macroeconomics closely. But Menand doesn't write for that audience. He assumes definitions of words like "deposits" and "money" that are going to be confusing or even incomprehensible to the lay reader.

For example, Menand describes ordinary retail banks as creating money, even saying that a bank loans money by simply incrementing the numbers in a customer's deposit account. This is correct in the technical economic definition of money (fractional reserve banking effectively creates new money), but it's going to sound to someone not well-versed in the terminology as if retail banks can create new dollars out of the ether. That power is, of course, reserved for the Federal Reserve, and indeed is largely the point of its existence. Much of this book relies on a very specific definition of money and money supply that will only be familiar to those with economics training.

Similarly, the history of the Federal Reserve is interesting but slight, and at no point does Menand explain clearly how the record-keeping between it and retail banks works, or what the Fed's "balance sheet" means in practice. I realize this book isn't trying to give a detailed description or history of the Federal Reserve system, but the most obvious audience is likely to flounder at the level of detail Menand provides.

Perhaps, therefore, this book is aimed at an audience already familiar with macroeconomics? But, if so, I'm not sure it says anything new. I follow macroeconomic policy moderately closely and found most of Menand's observations obvious and not very novel. There were tidbits here and there that I hadn't understood, but my time would probably have been better invested in another book. Menand proposes some reforms, but they mostly consist of "Congress should do its job and not create situations where the Federal Reserve has to act without the right infrastructure and political oversight," and, well, yes. It's hard to disagree with that, and it's also hard to see how it will ever happen. It's far too convenient to outsource such problems to central banking, where they are hidden behind financial mechanics that are incomprehensible to the average voter.

This is an important topic, but I don't think this is the book to read about it. If you want a clearer and easier-to-understand role of the Federal Reserve in shadow banking crises, read Crashed instead. If you want to learn more about how retail bank regulation works, and hear a strong case for why the same principles should be applied to shadow banks, see Sheila Bair's Bull by the Horns. I'm still looking for a great history and explainer of the Federal Reserve system as a whole, but that is not this book.

Rating: 5 out of 10

30 November, 2022 05:08AM

November 29, 2022

hackergotchi for Jonathan McDowell

Jonathan McDowell

onak 0.6.2 released

Over the weekend I released a new version of onak, my OpenPGP compatible keyserver. At 2 years since the last release that means I’ve at least managed to speed up a bit, but it’s fair to say its development isn’t a high priority for me at present.

This release is largely driven by a collection of minor fixes that have built up, and the knowledge that a Debian freeze is coming in the new year. The fixes largely revolve around the signature verification that was introduced in 0.6.0, which makes it a bit safer to run a keyserver by only accepting key material that can be validated. All of the major items I wanted to work on post 0.6.0 remain outstanding.

For the next release I’d like to get some basic Stateless OpenPGP Command Line Interface support integrated. That would then allow onak to be tested with the OpenPGP interoperability test suite, which has recently added support for verification only OpenPGP implementations.

I realise most people like to dismiss OpenPGP, and the tooling has been fairly dreadful for as long as I’ve been using it, but I do think it fills a space that no competing system has bothered to try and replicate. And that’s the web of trust, which helps provide some ability to verify keys without relying on (but also without preventing) a central authority to do so.

Anyway. Available locally or via GitHub.

0.6.2 - 27th November 2022

  • Don’t take creation time from unhashed subpackets
  • Fix ECDSA/SHA1 signature check
  • Fix handling of other signature requirement
  • Fix deletion of keys with PostgreSQL backend
  • Add support for verifying v3 signature packets

29 November, 2022 09:41PM

hackergotchi for Daniel Pocock

Daniel Pocock

Who predicted Elon Musk’s Twitter acquisition in 2018?

In 2018, attending the UN forum on business and human rights, I blogged a photo of the Afghan president and made some brief comments about the possibility of Twitter falling into the wrong hands.

The full video is available from UN Web TV. There is an extract below where Bennett Freeman of the Global Network Initiative gives some interesting responses to my comments.

Here is the former president of Afghanistan at Palais des Nations in 2018:

29 November, 2022 09:00PM

hackergotchi for Norbert Preining

Norbert Preining

FFmpeg – tips and tricks

Over the years, and in particular with preparing videos of a conference for publication on YouTube, I have accumulated a few reminders how to do things with FFmpeg. It is such a fantastic beast full of features, that I will never run out of new things to learn.
(Update 2022-11-30: Found that excellent lengthy intro: FFmpeg – The Ultimate Guide)

So here we go, in some rather random order:

split at time stamps

Basic usage:

ffmpeg -i INPUT -ss STARTTS -to ENDTS -c copy OUTFILE

WARNING: since splitting is only possible at keyframes, this will by default set an “editlist” and start at the previous keyframe, and tell the player to start playing after the required difference.

Now that is a pain, because watermarking (see below) requires
--ignore_editlist 1
and then the audio gets out of sync!!!!

In this case one need to do the splitting already in addition to recoding and resizing so that it splits at the exact time!!!

concat two mp4 files

ffmpeg -i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i input2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate2.ts
ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy -bsf:a aac_adtstoasc output.mp4

see Concatenation of files with same codecs, at concat protcocol

cut out a part from STARTCUT – ENDCUT

This is done by creating two intermediate files by rendering only the initial or end part and at the same time encoding to mpeg-2 transport streams:

ffmpeg -i INPUT -ss 00:00:00 -to STARTCUT -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i INPUT -ss ENDCUT -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy -bsf:a aac_adtstoasc output.mp4

concat different formats

ffmpeg -i file1.mp4 -i file2.mp4 -i file3.mp4 \
  -filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]concat=n=3:v=1:a=1[outv][outa]"	\
  -map "[outv]" -map "[outa]" output.mp4

see here.


ffmpeg -i INPUT.mp4 -i TUG-33.png -filter_complex "overlay=x=(main_w-overlay_w-30):y=(main_h-overlay_h-30)" OUTPUT.mp4

If there is an error like

Could not find codec parameters for stream 0 ...
unspecified pixel format

one needs to add -ignore_editlist 1 which in turn requires proper cutting, see above.

(if necessary, add -analyzeduration 100M -probesize 100M)

simple scaling

ffmpeg -i input.avi -vf scale=320:240 output.avi

scale and watermark

ffmpeg -ignore_editlist 1 -i INPUT -i TUG-33.png -filter_complex "[0:v]scale=1920:1080 [mnn], [mnn][1:v]overlay=x=(main_w-overlay_w-30):y=(main_h-overlay_h-30)" OUTPUT

You can use the -ss option to specify a start timestamp, and the -t option to specify the encoding duration. The timestamps need to be in HH:MM:SS.xxx format or in seconds (s.msec).

The following would clip the first 30 seconds, and then clip everything that is 10 seconds after that:

ffmpeg -ss 00:00:30.0 -i input.wmv -c copy -t 00:00:10.0 output.wmv
ffmpeg -ss 30 -i input.wmv -c copy -t 10 output.wmv

Note that -t is an output option and always needs to be specified after -i.

Some tips:

For older ffmpeg versions, if you use -ss after -i, you get more accurate seeking at the expense of a slower execution altogether. See also: Seeking with FFmpeg. You can use -to instead of -t to specify the timestamp to which you want to cut. So, instead of -i -ss 30 -t 10 you could also do -i -ss 30 -to 40 to achieve the same thing.

If your ffmpeg does not support -c, or -to, it is likely very outdated. Compile a new version yourself or download a static build from their homepage. It’s really not complicated.

still image (screenshot) from video

Trivial solution would be

ffmpeg -y -ss 00:$i -i $INPUT -frames:v 1 -q:v 2 ${OUTPUT}.jpg

but that does not keep aspect ratio (DAR Display Aspect Ratio). To get correct output:

ffmpeg -y -ss 00:$i -i $INPUT -vf scale='trunc(ih*dar):ih',setsar=1/1 -frames:v 1 -q:v 2 ${OUTPUT}.jpg

29 November, 2022 02:11PM by Norbert Preining

November 28, 2022

Jelmer Vernooij

Detecting Package Transitions

Larger transitions in Debian are usually announced on e.g. debian-devel, but it’s harder to track the current status of all transitions. Having done a lot of QA uploads recently, I have on occasion uploaded packages involved in a transition. This can be unhelpful for the people handling the transition, but there’s also often not much point in uploading if your uploads are going to get stuck.

Talking to one of the release managers at a recent BSP, it was great to find out that the release team actually publish a data dump with which packages are involved in which transitions.

Here’s the script I use to find out about the transitions the package in my current working directory is involved in:


from urllib.request import urlopen

import sys

from debian.deb822 import Deb822
import yaml

with open('debian/control', 'r') as f:
    package = Deb822(f)['Source']

with urlopen("https://release.debian.org/transitions/export/packages.yaml") as f:
    data = yaml.safe_load(f)

def find_transitions(data, package):
    for entry in data:
        if entry['name'] != package:
        return dict(entry['list'])
    return {}

transitions = find_transitions(data, package)
sys.exit(1 if 'ongoing' in transitions.values() else 0)

In practice, the output looks something like this:

$ debcheckout bctoolbox
git clone https://salsa.debian.org/pkg-voip-team/linphone-stack/bctoolbox.git bctoolbox ...
Cloning into 'bctoolbox'...
$ cd bctoolbox
$ in-transition.py
{'auto-upperlimit-libbctoolbox1': 'ongoing'}

28 November, 2022 11:04PM by Jelmer Vernooij

hackergotchi for Daniel Pocock

Daniel Pocock

Debian aggression: woman asked about her profession

In my last blog about the DebConf6 assault, I looked at the way the entire DebConf6 population was fooled to believe a dentist was a prostitute.

As it turns out, at least one person felt it was acceptable to walk up to this female dentist and simply ask her if she was a paid dinner date.

The email below also demonstrates the manner in which at least one member of the women's clique, Amaya Rodrigo, was trying to convince Ted's date that Ted might be dangerous. This is a hideous example of backstabbing and sabotage.

While this was 16 years ago, Debian hasn't changed. We can see how easy it was to hoodwink developers with the false accusations that Jacob Appelbaum is a rapist. In 2019, at FOSDEM, the OSI president used her speech to promote a "whisper network". Amaya's email shows us what the "whisper network" really means, soul-destroying tactics deployed for purposes of wrecking a man's personal life.

If people are so gullible and if an organization has no checks and balances in grievance procedures then the organization is putty in the hands of manipulators. These cultural flaws have serious repercussions for the quality and the security of the Debian operating system.

Hilda Rivera Ramirez, Ted Jonathan Walther, DebConf6
Subject: Re: Ted Walther's Expulsion
Date: Wed, 24 May 2006 14:03:49 +0200
From: Amaya Rodrigo <amaya@debian.org>
Organization: Debian - http://www.debian.org/
To: debian-private@lists.debian.org

Hi all,

Gunnar, I have been giving this whole thing a lot of thought, and I
guess you somehow expect an answer from me. I have skipped the rest of
recipients for my own karma and sanity reasons. 
Gunnar Wolf wrote:
> [ Note that I'm Cc:ing Amaya on this reply, as I know she does not
>   read -private 

I do read private, I wonder where you got that impression from :)

>   Amaya, please share my text with those other people, as it is
>   important to me that my close personal friends involved in this get
>   my appreciation on the matter. ]


> But I did get quite angry at, and I do condemn, the treatment of the
> situation during the formal dinner. 

I am not particularly proud of my own actions that night. Especially of
the way he managed to bring out the worst in me, the absolutely worst.
Even to the point of irrationality that made me comfront people I call
friends, people I admire, respect and love, like yourself, mooch or
> The woman who came with Ted was _believed_ to be a prostitute, because
> nobody thought he would have had the time to make friends with locals,
> and nobody took the time to get the hard facts straight. 

I did. I asked your friend Fabio in the most straight manner that I
could and got what I absolutely believed was a positive answer. He later
denied everything, which also fustrated and angried me, even when I had
asked him to warn this lady about what I though was a dangerous
customer, and I was quite explicit to him, and he didn't make any intend
to correct my understanding of the situation, only later did he deny
> Some people say this lady was frontally asked whether she was a
> prostitute, question to which she answered affirmatively. Of course,
> given the huge lack of respect to her and to the only person she knew
> in the crowd. 

I talked to the lady very briefly twice, never mentioning the word
prostitute or making any insinuations at all. I also would have found
extremely disrespectful to do so. Whatever her income comes from, she is
a human being to me with full dignity.

The first time i talked to her I explained to her that what was going on
had nothing to do with her, that it was a problem with Ted and that I
believed Ted was a dangerous person and that she should be careful. 
The second time I repeated that it was nothing against her and that she
in fact was very welcome to share our dinner and dance afterwards, that
the issue was exclusively with Ted. 
Both times I talked to her with respect, I especially din't want to
instult her, on contrary. I want to comment on her reaction to me,
though. She looked at me, then at Fabio, then slightly pushed me away
and said something like (I can't recall exactly) "I can't talk to you" or
"I am not allowed to talk to you", which of course only contributed to
make me feel worse, as I read the whole situation as she was not even
there on her own free will.

Of course I think you, Gunnar, are aware that I didn't like Fabio at
all, for reasons that I can clarify is there is any interest. I know he
is your friend, and I am willing to talk about this with you in private.

> I was far too disappointed with the irrational attitude of my friends.

Yes, he managed to bring the worst, most irrational, airated /me I have
been in my life. Am I proud about it? Sure not.

> Next time something insults any of us (because it will happen, that's
> a fact), please try to use your head before it gets to the boiling
> temperature.

Sure, Gunnar. I have learnt a couple of things out of this whole
situation. I think we all have. I expect to never have to face such a
situation in my life, but if I have to, there's a couple of skills I
have learned.

My intention is not to revive this thread, just to clarify that I
treated the lady with respect. 
If anybody has further questions, I am happy (while absolutely unhappy)
to answer. I think the questions can be addressed directly to me. Let's
kill this thread.

If possible at all, I would be happy if this message is not published
when debian-private's archive is made public.

 .''`.   Sometimes the littlest feet leave the biggest impressions : :' :   `. `'           Proudly running unstable Debian GNU/Linux
  `-     www.amayita.com  www.malapecora.com  www.chicasduras.com

28 November, 2022 06:30PM

hackergotchi for Steve Kemp

Steve Kemp

I put an LSP in your LISP ..

I recently wrote about yet another lisp I'd been having fun with.

Over the past couple of years I've played with a few toy scripting languages, or random interpreters, and this time I figured I'd do something beyond the minimum, by implementing the Language Server Protocol.

In brief the language server protocol (LSP) is designed to abstract functionality that might be provided by an editor, or IDE, into a small "language server". If the language-server knows how to jump to definitions, provide completion, etc, etc, then the editor doesn't need to implement those things for NN different languages - it just needs to launch and communicate with something that does know how to do the job.

Anyway LSP? LISP? Only one letter different, so that's practically enough reason to have a stab at it.

Thankfully I found a beautiful library that implements a simple framework allowing the easy implementation of a golang-based LSP-serverÖ

Using that I quickly hacked up a server that can provide:

  • Overview of all standard-library functions, on hover.
  • Completion of all standard-library functions.

I've tested this in both GNU Emacs and Neovim, so that means I'm happy I support all editors! (More seriously if it works in two then that probably means that the LSP stuff should work elsewhere too.)

Here's what the "help on hover" looks like, within Emacs:

Vim looks similar but you have to press K to see the wee popup. Still kinda cute, and was a good experiment.

28 November, 2022 05:09PM

John Goerzen

Flying Joy

Wisdom from my 5-year-old: When flying in a small plane, it is important to give your dolls a headset and let them see out the window, too!

Moments like this make me smile at being a pilot dad.

A week ago, I also got to give 8 children and one adult their first ever ride in any kind of airplane, through EAA’s Young Eagles program. I got to hear several say, “Oh wow! It’s SO beautiful!” “Look at all the little houses!”

And my favorite: “How can I be a pilot?”

28 November, 2022 04:04PM by John Goerzen

November 26, 2022

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

The Financial Times has been printing an obvious error on its “Market Data” page for 18 months and nobody else seems to have noticed

Market Data section of the Financial Times US Edition print edition from May 5, 2021.

If you’ve flipped through printed broadsheet newspapers, you’ve probably seen pages full of tiny text listing prices and other market information for stocks and commodities. And you’ve almost certainly just turned the page. Anybody interested in this market prices today will turn to the internet where these numbers are available in real time and where you don’t need to squint to find what you need. This is presumably why many newspapers have stopped printing these types of pages or dramatically reduced the space devoted to them. Major financial newspapers however—like the Financial Times (FT)—still print multiple pages of market data daily. But does anybody read them?

The answer appears to be “no.” How do I know? I noticed an error in the FT‘s “Market Data” page that anybody looking in the relevant section of the page would have seen. And I have watched it been reproduced every single day for the last 18 months.

In early May last year, I noticed that the the Japanese telecom giant Nippon Telegraph and Telephon (NTT) was listed twice on the FT‘s list of the 500 largest global companies: once as “Nippon T&T” and also as “Nippon TT.” One right above the other. All the numbers are identical. Clearly a mistake.

Reproduction of the FT Market Data section showing a subset of Japanese companies from the FT 500 list of global companies. The duplicate lines are highlighted in yellow. This page is from today’s paper (November 26, 2022).

Wondering if it was a one-off error, I looked at a copy of the paper from about a week before and saw that the error did not exist then. I looked a copy from one day before and saw that it did. Since the issue was apparently recurring, but new at the time, I figured someone at the paper would notice and fix it quickly. I was wrong. It has been 18 months now and the error has been reproduced every single day.

Looking through the archives, it seems that the first day the error showed up was May 5, 2021. I’ve included a screenshot from the electronic paper version from that day—and from the fifth of every month since then (or the sixth if the paper was not printed on the fifth)—that shows that the error is reproduced every day. A quick look in the archives suggests it not only appears in the US edition but also in the UK, European, Asian, and Middle East editions. All of them.

Why does this matter? The FT prints over 112,000 copies of its paper, six days a week. This duplicate line takes up almost no space, of course, so it’s not a big deal on its own. But devoting two full broadsheet pages to market data that is out date as soon as it is printed—much of which nobody appears to be reading—doesn’t seem like a great use of resources. There’s an argument to made that papers like the FT print these these pages not because they are useful but because doing so is a signal of the publications’ identities as serious financial papers. But that hardly seems like a good enough reason on its own if nobody is looking at them. It seems well past time for newspapers to stop wasting paper and ink on these pages.

I respect that some people think that printing paper newspapers at all is wasteful when one can just read the material online. Plenty of people disagree, of course. But who will disagree with a call to stop printing material that evidence suggests is not being seen by anybody? If an error this obvious can exist for so long, it seems clear that nobody—not even anybody at the FT itself—is reading it.

26 November, 2022 08:37PM by Benjamin Mako Hill

hackergotchi for Matthew Garrett

Matthew Garrett

Poking a mobile hotspot

I've been playing with an Orbic Speed, a relatively outdated device that only speaks LTE Cat 4, but the towers I can see from here are, uh, not well provisioned so throughput really isn't a concern (and refurbs are $18, so). As usual I'm pretty terrible at just buying devices and using them for their intended purpose, and in this case it has the irritating behaviour that if there's a power cut and the battery runs out it doesn't boot again when power returns, so here's what I've learned so far.

First, it's clearly running Linux (nmap indicates that, as do the headers from the built-in webserver). The login page for the web interface has some text reading "Open Source Notice" that highlights when you move the mouse over it, but that's it - there's code to make the text light up, but it's not actually a link. There's no exposed license notices at all, although there is a copy on the filesystem that doesn't seem to be reachable from anywhere. The notice tells you to email them to receive source code, but doesn't actually provide an email address.

Still! Let's see what else we can figure out. There's no open ports other than the web server, but there is an update utility that includes some interesting components. First, there's a copy of adb, the Android Debug Bridge. That doesn't mean the device is running Android, it's common for embedded devices from various vendors to use a bunch of Android infrastructure (including the bootloader) while having a non-Android userland on top. But this is still slightly surprising, because the device isn't exposing an adb interface over USB. There's also drivers for various Qualcomm endpoints that are, again, not exposed. Running the utility under Windows while the modem is connected results in the modem rebooting and Windows talking about new hardware being detected, and watching the device manager shows a bunch of COM ports being detected and bound by Qualcomm drivers. So, what's it doing?

Sticking the utility into Ghidra and looking for strings that correspond to the output that the tool conveniently leaves in the logs subdirectory shows that after finding a device it calls vendor_device_send_cmd(). This is implemented in a copy of libusb-win32 that, again, has no offer for source code. But it's also easy to drop that into Ghidra and discover thatn vendor_device_send_cmd() is just a wrapper for usb_control_msg(dev,0xc0,0xa0,0,0,NULL,0,1000);. Sending that from Linux results in the device rebooting and suddenly exposing some more USB endpoints, including a functional adb interface. Although, annoyingly, the rndis interface that enables USB tethering via the modem is now missing.

Unfortunately the adb user is unprivileged, but most files on the system are world-readable. data/logs/atfwd.log is especially interesting. This modem has an application processor built into the modem chipset itself, and while the modem implements the Hayes Command Set there's also a mechanism for userland to register that certain AT commands should be pushed up to userland. These are handled by the atfwd_daemon that runs as root, and conveniently logs everything it's up to. This includes having logged all the communications executed when the update tool was run earlier, so let's dig into that.

The system sends a bunch of AT+SYSCMD= commands, each of which is in the form of echo (stuff) >>/usrdata/sec/chipid. Once that's all done, it sends AT+CHIPID, receives a response of CHIPID:PASS, and then AT+SER=3,1, at which point the modem reboots back into the normal mode - adb is gone, but rndis is back. But the logs also reveal that between the CHIPID request and the response is a security check that involves RSA. The logs on the client side show that the text being written to the chipid file is a single block of base64 encoded data. Decoding it just gives apparently random binary. Heading back to Ghidra shows that atfwd_daemon is reading the chipid file and then decrypting it with an RSA key. The key is obtained by calling a series of functions, each of which returns a long base64-encoded string. Decoding each of these gives 1028 bytes of high entropy data, which is then passed to another function that decrypts it using AES CBC using a key of 000102030405060708090a0b0c0d0e0f and an initialization vector of all 0s. This is somewhat weird, since there's 1028 bytes of data and 128 bit AES works on blocks of 16 bytes. The behaviour of OpenSSL is apparently to just pad the data out to a multiple of 16 bytes, but that implies that we're going to end up with a block of garbage at the end. It turns out not to matter - despite the fact that we decrypt 1028 bytes of input only the first 200 bytes mean anything, with the rest just being garbage. Concatenating all of that together gives us a PKCS#8 private key blob in PEM format. Which means we have not only the private key, but also the public key.

So, what's in the encrypted data, and where did it come from in the first place? It turns out to be a JSON blob that contains the IMEI and the serial number of the modem. This is information that can be read from the modem in the first place, so it's not secret. The modem decrypts it, compares the values in the blob to its own values, and if they match sets a flag indicating that validation has succeeeded. But what encrypted it in the first place? It turns out that the json blob is just POSTed to http://pro.w.ifelman.com/api/encrypt and an encrypted blob returned. Of course, the fact that it's being encrypted on the server with the public key and sent to the modem that decrypted with the private key means that having access to the modem gives us the public key as well, which means we can just encrypt our own blobs.

What does that buy us? Digging through the code shows the only case that it seems to matter is when parsing the AT+SER command. The first argument to this is the serial mode to transition to, and the second is whether this should be a temporary transition or a permanent one. Once parsed, these arguments are passed to /sbin/usb/compositions/switch_usb which just writes the mode out to /usrdata/mode.cfg (if permanent) or /usrdata/mode_tmp.cfg (if temporary). On boot, /data/usb/boot_hsusb_composition reads the number from this file and chooses which USB profile to apply. This requires no special permissions, except if the number is 3 - if so, the RSA verification has to be performed first. This is somewhat strange, since mode 9 gives the same rndis functionality as mode 3, but also still leaves the debug and diagnostic interfaces enabled.

So what's the point of all of this? I'm honestly not sure! It doesn't seem like any sort of effective enforcement mechanism (even ignoring the fact that you can just create your own blobs, if you change the IMEI on the device somehow, you can just POST the new values to the server and get back a new blob), so the best I've been able to come up with is to ensure that there's some mapping between IMEI and serial number before the device can be transitioned into production mode during manufacturing.

But, uh, we can just ignore all of this anyway. Remember that AT+SYSCMD= stuff that was writing the data to /usrdata/sec/chipid in the first place? Anything that's passed to AT+SYSCMD is just executed as root. Which means we can just write a new value (including 3) to /usrdata/mode.cfg in the first place, without needing to jump through any of these hoops. Which also means we can just adb push a shell onto there and then use the AT interface to make it suid root, which avoids needing to figure out how to exploit any of the bugs that are just sitting there given it's running a 3.18.48 kernel.

Anyway, I've now got a modem that's got working USB tethering and also exposes a working adb interface, and I've got root on it. Which let me dump the bootloader and discover that it implements fastboot and has an oem off-mode-charge command which solves the problem I wanted to solve of having the device boot when it gets power again. Unfortunately I still need to get into fastboot mode. I haven't found a way to do it through software (adb reboot bootloader doesn't do anything), but this post suggests it's just a matter of grounding a test pad, at which point I should just be able to run fastboot oem off-mode-charge and it'll be all set. But that's a job for tomorrow.

Edit: Got into fastboot mode and ran fastboot oem off-mode-charge 0 but sadly it doesn't actually do anything, so I guess next is going to involve patching the bootloader binary. Since it's signed with a cert titled "General Use Test Key (for testing only)" it apparently doesn't have secure boot enabled, so this should be easy enough.

comment count unavailable comments

26 November, 2022 07:09AM

November 25, 2022

Russ Allbery

podlators 5.00

podlators is my collection of POD formatting modules, which generate *roff or text (possibly with escape sequence markup) from the documentation format used by Perl and some other packages.

This is a major release, the biggest since the Pod::Simple rewrite in 2005. The headline news is that after some fairly extensive investigation, this release of Pod::Man finally changes the default output format to Unicode. No more replacement of characters in people's names, or text in non-English languages, with ugly X characters! There is a new encoding option to set the output encoding, and new options groff (which uses the groff extension for Unicode code points and is the default on EBCDIC systmes) and roff (which does the old, broken X substitution).

Since this was a major backward-incompatible change, I also finally removed most of the formatting touch-ups that Pod::Man tried to do for troff output but which would be invisible for the (by far more commonly used) nroff output. These have been an endless source of bugs and are very difficult to maintain, most of them were of marginal utility, and I am dubious many people are using troff to print Perl manual pages these days instead of, say, printing the rendered output from one of the many excellent POD to HTML modules.

There is some remaining somewhat-Perl-specific guesswork applied to the formatting, which is much simpler, but even that can now be turned off with the new guesswork option. This will allow people using POD to generate manual pages for things other than Perl modules to disable the Perl-specific markup logic.

Pod::Text also now supports encoding and gets some major encoding cleanups, including using Encode instead of PerlIO encoding layers for its output.

There are also numerous other fixes and improvements: a new language option to Pod::Man to configure (in an unfortunately groff-specific way) the line-breaking rules for languages like Chinese and Japanese, conversion of zero-width spaces to the *roff \: equivalent, a fix for wrapping L<> inside S<>, and various other bug fixes.

Perhaps the most interesting is a fix to a long-standing problem with the Pod::Man output where bold and italic text would extend too far if used in combination with C<> fixed-width text. This bug has been around forever without being noticed, and then two different people noticed it while I was preparing this release.

You can get the latest release from CPAN or from the podlators distribution page. These changes should be incorporated into Perl core in due course, although given the substantial changes, that may require a baking period.

25 November, 2022 10:39PM

hackergotchi for Daniel Pocock

Daniel Pocock

Violence, sexism, racist harassment and physical abuse at FOSDEM, DebConf, FrOSCon, Debian, OSI

FOSDEM organization is getting under way and volunteers have already started receiving anonymous threats and insults. This is not unprecedented and it is no surprise.

At FOSDEM itself, the former president of Open Source Initiative showed the infamous slide with a cat behind bars. The woman is not a developer. She does not have money to pay developers. People don't consent to be in these experiments. Therefore, the picture implies some force is used to impose upon developers against our will. What she has illustrated here is a concentration camp. If she displayed this slide in Germany she could be prosecuted for glorifying the holocaust.

Molly de Blanc, cat behind bars

The same woman went to FrOSCon in Germany and displayed a slide showing three users pushing a developer. Yet this has actually happened. This was the infamous sexist/racist attack on Ted (Jonathan) Walther and his Mexican dinner date at DebConf6 in Oaxtepec, Mexico. Having seen evidence that rogue elements of the community are planning a repeat of the violence for FOSDEM 2023, it is time to dig out the messages from debian-private. It is 15 years ago but looking at the slides about pushing, not much has changed.

Here is the infamous pushing video from FrOSCon:

Molly de Blanc: Well we can use our collective power to push others

(full video from FrOSCon 2019)

Many Debian scandals, like the Debian Day Suicide, have been kept hidden for years but the DebConf6 violence quickly became public knowledge. There is a large thread in Slashdot where John Sokol gives a first-hand account:

This group, after spreading many rumors about Walther that were totally false, one of which that his date, Hilda (a local entrepreneur in a very small town where everyone knows everyone and runs a dental administration company) was accused of being a prostitute, and this was because Ted couldn't possible get a woman on his own. Being that I was there when Ted met her at the local Internet cafe, this vicious rumor is definitely not true.

After rumors failed, 7 guys tried to rush him and became quite agitated and violent. I am a fairly big guy and happen to be standing in the doorway at the time of the attack against Ted.

Several people where behind him and they would have pushed Walther and several innocent bystanders over the 2 foot ledge that lead to the street, probably injuring some of them.

The attack on Ted Walther sounds a lot like the culture that the OSI president has depicted in the slide at FrOSCon (see the video above).

Looking at the prostitute insult, the victim is a woman and she is from Mexico. Therefore, this is a combination of both sexism and racism. Hilda Rivera Ramirez runs a local dental clinic.

Hilda Rivera Ramirez, Ted Jonathan Walther, DebConf6

How incredibly racist and sexist one must be to confuse a dentist with a prostitute. No doubt jealousy is a factor: Walther had the courage to ask the woman on a date while other developers have spent the whole week cocooned in their DebConf bubble.

Fast forward to DebConf15 and the same rudeness was turned on Carla and I even before we got there. We were going to pay for Carla's flights ourselves but the Google woman told me Carla wouldn't be welcome at the meals.

Now we can fast forward to 2018 and see what happened when the Albanian female interns asked if somebody could purchase their flights for them. Male interns were told to purchase their own tickets and wait months for a reimbursement. Albanian female interns received payment up-front:

Subject: Re: [rt.debian.org #7328] DebConf travel pre-payment requests
From: Martin Michlmayr
Time: Fri Jun 29 08:56:42 2018

* Hector Oron [2018-06-28 10:55]:
> I added Martin to the list, he'll be taking care of flight ticket
> purchase if you send him flight details.

This has been taken care of.

Martin Michlmayr

Here are a couple of messages about the Mexican dentist. Notice that one of the suspects in the DebConf6 violence, Holger Levsen, is in some of the pictures with the table full of Albanian women at DebConf19.

DebConf19, Albanian women, Holger Levsen, Chris Lamb

Notice that in the report from Debian leader Anthony Towns, there is evidence of bias, racism and sexism. For example, using the word "lady" instead of "girl" or "woman" encourages the reader to see the woman in a particular way.

Subject: Events at the DebConf Dinner
Date: Fri, 19 May 2006 10:41:19 -0500
From: Anthony Towns <aj@azure.humbug.org.au>
To: debian-private@lists.debian.org
CC: ted@reactor-core.org


As promised, details from the dinner:

Ted was seen walking with his "clue" bat and a crown out of the conference
venue, avoiding the busses provided to attend the dinner. When asked
later, he indicated he arrived by car with one of the conference sponsors,
named Fabio, and a lady, named Hilda. It is worth noting at this point,
that Ted had not communicated with the conference organisers to ensure
that Hilda would be able to be served, as she had not otherwise been an
attendee at DebConf.

Upon arriving, Ted was wearing his crown and carrying his "clue" bat. One
of the attendees snuck up behind him, and grabbed the crown, and gave it
to me. Ted then came over, at which point I tried grabbing the "clue"
bat and hitting him with it, with limited success. He indicated the
crown was the wrong way around, and we posed for some photos from far
too many cameras, which you may well have already seen.

Some attendees who had had prior bad experiences with Ted, such as
described in the previous mail, were disturbed by him receiving the
attention of the entire conference, and at some point made jests of the
form that Hilda might be his nurse or a social worker. As I understand it,
Amaya Rodrigo Sastre asked Fabio who the lady was, and was led to believe
that she was a prostitute. She and others found that uncomfortable and
offensive, and were also concerned for the woman's safety after his
blog post from the previous evening that spoke extensively about the
death penalty for prostitutes and that being a religious obligation in
certain cases.

One of the volunteers, Ana Guerrero, spoke to Hilda and suggested that
it might be a good idea for her to leave. I am not sure what was said,
but I believe it was related to the general concerns about Ted's religious
beliefs, and the question of whether she was or was not a prostitute was
not raised. As I understand it, at this point Hilda did leave, however
was encouraged to return by Ted and his long-time friend and co-worker,
John Sokol, a former developer of 386BSD and an attendee at DebConf.

At some point, I believe shortly after she returned, some attendees then
decided that it would be all right if she stayed, but Ted left. At this
point Holger and Moray, as mentioned above, manhandled Ted across the
dining hall to the door, where they were intercepted by John. At this
point the rest of the attendees noticed that something was going on, and a
group gathered around trying to calm the situation down, and find out what
had caused it. As part of this, Martin Krafft attempted to calm down the
situation at which point Ted told him to "Get off me you fucking Nazi".

By this point the rain had reached the point where the sound of it hitting
the tin roof was enough to drown out the mariachi band, and without having
known the details of what was going on, it seemed to me that throwing
people out into the rain was uncalled for. As such I escorted Ted and
Hilda back to their seats to separate them from the other attendees who
were clearly angry, and tried to determine what was going on. At about
this time, Amaya and Fabio had a heated argument in Spanish, which Amaya
later indicated involved Fabio claiming that Hilda was not a prostitute,
and that he had never said anything that should give that impression.

I also spoke with Moray and Holger, who by this time were talking with
Bdale and some other developers who hadn't been involved, and indicated
that grabbing people to throw them out of the room was entirely wrong. At
this point they both remained extremely angry at the situation, and had
very little to say. I don't believe resorting to violence is remotely
appropriate within Debian, and had the opportunity to speak further about
this with Moray this morning, and hope to have a similar opportunity to
speak with Holger shortly.

In any case, Amaya, Moray, Holger, and a number of others decided they
would prefer to leave the dinner at this point rather than remain there
with Ted, rain or not; at approximately which time the lighting failed
entirely due to the rain, and presumably leaks near the electrical
system. They remained at the dining hall for a little while longer
until the rain eased, eventually leaving for dinner along with a local
developer, with the intention of finding their own way home.

For the remainder of the evening Ted and Hilda remained mostly in their
seats, and I tried to ensure he avoided further aggravating the other

Not long after most people had eaten, the organisers decided to cut
our losses and arrange for the busses to transport people back to the
conference venue. At this point, most of the people most offended by Ted
took the opportunity to leave along with others who wished to leave early,
as I understand it, slightly over-filling two busses. Ted also left to
leave on these busses, along with Hilda, which I suggested would not
be a good idea, and that it might be better to take a taxi. John Sokol
loaned Ted some money at this point, and Ted and Hilda left.

By the time I returned, Ted was sitting in the middle of the hacklab. As
I understand it, there were a small number of further confrontations as a
result of this.

It's not entirely clear what the facts of the matter are; and as you might
expect, a number of rumours flew around during the dinner. The above
is my best understanding of what happened. 
Obviously there were a large number of other developers present at this
event, who should be able to provide additional information on the dinner;
but it is important to realise that it was not this event alone that
led to either the loss of tempers involved, nor this expulsion.


Anthony Towns
Debian Project Leader

Reactions from other Debian Developers

Josip Rodin appears to endorse violence:

Subject: Re: Events at the DebConf Dinner
Date: Thu, 1 Jun 2006 18:39:15 +0100
From: Ian Jackson 
To: debian-private@lists.debian.org
Newsgroups: chiark.private.mail.debian.private
References: <20060519154119.GA4372@azure.humbug.org.au> <17524.43309.244563.319981@davenant.relativity.greenend.org.uk> <20060526222120.GA16999@keid.carnet.hr>

Josip Rodin writes ("Re: Events at the DebConf Dinner"):
> I went back and read a few of the krooger-induced flamewars again, those
> from previous years... My conclusion is that he was fairly lucky to be
> on the receiving end of only a single incident of manhandling. FWIW.

Debian is not supposed to be some downtown bar where if you insult
someone you can expect to be beaten.

Most of the other communities I'm involved with take a very strong
line against violence.  For example, if someone looks like they're
going to get physical, their friends will intervene.  Now we don't
perhaps all know each other as well as we do our meatlife friends; but
I would hope that if I was at Debconf and threatened to behave as
described, other developers would step in to dissuade me.  I also hope
that if I did do anything along these lines the project would hold me
to account and demand, at the very least, an apology.

We have ways of resolving these disputes that do not involve
individuals taking things into their own hands.  One problem we have
in Debian is that those people in charge have usually been reluctant
to enforce justifiable social norms of behaviour, both online and when
we meet in person.  Tolerating violence amongst our members is just
one example of this.


Wouter doesn't appear to realize that violence is a crime:

Subject: Re: Events at the DebConf Dinner
Date: Sat, 03 Jun 2006 08:54:12 -0700
From: Bruce Perens 
To: Wouter Verhelst 
CC: debian-private@lists.debian.org

Wouter Verhelst wrote:
> How is manhandling or insult going to increase SPI's chances of getting sued?

The manhandling is the sort of crime that judges take seriously.


Why did the victim get expelled?

It looks like members of the mob realized they may face consequences so rather than apologizing, they expelled the victim and muddied the waters.

Subject: Re: Ted Walther's Expulsion
Date: Sun, 21 May 2006 14:52:05 -0700
From: Steve Langasek 
To: debian-private@lists.debian.org

I'm sorry, I don't want to prolong this thread, but I'm having difficulty
parsing your mail.

On Sat, May 20, 2006 at 12:52:47AM -0400, Bruno Barrera C. wrote:

> Hanging on the last point, there was a problem calling the Ted's guest
> 'Prostitute'. That's not a rumor, that's a fact.

What part are you asserting is a fact here?  That she was a prostitute, that
people called her a prostitute, or that it was a problem to call her a

> It was what I heard (from many people, obviously rumor moving along the
> crowd) when I was on the formal dinner.  The real rumor is that she wasn't
> a prostitute,

What do you mean by "real rumor"?  A rumor is a claim of doubtful veracity;
are you claiming that the *truth* is that she was not a prostitute, or that
the *rumor* that she was a prostitute is a rumor, thus implying that it is
false and the contrary is true (that she was a prostitute)?

> because I'm witness that they met her close the town because she was close
> to a business man that they met a few days ago, and Ted kindly offered her
> to join him to the Formal dinner.

I regret not having accurate first-hand information about this situation.  I
wish I did, because I would like to better understand what Went Wrong 
at the formal dinner.  It is pretty clear that Ted & Co. did not do much
initially to *discourage* the impression that Hilda was a paid escort,
contributing to the confrontation.


The conclusion: if you bring a female friend to a free/open source software event, you have to give every other person a verbose explanation about her profession and how you met.

25 November, 2022 07:30PM

November 24, 2022

Russ Allbery

Review: Servant Mage

Review: Servant Mage, by Kate Elliott

Publisher: Tordotcom
Copyright: 2022
ISBN: 1-250-76904-3
Format: Kindle
Pages: 165

Servant Mage is a (so far at least) standalone fantasy novella.

Fellian is a servant mage, which means that she makes Lamps for the customers of the inn, cleans the privies, and watches her effective owners find ways to put her deeper into indentured servitude. That's the only life permitted by the August Protector to those found to carry the taint of magical talent, caused by (it is said) demons who have bound themselves to their souls. Fellian's effective resistance is limited to giving covert reading lessons. Or was, before she is hired by a handsome man who needs a Lamplighter. A man who has been looking for her specifically, is a magical Adept, and looks suspiciously like a soldier for the despised and overthrown monarchists.

A difficulty with writing a story that reverses cliches is that you have to establish the cliches in order to reverse them, which runs the risk that a lot of the story may feel cliched. I fear some of that happened here.

Magic, in this world, is divided into elemental spheres, each of which has been restricted to limited and practical tasks by the Liberationists. The new regime searches the population for the mage-gifted and forces them into public service for the good of all (or at least that's how they describe it), with as little education as possible. Fellian was taught to light Lamps, but what she has is fire magic, and she's worked out some additional techniques on her own. The Adept is air, and one of the soldiers with him is earth. If you're guessing that two more elements turn up shortly and something important happens if you get all five of them together, you're perhaps sensing a bit of unoriginality in the magic system.

That's not the cliche that's the primary target of this story, though. The current rulers of this country, led by the austere August Protector, are dictatorial anti-monarchists who are happy to force mages into indenture and deny people schooling. Fellian has indeed fallen in with the monarchists, who unsurprisingly are attempting to reverse the revolution. They, unlike the Liberationists, respect mages and are willing to train them, and they would like to recruit Fellian.

I won't spoil the details of where Elliott is going with the plot, but it does eventually go somewhere refreshingly different. The path to get there, though, is familiar from any number of fantasy epics that start with a slave with special powers. Servant Mage is more aware of this than most, and Fellian is sharp-tongued and suspicious rather than innocent and trainable, but there are a lot of familiar character tropes and generic fantasy politics.

This is the second story (along with the Spiritwalker trilogy) of Elliott's I've read that uses the French Revolution as a political model but fails to provide satisfying political depth. This one is a novella and can therefore be forgiven for not having the time to dive into revolutionary politics, but I wish Elliott would do more with this idea. Here, the anti-monarchists are straight-up villains, and while that's partly setup for more nuance than you might be expecting, it still felt like a waste of the setting. I want the book that tackles the hard social problem of reconciling the chaos and hopefulness of political revolution with magical powers that can be dangerous and oppressive without the structure of tradition. It feels like Elliott keeps edging towards that book but hasn't found the right hook to write it.

Instead, we get a solid main character in Fellian, a bunch of supporting characters who mostly register as "okay," some magical set pieces that have all the right elements and yet didn't grab my sense of wonder, and a story that seemed heavily signposted. The conclusion was the best part of the story, but by the time we got there it wasn't as much of a surprise as I wanted it to be. I had this feeling with the Spiritwalker series as well: the pieces making up the story are of good quality, and Elliott's writing is solid, but the narrative magic never quite coheres for me. It's the sort of novella where I finished reading, went "huh," and then was excited to start a new book.

I have no idea if there are plans for more stories in this universe, but Servant Mage felt like a prelude to a longer series. If that series does materialize, there are some signs that it could be more satisfying. At the end of the story, Fellian is finally in a place to do something original and different, and I am mildly curious what she might do. Maybe enough to read the next book, if one turns up.

Mostly, though, I'm waiting for the sequel to Unconquerable Sun. Next April!

Rating: 6 out of 10

24 November, 2022 04:39AM

November 23, 2022

hackergotchi for Jonathan Dowland

Jonathan Dowland

eventual consistency

Reading some documentation about using hledger, a particular passage jumped out at me:

It should be easy to work towards eventual consistency. …I should be able to do them bit by little bit, leaving things half-done, and picking them up later with little (mental) effort. Eventually my records would be perfect and consistent.

This has been an approach I've used for many things in my life for a long time. It has something in common with eat the elephant one mouthful at a time. I think there are some categories of problems that you can't solve this way: perhaps because with this approach you are always stuck in a local minima/maxima. On the other hand I often feel that there's no way I can address some problems at all without doing so in the smallest of steps.

Personal finance is definitely one of those.

23 November, 2022 11:25AM

François Marier

Name resolution errors in Ubuntu repositories while building a docker container

I ran into what seemed to be a DNS problem when building a Docker container:

Err http://archive.ubuntu.com jammy Release.gpg
  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.

I found that many solutions talked about setting the default DNS server explicitly in /etc/docker/daemon.json:

    "dns": [""]

but that didn't work for me.

I noticed however that I was having these problems whenever I connected to my VPN. So what did work for me was restarting the docker daemon whenever there is a change in networking (e.g. enabling/disabling VPN) by putting the following in /etc/NetworkManager/dispatcher.d/docker-local:



if [ -z "$1" ]; then
    echo "$0: called with no interface" >> $LOGFILE
    exit 1;

if [ "$1" = lo ]; then
    # Ignoring the loopback interface
    exit 0;

case "$2" in
        echo "$0: restarting docker due to action \`$2' on interface \`$1'" >> $LOGFILE
        /bin/systemctl restart docker.service
        echo "$0: ignoring action \`$2' on interface \`$1'" >> $LOGFILE

and then making that new file executable:

chmod +x /etc/NetworkManager/dispatcher.d/docker-local

You can confirm that it's working as intended by watching the logs:

tail -f /var/log/docker-restarts.log

while enabling/disable your VPN or your network connection. If you don't see any output, then something is wrong and the Docker restart isn't happening.

23 November, 2022 08:12AM

November 22, 2022

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

spdl 0.0.1 on CRAN: New Package!

A new package made it to CRAN today: spdl. It provides a very simple interface to spdlog with the key focus being that provides the same interface from both R and C++. I had brought spdlog to R a little while ago via RcppSpdlog, but its use was generally limited to C++ code where we would write something like (taken from one of the examples included in the package)

spdlog::info("Welcome to spdlog!");

I then provided C-level exports from the package accessible in other package but with that the signature only became longer as it also included the package name, here RcppSpdlog. So I experimented a little with that and added another ‘more compactly named’ namespace spdl around it. So then it becomes

spdl::info("Info message with values {} and {}", 42, 1.23);  // C++ code

which now also shows the power of the included fmt library. And I got quite used to that … and now wanted to same from R! But to create a new namespace, we need a new package. So … we made one. Now from R too:

spdl::info("Info message with values {} and {}", 42L, 1.23)  # R code

We layered a very simple wrapper for fmt over this. All R argument are passed to format() and we simply send a vector of strings over the C interface to the RcppSpdlog package where the templated formatter of fmt is invoked – but for character values. You can equally well format the string locally first. Everything works: from paste() to sprintf() to any of the existing string interpolators all of which are nice. But none works just like fmt and I wanted to have the same formatter in both languages, and like how fmt works.

By tradition, the NEWS entry for this release follows.

Changes in spld version 0.0.1 (2022-11-21)

  • Initial release of R and C++ loggers using spdl::* namespace

More detailed information is on the spdl page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 November, 2022 10:39PM

November 21, 2022

RcppClassic 0.9.13 on CRAN: Minor Update

A maintenance release 0.9.14 of the RcppClassic package arrived earlier today on CRAN. This package provides a maintained version of the otherwise deprecated initial Rcpp API which no new projects should use as the normal Rcpp API is so much better.

The changes is. CRAN was reporting (for all four macOS builds, and only there) that an absolute path was embedded, so we updated the (old …) call to install_name_tool use on that (and only that) OS. No other changes were made.

CRANberries also reports the changes relative to the previous release from nearly three years ago.

Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 November, 2022 10:36PM

November 19, 2022

Antoine Beaupré

Wayland: i3 to Sway migration

I started migrating my graphical workstations to Wayland, specifically migrating from i3 to Sway. This is mostly to address serious graphics bugs in the latest Framwork laptop, but also something I felt was inevitable.

The current status is that I've been able to convert my i3 configuration to Sway, and adapt my systemd startup sequence to the new environment. Screen sharing only works with Pipewire, so I also did that migration, which basically requires an upgrade to Debian bookworm to get a nice enough Pipewire release.

I'm testing Wayland on my laptop, but I'm not using it as a daily driver because I first need to upgrade to Debian bookworm on my main workstation.

Most irritants have been solved one way or the other. My main problem with Wayland right now is that I spent a frigging week doing the conversion: it's exciting and new, but it basically sucked the life out of all my other projects and it's distracting, and I want it to stop.

The rest of this page documents why I made the switch, how it happened, and what's left to do. Hopefully it will keep you from spending as much time as I did in fixing this.

TL;DR: Wayland is mostly ready. Main blockers you might find are that you need to do manual configurations, DisplayLink (multiple monitors on a single cable) doesn't work in Sway, HDR and color management are still in development.

I had to install the following packages:

apt install \
    brightnessctl \
    foot \
    gammastep \
    gdm3 \
    grim slurp \
    pipewire-pulse \
    sway \
    swayidle \
    swaylock \
    wdisplays \
    wev \
    wireplumber \
    wlr-randr \

And did some of tweaks in my $HOME, mostly dealing with my esoteric systemd startup sequence, which you won't have to deal with if you are not a fan.

Why switch?

I originally held back from migrating to Wayland: it seemed like a complicated endeavor hardly worth the cost. It also didn't seem actually ready.

But after reading this blurb on LWN, I decided to at least document the situation here. The actual quote that convinced me it might be worth it was:

It’s amazing. I have never experienced gaming on Linux that looked this smooth in my life.

... I'm not a gamer, but I do care about latency. The longer version is worth a read as well.

The point here is not to bash one side or the other, or even do a thorough comparison. I start with the premise that Xorg is likely going away in the future and that I will need to adapt some day. In fact, the last major Xorg release (21.1, October 2021) is rumored to be the last ("just like the previous release...", that said, minor releases are still coming out, e.g. 21.1.4). Indeed, it seems even core Xorg people have moved on to developing Wayland, or at least Xwayland, which was spun off it its own source tree.

X, or at least Xorg, in in maintenance mode and has been for years. Granted, the X Window System is getting close to forty years old at this point: it got us amazingly far for something that was designed around the time the first graphical interface. Since Mac and (especially?) Windows released theirs, they have rebuilt their graphical backends numerous times, but UNIX derivatives have stuck on Xorg this entire time, which is a testament to the design and reliability of X. (Or our incapacity at developing meaningful architectural change across the entire ecosystem, take your pick I guess.)

What pushed me over the edge is that I had some pretty bad driver crashes with Xorg while screen sharing under Firefox, in Debian bookworm (around November 2022). The symptom would be that the UI would completely crash, reverting to a text-only console, while Firefox would keep running, audio and everything still working. People could still see my screen, but I couldn't, of course, let alone interact with it. All processes still running, including Xorg.

(And no, sorry, I haven't reported that bug, maybe I should have, and it's actually possible it comes up again in Wayland, of course. But at first, screen sharing didn't work of course, so it's coming a much further way. After making screen sharing work, though, the bug didn't occur again, so I consider this a Xorg-specific problem until further notice.)

There were also frustrating glitches in the UI, in general. I actually had to setup a compositor alongside i3 to make things bearable at all. Video playback in a window was laggy, sluggish, and out of sync.

Wayland fixed all of this.

Wayland equivalents

This section documents each tool I have picked as an alternative to the current Xorg tool I am using for the task at hand. It also touches on other alternatives and how the tool was configured.

Note that this list is based on the series of tools I use in desktop.

TODO: update desktop with the following when done, possibly moving old configs to a ?xorg archive.

Window manager: i3 → sway

This seems like kind of a no-brainer. Sway is around, it's feature-complete, and it's in Debian.

I'm a bit worried about the "Drew DeVault community", to be honest. There's a certain aggressiveness in the community I don't like so much; at least an open hostility towards more modern UNIX tools like containers and systemd that make it hard to do my work while interacting with that community.

I'm also concern about the lack of unit tests and user manual for Sway. The i3 window manager has been designed by a fellow (ex-)Debian developer I have a lot of respect for (Michael Stapelberg), partly because of i3 itself, but also working with him on other projects. Beyond the characters, i3 has a user guide, a code of conduct, and lots more documentation. It has a test suite.

Sway has... manual pages, with the homepage just telling users to use man -k sway to find what they need. I don't think we need that kind of elitism in our communities, to put this bluntly.

But let's put that aside: Sway is still a no-brainer. It's the easiest thing to migrate to, because it's mostly compatible with i3. I had to immediately fix those resources to get a minimal session going:

i3 Sway note
set_from_resources set no support for X resources, naturally
new_window pixel 1 default_border pixel 1 actually supported in i3 as well

That's it. All of the other changes I had to do (and there were actually a lot) were all Wayland-specific changes, not Sway-specific changes. For example, use brightnessctl instead of xbacklight to change the backlight levels.

See a copy of my full sway/config for details.

Other options include:

  • dwl: tiling, minimalist, dwm for Wayland, not in Debian
  • Hyprland: tiling, fancy animations, not in Debian
  • Qtile: tiling, extensible, in Python, not in Debian (1015267)
  • river: Zig, stackable, tagging, not in Debian (1006593)
  • velox: inspired by xmonad and dwm, not in Debian
  • vivarium: inspired by xmonad, not in Debian

Status bar: py3status → waybar

I have invested quite a bit of effort in setting up my status bar with py3status. It supports Sway directly, and did not actually require any change when migrating to Wayland.

Unfortunately, I had trouble making nm-applet work. Based on this nm-applet.service, I found that you need to pass --indicator for it to show up at all.

In theory, tray icon support was merged in 1.5, but in practice there are still several limitations, like icons not clickable. Also, on startup, nm-applet --indicator triggers this error in the Sway logs:

nov 11 22:34:12 angela sway[298938]: 00:49:42.325 [INFO] [swaybar/tray/host.c:24] Registering Status Notifier Item ':1.47/org/ayatana/NotificationItem/nm_applet'
nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet IconPixmap: No such property “IconPixmap”
nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet AttentionIconPixmap: No such property “AttentionIconPixmap”
nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet ItemIsMenu: No such property “ItemIsMenu”
nov 11 22:36:10 angela sway[313419]: info: fcft.c:838: /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf: size=24.00pt/32px, dpi=96.00

... but that seems innocuous. The tray icon displays but is not clickable.

Note that there is currently (November 2022) a pull request to hook up a "Tray D-Bus Menu" which, according to Reddit might fix this, or at least be somewhat relevant.

If you don't see the icon, check the bar.tray_output property in the Sway config, try: tray_output *.

The non-working tray was the biggest irritant in my migration. I have used nmtui to connect to new Wifi hotspots or change connection settings, but that doesn't support actions like "turn off WiFi".

I eventually fixed this by switching from py3status to waybar, which was another yak horde shaving session, but ultimately, it worked.

Web browser: Firefox

Firefox has had support for Wayland for a while now, with the team enabling it by default in nightlies around January 2022. It's actually not easy to figure out the state of the port, the meta bug report is still open and it's huge: it currently (Sept 2022) depends on 76 open bugs, it was opened twelve (2010) years ago, and it's still getting daily updates (mostly linking to other tickets).

Firefox 106 presumably shipped with "Better screen sharing for Windows and Linux Wayland users", but I couldn't quite figure out what those were.

TL;DR: echo MOZ_ENABLE_WAYLAND=1 >> ~/.config/environment.d/firefox.conf && apt install xdg-desktop-portal-wlr

How to enable it

Firefox depends on this silly variable to start correctly under Wayland (otherwise it starts inside Xwayland and looks fuzzy and fails to screen share):


To make the change permanent, many recipes recommend adding this to an environment startup script:

if [ "$XDG_SESSION_TYPE" == "wayland" ]; then

At least that's the theory. In practice, Sway doesn't actually run any startup shell script, so that can't possibly work. Furthermore, XDG_SESSION_TYPE is not actually set when starting Sway from gdm3 which I find really confusing, and I'm not the only one. So the above trick doesn't actually work, even if the environment (XDG_SESSION_TYPE) is set correctly, because we don't have conditionals in environment.d(5).

(Note that systemd.environment-generator(7) do support running arbitrary commands to generate environment, but for some some do not support user-specific configuration files... Even then it may be a solution to have a conditional MOZ_ENABLE_WAYLAND environment, but I'm not sure it would work because ordering between those two isn't clear: maybe the XDG_SESSION_TYPE wouldn't be set just yet...)

At first, I made this ridiculous script to workaround those issues. Really, it seems to me Firefox should just parse the XDG_SESSION_TYPE variable here... but then I realized that Firefox works fine in Xorg when the MOZ_ENABLE_WAYLAND is set.

So now I just set that variable in environment.d and It Just Works™:


Screen sharing

Out of the box, screen sharing doesn't work until you install xdg-desktop-portal-wlr or similar (e.g. xdg-desktop-portal-gnome on GNOME). I had to reboot for the change to take effect.

Without those tools, it shows the usual permission prompt with "Use operating system settings" as the only choice, but when we accept... nothing happens. After installing the portals, it actualyl works, and works well!

This was tested in Debian bookworm/testing with Firefox ESR 102 and Firefox 106.

Major caveat: we can only share a full screen, we can't currently share just a window. The major upside to that is that, by default, it streams only one output which is actually what I want most of the time! See the screencast compatibility for more information on what is supposed to work.

This is actually a huge improvement over the situation in Xorg, where Firefox can only share a window or all monitors, which led me to use Chromium a lot for video-conferencing. With this change, in other words, I will not need Chromium for anything anymore, whoohoo!

If slurp, wofi, or bemenu are installed, one of them will be used to pick the monitor to share, which effectively acts as some minimal security measure. See xdg-desktop-portal-wlr(1) for how to configure that.

Side note: Chrome fails to share a full screen

I was still using Google Chrome (or, more accurately, Debian's Chromium package) for some videoconferencing. It's mainly because Chromium was the only browser which will allow me to share only one of my two monitors, which is extremely useful.

To start chrome with the Wayland backend, you need to use:

chromium  -enable-features=UseOzonePlatform -ozone-platform=wayland

If it shows an ugly gray border, check the Use system title bar and borders setting.

It can do some screensharing. Sharing a window and a tab seems to work, but sharing a full screen doesn't: it's all black. Maybe not ready for prime time.

And since Firefox can do what I need under Wayland now, I will not need to fight with Chromium to work under Wayland:

apt purge chromium

Note that a similar fix was necessary for Signal Desktop, see this commit. Basically you need to figure out a way to pass those same flags to signal:

--enable-features=WaylandWindowDecorations --ozone-platform-hint=auto

Email: notmuch

See Emacs, below.

File manager: thunar


News: feed2exec, gnus

See Email, above, or Emacs in Editor, below.

Editor: Emacs okay-ish

Emacs is being actively ported to Wayland. According to this LWN article, the first (partial, to Cairo) port was done in 2014 and a working port (to GTK3) was completed in 2021, but wasn't merged until late 2021. That is: after Emacs 28 was released (April 2022).

So we'll probably need to wait for Emacs 29 to have native Wayland support in Emacs, which, in turn, is unlikely to arrive in time for the Debian bookworm freeze. There are, however, unofficial builds for both Emacs 28 and 29 provided by spwhitton which may provide native Wayland support.

I tested the snapshot packages and they do not quite work well enough. First off, they completely take over the builtin Emacs — they hijack the $PATH in /etc! — and certain things are simply not working in my setup. For example, this hook never gets ran on startup:

(add-hook 'after-init-hook 'server-start t) 

Still, like many X11 applications, Emacs mostly works fine under Xwayland. The clipboard works as expected, for example.

Scaling is a bit of an issue: fonts look fuzzy.

I have heard anecdotal evidence of hard lockups with Emacs running under Xwayland as well, but haven't experienced any problem so far. I did experience a Wayland crash with the snapshot version however.

TODO: look again at Wayland in Emacs 29.

Backups: borg

Mostly irrelevant, as I do not use a GUI.

Color theme: srcery, redshift → gammastep

I am keeping Srcery as a color theme, in general.

Redshift is another story: it has no support for Wayland out of the box, but it's apparently possible to apply a hack on the TTY before starting Wayland, with:

redshift -m drm -PO 3000

This tip is from the arch wiki which also has other suggestions for Wayland-based alternatives. Both KDE and GNOME have their own "red shifters", and for wlroots-based compositors, they (currently, Sept. 2022) list the following alternatives:

I configured gammastep with a simple gammastep.service file associated with the sway-session.target.

Display manager: lightdm → gdm3

Switched because lightdm failed to start sway:

nov 16 16:41:43 angela sway[843121]: 00:00:00.002 [ERROR] [wlr] [libseat] [common/terminal.c:162] Could not open target tty: Permission denied

Possible alternatives:

Terminal: xterm → foot

One of the biggest question mark in this transition was what to do about Xterm. After writing two articles about terminal emulators as a professional journalist, decades of working on the terminal, and probably using dozens of different terminal emulators, I'm still not happy with any of them.

This is such a big topic that I actually have an entire blog post specifically about this.

For starters, using xterm under Xwayland works well enough, although the font scaling makes things look a bit too fuzzy.

I have also tried foot: it ... just works!

Fonts are much crisper than Xterm and Emacs. URLs are not clickable but the URL selector (control-shift-u) is just plain awesome (think "vimperator" for the terminal).

There's cool hack to jump between prompts.

Copy-paste works. True colors work. The word-wrapping is excellent: it doesn't lose one byte. Emojis are nicely sized and colored. Font resize works. There's even scroll back search (control-shift-r).

Foot went from a question mark to being a reason to switch to Wayland, just for this little goodie, which says a lot about the quality of that software.

The selection clicks are a not quite what I would expect though. In rxvt and others, you have the following patterns:

  • single click: reset selection, or drag to select
  • double: select word
  • triple: select quotes or line
  • quadruple: select line

I particularly find the "select quotes" bit useful. It seems like foot just supports double and triple clicks, with word and line selected. You can select a rectangle with control,. It correctly extends the selection word-wise with right click if double-click was first used.

One major problem with Foot is that it's a new terminal, with its own termcap entry. Support for foot was added to ncurses in the 20210731 release, which was shipped after the current Debian stable release (Debian bullseye, which ships 6.2+20201114-2). A workaround for this problem is to install the foot-terminfo package on the remote host, which is available in Debian stable.

This should eventually resolve itself, as Debian bookworm has a newer version. Note that some corrections were also shipped in the 20211113 release, but that is also shipped in Debian bookworm.

That said, I am almost certain I will have to revert back to xterm under Xwayland at some point in the future. Back when I was using GNOME Terminal, it would mostly work for everything until I had to use the serial console on a (HP ProCurve) network switch, which have a fancy TUI that was basically unusable there. I fully expect such problems with foot, or any other terminal than xterm, for that matter.

The foot wiki has good troubleshooting instructions as well.

Update: I did find one tiny thing to improve with foot, and it's the default logging level which I found pretty verbose. After discussing it with the maintainer on IRC, I submitted this patch to tweak it, which I described like this on Mastodon:

today's reason why i will go to hell when i die (TRWIWGTHWID?): a 600-word, 63 lines commit log for a one line change: https://codeberg.org/dnkl/foot/pulls/1215

It's Friday.

Launcher: rofi → rofi??

rofi does not support Wayland. There was a rather disgraceful battle in the pull request that led to the creation of a fork (lbonn/rofi), so it's unclear how that will turn out.

Given how relatively trivial problem space is, there is of course a profusion of options:

Tool In Debian Notes
alfred yes general launcher/assistant tool
bemenu yes, bookworm+ inspired by dmenu
cerebro no Javascript ... uh... thing
dmenu-wl no fork of dmenu, straight port to Wayland
Fuzzel ITP 982140 dmenu/drun replacement, app icon overlay
gmenu no drun replacement, with app icons
kickoff no dmenu/run replacement, fuzzy search, "snappy", history, copy-paste, Rust
krunner yes KDE's runner
mauncher no dmenu/drun replacement, math
nwg-launchers no dmenu/drun replacement, JSON config, app icons, nwg-shell project
Onagre no rofi/alfred inspired, multiple plugins, Rust
πmenu no dmenu/drun rewrite
Rofi (lbonn's fork) no see above
sirula no .desktop based app launcher
Ulauncher ITP 949358 generic launcher like Onagre/rofi/alfred, might be overkill
tofi yes, bookworm+ dmenu/drun replacement, C
wmenu no fork of dmenu-wl, but mostly a rewrite
Wofi yes dmenu/drun replacement, not actively maintained
yofi no dmenu/drun replacement, Rust

The above list comes partly from https://arewewaylandyet.com/ and awesome-wayland. It is likely incomplete.

I have read some good things about bemenu, fuzzel, and wofi.

A particularly tricky option is that my rofi password management depends on xdotool for some operations. At first, I thought this was just going to be (thankfully?) impossible, because we actually like the idea that one app cannot send keystrokes to another. But it seems there are actually alternatives to this, like wtype or ydotool, the latter which requires root access. wl-ime-type does that through the input-method-unstable-v2 protocol (sample emoji picker, but is not packaged in Debian.

As it turns out, wtype just works as expected, and fixing this was basically a two-line patch. Another alternative, not in Debian, is wofi-pass.

The other problem is that I actually heavily modified rofi. I use "modis" which are not actually implemented in wofi or tofi, so I'm left with reinventing those wheels from scratch or using the rofi + wayland fork... It's really too bad that fork isn't being reintegrated...

For now, I'm actually still using rofi under Xwayland. The main downside is that fonts are fuzzy, but it otherwise just works.

Note that wlogout could be a partial replacement (just for the "power menu").

Image viewers: geeqie → ?

I'm not very happy with geeqie in the first place, and I suspect the Wayland switch will just make add impossible things on top of the things I already find irritating (Geeqie doesn't support copy-pasting images).

In practice, Geeqie doesn't seem to work so well under Wayland. The fonts are fuzzy and the thumbnail preview just doesn't work anymore (filed as Debian bug 1024092). It seems it also has problems with scaling.


See also this list and that list for other list of image viewers, not necessarily ported to Wayland.

TODO: pick an alternative to geeqie, nomacs would be gorgeous if it wouldn't be basically abandoned upstream (no release since 2020), has an unpatched CVE-2020-23884 since July 2020, does bad vendoring, and is in bad shape in Debian (4 minor releases behind).

So for now I'm still grumpily using Geeqie.

Media player: mpv, gmpc / sublime

This is basically unchanged. mpv seems to work fine under Wayland, better than Xorg on my new laptop (as mentioned in the introduction), and that before the version which improves Wayland support significantly, by bringing native Pipewire support and DMA-BUF support.

gmpc is more of a problem, mainly because it is abandoned. See 2022-08-22-gmpc-alternatives for the full discussion, one of the alternatives there will likely support Wayland.

Finally, I might just switch to sublime-music instead... In any case, not many changes here, thankfully.

Screensaver: xsecurelock → swaylock

I was previously using xss-lock and xsecurelock as a screensaver, with xscreensaver "hacks" as a backend for xsecurelock.

The basic screensaver in Sway seems to be built with swayidle and swaylock. It's interesting because it's the same "split" design as xss-lock and xsecurelock.

That, unfortunately, does not include the fancy "hacks" provided by xscreensaver, and that is unlikely to be implemented upstream.

Other alternatives include gtklock and waylock (zig), which do not solve that problem either.

It looks like swaylock-plugin, a swaylock fork, which at least attempts to solve this problem, although not directly using the real xscreensaver hacks. swaylock-effects is another attempt at this, but it only adds more effects, it doesn't delegate the image display.

Other than that, maybe it's time to just let go of those funky animations and just let swaylock do it's thing, which is display a static image or just a black screen, which is fine by me.

In the end, I am just using swayidle with a configuration based on the systemd integration wiki page but with additional tweaks from this service, see the resulting swayidle.service file.

Interestingly, damjan also has a service for swaylock itself, although it's not clear to me what its purpose is...

Screenshot: maim → grim, pubpaste

I'm a heavy user of maim (and a package uploader in Debian). It looks like the direct replacement to maim (and slop) is grim (and slurp). There's also swappy which goes on top of grim and allows preview/edit of the resulting image, nice touch (not in Debian though).

See also awesome-wayland screenshots for other alternatives: there are many, including X11 tools like Flameshot that also support Wayland.

One key problem here was that I have my own screenshot / pastebin software which will needed an update for Wayland as well. That, thankfully, meant actually cleaning up a lot of horrible code that involved calling xterm and xmessage for user interaction. Now, pubpaste uses GTK for prompts and looks much better. (And before anyone freaks out, I already had to use GTK for proper clipboard support, so this isn't much of a stretch...)

Screen recorder: simplescreenrecorder → wf-recorder

In Xorg, I have used both peek or simplescreenrecorder for screen recordings. The former will work in Wayland, but has no sound support. The latter has a fork with Wayland support but it is limited and buggy ("doesn't support recording area selection and has issues with multiple screens").

It looks like wf-recorder will just do everything correctly out of the box, including audio support (with --audio, duh). It's also packaged in Debian.

One has to wonder how this works while keeping the "between app security" that Wayland promises, however... Would installing such a program make my system less secure?

Many other options are available, see the awesome Wayland screencasting list.

RSI: workrave → nothing?

Workrave has no support for Wayland. activity watch is a time tracker alternative, but is not a RSI watcher. KDE has rsiwatcher, but that's a bit too much on the heavy side for my taste.

SafeEyes looks like an alternative at first, but it has many issues under Wayland (escape doesn't work, idle doesn't work, it just doesn't work really). timekpr-next could be an alternative as well, and has support for Wayland.

I am also considering just abandoning workrave, even if I stick with Xorg, because it apparently introduces significant latency in the input pipeline.

And besides, I've developed a pretty unhealthy alert fatigue with Workrave. I have used the program for so long that my fingers know exactly where to click to dismiss those warnings very effectively. It makes my work just more irritating, and doesn't fix the fundamental problem I have with computers.

Other apps

This is a constantly changing list, of course. There's a bit of a "death by a thousand cuts" in migrating to Wayland because you realize how many things you were using are tightly bound to X.

  • .Xresources - just say goodbye to that old resource system, it was used, in my case, only for rofi, xterm, and ... Xboard!?

  • keyboard layout switcher: built-in to Sway since 2017 (PR 1505, 1.5rc2+), requires a small configuration change, see this answer as well, looks something like this command:

     swaymsg input 0:0:X11_keyboard xkb_layout de

    or using this config:

     input * {
         xkb_layout "ca,us"
         xkb_options "grp:sclk_toggle"

    That works refreshingly well, even better than in Xorg, I must say.

    swaykbdd is an alternative that supports per-window layouts (in Debian).

  • wallpaper: currently using feh, will need a replacement, TODO: figure out something that does, like feh, a random shuffle. swaybg just loads a single image, duh. oguri might be a solution, but unmaintained, used here, not in Debian. wallutils is another option, also not in Debian. For now I just don't have a wallpaper, the background is a solid gray, which is better than Xorg's default (which is whatever crap was left around a buffer by the previous collection of programs, basically)

  • notifications: currently using dunst in some places, which works well in both Xorg and Wayland, not a blocker, salut a possible alternative (not in Debian), damjan uses mako. TODO: install dunst everywhere

  • notification area: I had trouble making nm-applet work. based on this nm-applet.service, I found that you need to pass --indicator. In theory, tray icon support was merged in 1.5, but in practice there are still several limitations, like icons not clickable. On startup, nm-applet --indicator triggers this error in the Sway logs:

     nov 11 22:34:12 angela sway[298938]: 00:49:42.325 [INFO] [swaybar/tray/host.c:24] Registering Status Notifier Item ':1.47/org/ayatana/NotificationItem/nm_applet'
     nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet IconPixmap: No such property “IconPixmap”
     nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet AttentionIconPixmap: No such property “AttentionIconPixmap”
     nov 11 22:34:12 angela sway[298938]: 00:49:42.327 [ERROR] [swaybar/tray/item.c:127] :1.47/org/ayatana/NotificationItem/nm_applet ItemIsMenu: No such property “ItemIsMenu”
     nov 11 22:36:10 angela sway[313419]: info: fcft.c:838: /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf: size=24.00pt/32px, dpi=96.00

    ... but it seems innocuous. The tray icon displays but, as stated above, is not clickable. If you don't see the icon, check the bar.tray_output property in the Sway config, try: tray_output *. Note that there is currently (November 2022) a pull request to hook up a "Tray D-Bus Menu" which, according to Reddit might fix this, or at least be somewhat relevant.

    This was the biggest irritant in my migration. I have used nmtui to connect to new Wifi hotspots or change connection settings, but that doesn't support actions like "turn off WiFi".

    I eventually fixed this by switching from py3status to waybar.

  • window switcher: in i3 I was using this bespoke i3-focus script, which doesn't work under Sway, swayr an option, not in Debian. So I put together this other bespoke hack from multiple sources, which works.

  • PDF viewer: currently using atril (which supports Wayland), could also just switch to zatura/mupdf permanently, see also calibre for a discussion on document viewers

See also this list of useful addons and this other list for other app alternatives.

More X11 / Wayland equivalents

For all the tools above, it's not exactly clear what options exist in Wayland, or when they do, which one should be used. But for some basic tools, it seems the options are actually quite clear. If that's the case, they should be listed here:

X11 Wayland In Debian
arandr wdisplays yes
autorandr kanshi yes
xdotool wtype yes
xev wev yes
xlsclients swaymsg -t get_tree yes
xrandr wlr-randr yes

lswt is a more direct replacement for xlsclients but is not packaged in Debian.

See also:

Note that arandr and autorandr are not directly part of X. arewewaylandyet.com refers to a few alternatives. We suggest wdisplays and kanshi above (see also this service file) but wallutils can also do the autorandr stuff, apparently, and nwg-displays can do the arandr part. Neither are packaged in Debian yet.

So I have tried wdisplays and it Just Works, and well. The UI even looks better and more usable than arandr, so another clean win from Wayland here.

TODO: test kanshi as a autorandr replacement

Other issues

systemd integration

I've had trouble getting session startup to work. This is partly because I had a kind of funky system to start my session in the first place. I used to have my whole session started from .xsession like this:


. ~/.shenv

systemctl --user import-environment

exec systemctl --user start --wait xsession.target

But obviously, the xsession.target is not started by the Sway session. It seems to just start a default.target, which is really not what we want because we want to associate the services directly with the graphical-session.target, so that they don't start when logging in over (say) SSH.

damjan on #debian-systemd showed me his sway-setup which features systemd integration. It involves starting a different session in a completely new .desktop file. That work was submitted upstream but refused on the grounds that "I'd rather not give a preference to any particular init system." Another PR was abandoned because "restarting sway does not makes sense: that kills everything".

The work was therefore moved to the wiki.

So. Not a great situation. The upstream wiki systemd integration suggests starting the systemd target from within Sway, which has all sorts of problems:

  • you don't get Sway logs anywhere
  • control groups are all messed up

I have done a lot of work trying to figure this out, but I remember that starting systemd from Sway didn't actually work for me: my previously configured systemd units didn't correctly start, and especially not with the right $PATH and environment.

So I went down that rabbit hole and managed to correctly configure Sway to be started from the systemd --user session. I have partly followed the wiki but also picked ideas from damjan's sway-setup and xdbob's sway-services. Another option is uwsm (not in Debian).

This is the config I have in .config/systemd/user/:

I have also configured those services, but that's somewhat optional:

You will also need at least part of my sway/config, which sends the systemd notification (because, no, Sway doesn't support any sort of readiness notification, that would be too easy). And you might like to see my swayidle-config while you're there.

Finally, you need to hook this up somehow to the login manager. This is typically done with a desktop file, so drop sway-session.desktop in /usr/share/wayland-sessions and sway-user-service somewhere in your $PATH (typically /usr/bin/sway-user-service).

The session then looks something like this:

$ systemd-cgls | head -101
Control group /:
├─user.slice (#472)
│ → user.invocation_id: bc405c6341de4e93a545bde6d7abbeec
│ → trusted.invocation_id: bc405c6341de4e93a545bde6d7abbeec
│ └─user-1000.slice (#10072)
│   → user.invocation_id: 08f40f5c4bcd4fd6adfd27bec24e4827
│   → trusted.invocation_id: 08f40f5c4bcd4fd6adfd27bec24e4827
│   ├─user@1000.service … (#10156)
│   │ → user.delegate: 1
│   │ → trusted.delegate: 1
│   │ → user.invocation_id: 76bed72a1ffb41dca9bfda7bb174ef6b
│   │ → trusted.invocation_id: 76bed72a1ffb41dca9bfda7bb174ef6b
│   │ ├─session.slice (#10282)
│   │ │ ├─xdg-document-portal.service (#12248)
│   │ │ │ ├─9533 /usr/libexec/xdg-document-portal
│   │ │ │ └─9542 fusermount3 -o rw,nosuid,nodev,fsname=portal,auto_unmount,subt…
│   │ │ ├─xdg-desktop-portal.service (#12211)
│   │ │ │ └─9529 /usr/libexec/xdg-desktop-portal
│   │ │ ├─pipewire-pulse.service (#10778)
│   │ │ │ └─6002 /usr/bin/pipewire-pulse
│   │ │ ├─wireplumber.service (#10519)
│   │ │ │ └─5944 /usr/bin/wireplumber
│   │ │ ├─gvfs-daemon.service (#10667)
│   │ │ │ └─5960 /usr/libexec/gvfsd
│   │ │ ├─gvfs-udisks2-volume-monitor.service (#10852)
│   │ │ │ └─6021 /usr/libexec/gvfs-udisks2-volume-monitor
│   │ │ ├─at-spi-dbus-bus.service (#11481)
│   │ │ │ ├─6210 /usr/libexec/at-spi-bus-launcher
│   │ │ │ ├─6216 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2…
│   │ │ │ └─6450 /usr/libexec/at-spi2-registryd --use-gnome-session
│   │ │ ├─pipewire.service (#10403)
│   │ │ │ └─5940 /usr/bin/pipewire
│   │ │ └─dbus.service (#10593)
│   │ │   └─5946 /usr/bin/dbus-daemon --session --address=systemd: --nofork --n…
│   │ ├─background.slice (#10324)
│   │ │ └─tracker-miner-fs-3.service (#10741)
│   │ │   └─6001 /usr/libexec/tracker-miner-fs-3
│   │ ├─app.slice (#10240)
│   │ │ ├─xdg-permission-store.service (#12285)
│   │ │ │ └─9536 /usr/libexec/xdg-permission-store
│   │ │ ├─gammastep.service (#11370)
│   │ │ │ └─6197 gammastep
│   │ │ ├─dunst.service (#11958)
│   │ │ │ └─7460 /usr/bin/dunst
│   │ │ ├─wterminal.service (#13980)
│   │ │ │ ├─69100 foot --title pop-up
│   │ │ │ ├─69101 /bin/bash
│   │ │ │ ├─77660 sudo systemd-cgls
│   │ │ │ ├─77661 head -101
│   │ │ │ ├─77662 wl-copy
│   │ │ │ ├─77663 sudo systemd-cgls
│   │ │ │ └─77664 systemd-cgls
│   │ │ ├─syncthing.service (#11995)
│   │ │ │ ├─7529 /usr/bin/syncthing -no-browser -no-restart -logflags=0 --verbo…
│   │ │ │ └─7537 /usr/bin/syncthing -no-browser -no-restart -logflags=0 --verbo…
│   │ │ ├─dconf.service (#10704)
│   │ │ │ └─5967 /usr/libexec/dconf-service
│   │ │ ├─gnome-keyring-daemon.service (#10630)
│   │ │ │ └─5951 /usr/bin/gnome-keyring-daemon --foreground --components=pkcs11…
│   │ │ ├─gcr-ssh-agent.service (#10963)
│   │ │ │ └─6035 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr
│   │ │ ├─swayidle.service (#11444)
│   │ │ │ └─6199 /usr/bin/swayidle -w
│   │ │ ├─nm-applet.service (#11407)
│   │ │ │ └─6198 /usr/bin/nm-applet --indicator
│   │ │ ├─wcolortaillog.service (#11518)
│   │ │ │ ├─6226 foot colortaillog
│   │ │ │ ├─6228 /bin/sh /home/anarcat/bin/colortaillog
│   │ │ │ ├─6230 sudo journalctl -f
│   │ │ │ ├─6233 ccze -m ansi
│   │ │ │ ├─6235 sudo journalctl -f
│   │ │ │ └─6236 journalctl -f
│   │ │ ├─afuse.service (#10889)
│   │ │ │ └─6051 /usr/bin/afuse -o mount_template=sshfs -o transform_symlinks -…
│   │ │ ├─gpg-agent.service (#13547)
│   │ │ │ ├─51662 /usr/bin/gpg-agent --supervised
│   │ │ │ └─51719 scdaemon --multi-server
│   │ │ ├─emacs.service (#10926)
│   │ │ │ ├─ 6034 /usr/bin/emacs --fg-daemon
│   │ │ │ └─33203 /usr/bin/aspell -a -m -d en --encoding=utf-8
│   │ │ ├─xdg-desktop-portal-gtk.service (#12322)
│   │ │ │ └─9546 /usr/libexec/xdg-desktop-portal-gtk
│   │ │ ├─xdg-desktop-portal-wlr.service (#12359)
│   │ │ │ └─9555 /usr/libexec/xdg-desktop-portal-wlr
│   │ │ └─sway.service (#11037)
│   │ │   ├─6037 /usr/bin/sway
│   │ │   ├─6181 swaybar -b bar-0
│   │ │   ├─6209 py3status
│   │ │   ├─6309 /usr/bin/i3status -c /tmp/py3status_oy4ntfnq
│   │ │   └─6969 Xwayland :0 -rootless -terminate -core -listen 29 -listen 30 -…
│   │ └─init.scope (#10198)
│   │   ├─5909 /lib/systemd/systemd --user
│   │   └─5911 (sd-pam)
│   └─session-7.scope (#10440)
│     ├─5895 gdm-session-worker [pam/gdm-password]
│     ├─6028 /usr/libexec/gdm-wayland-session --register-session sway-user-serv…

I think that's pretty neat.

Environment propagation

At first, my terminals and rofi didn't have the right $PATH, which broke a lot of my workflow. It's hard to tell exactly how Wayland gets started or where to inject environment. This discussion suggests a few alternatives and this Debian bug report discusses this issue as well.

I eventually picked environment.d(5) since I already manage my user session with systemd, and it fixes a bunch of other problems. I used to have a .shenv that I had to manually source everywhere. The only problem with that approach is that it doesn't support conditionals, but that's something that's rarely needed.


This is a whole topic onto itself, but migrating to Wayland also involves using Pipewire if you want screen sharing to work. You can actually keep using Pulseaudio for audio, that said, but that migration is actually something I've wanted to do anyways: Pipewire's design seems much better than Pulseaudio, as it folds in JACK features which allows for pretty neat tricks. (Which I should probably show in a separate post, because this one is getting rather long.)

I first tried this migration in Debian bullseye, and it didn't work very well. Ardour would fail to export tracks and I would get into weird situations where streams would just drop mid-way.

A particularly funny incident is when I was in a meeting and I couldn't hear my colleagues speak anymore (but they could) and I went on blabbering on my own for a solid 5 minutes until I realized what was going on. By then, people had tried numerous ways of letting me know that something was off, including (apparently) coughing, saying "hello?", chat messages, IRC, and so on, until they just gave up and left.

I suspect that was also a Pipewire bug, but it could also have been that I muted the tab by error, as I recently learned that clicking on the little tiny speaker icon on a tab mutes that tab. Since the tab itself can get pretty small when you have lots of them, it's actually quite frequently that I mistakenly mute tabs.

Anyways. Point is: I already knew how to make the migration, and I had already documented how to make the change in Puppet. It's basically:

apt install pipewire pipewire-audio-client-libraries pipewire-pulse wireplumber 

Then, as a regular user:

systemctl --user daemon-reload
systemctl --user --now disable pulseaudio.service pulseaudio.socket
systemctl --user --now enable pipewire pipewire-pulse
systemctl --user mask pulseaudio

An optional (but key, IMHO) configuration you should also make is to "switch on connect", which will make your Bluetooth or USB headset automatically be the default route for audio, when connected. In ~/.config/pipewire/pipewire-pulse.conf.d/autoconnect.conf:

context.exec = [
    { path = "pactl"        args = "load-module module-always-sink" }
    { path = "pactl"        args = "load-module module-switch-on-connect" }
    #{ path = "/usr/bin/sh"  args = "~/.config/pipewire/default.pw" }

See the excellent — as usual — Arch wiki page about Pipewire for that trick and more information about Pipewire. Note that you must not put the file in ~/.config/pipewire/pipewire.conf (or pipewire-pulse.conf, maybe) directly, as that will break your setup. If you want to add to that file, first copy the template from /usr/share/pipewire/pipewire-pulse.conf first.

So far I'm happy with Pipewire in bookworm, but I've heard mixed reports from it. I have high hopes it will become the standard media server for Linux in the coming months or years, which is great because I've been (rather boldly, I admit) on the record saying I don't like PulseAudio.

Rereading this now, I feel it might have been a little unfair, as "over-engineered and tries to do too many things at once" applies probably even more to Pipewire than PulseAudio (since it also handles video dispatching).

That said, I think Pipewire took the right approach by implementing existing interfaces like Pulseaudio and JACK. That way we're not adding a third (or fourth?) way of doing audio in Linux; we're just making the server better.

Improvements over i3

Tiling improvements

There's a lot of improvements Sway could bring over using plain i3. There are pretty neat auto-tilers that could replicate the configurations I used to have in Xmonad or Awesome, see:

Display latency tweaks

TODO: You can tweak the display latency in wlroots compositors with the max_render_time parameter, possibly getting lower latency than X11 in the end.

Sound/brightness changes notifications

TODO: Avizo can display a pop-up to give feedback on volume and brightness changes. Not in Debian. Other alternatives include SwayOSD and sway-nc, also not in Debian.

Debugging tricks

The xeyes (in the x11-apps package) will run in Wayland, and can actually be used to easily see if a given window is also in Wayland. If the "eyes" follow the cursor, the app is actually running in xwayland, so not natively in Wayland.

Another way to see what is using Wayland in Sway is with the command:

swaymsg -t get_tree

Other documentation


In general, this took me a long time, but it mostly works. The tray icon situation is pretty frustrating, but there's a workaround and I have high hopes it will eventually fix itself. I'm also actually worried about the DisplayLink support because I eventually want to be using this, but hopefully that's another thing that will hopefully fix itself before I need it.

A word on the security model

I'm kind of worried about all the hacks that have been added to Wayland just to make things work. Pretty much everywhere we need to, we punched a hole in the security model:

Wikipedia describes the security properties of Wayland as it "isolates the input and output of every window, achieving confidentiality, integrity and availability for both." I'm not sure those are actually realized in the actual implementation, because of all those holes punched in the design, at least in Sway. For example, apparently the GNOME compositor doesn't have the virtual-keyboard protocol, but they do have (another?!) text input protocol.

Wayland does offer a better basis to implement such a system, however. It feels like the Linux applications security model lacks critical decision points in the UI, like the user approving "yes, this application can share my screen now". Applications themselves might have some of those prompts, but it's not mandatory, and that is worrisome.

19 November, 2022 07:03PM

Joerg Jaspert

From QNAP QTS to TrueNAS Scale

History, Setup

So for quite some time I have a QNAP TS-873x here, equipped with 8 Western Digital Red 10 TB disks, plus 2 WD Blue 500G M2 SSDs. The QNAP itself has an “AMD Embedded R-Series RX-421MD” with 4 cores and was equipped with 48G RAM.

Initially I had been quite happy, the system is nice. It was fast, it was easy to get to run and the setup of things I wanted was simple enough. All in a web interface that tries to imitate a kind of workstation feeling and also tries to hide that it is actually a webinterface.

Natually with that amount of disks I had a RAID6 for the disks, plus RAID1 for the SSDs. And then configured as a big storage pool with the RAID1 as cache. Below the hood QNAP uses MDADM Raid and LVM (if you want, with thin provisioning), in some form of emdedded linux. The interface allows for regular snapshots of your storage with flexible enough schedules to create them, so it all appears pretty good.

QNAP slow

Fast forward some time and it gets annoying. First off you really should have regular raid resyncs scheduled, and while you can set priorities on them and have them low priority, they make the whole system feel very sluggish, quite annoying. And sure, power failure (rare, but can happen) means another full resync run. Also, it appears all of the snapshots are always mounted to some /mnt/snapshot/something place (df on the system gets quite unusable).

Second, the reboot times. QNAP seems to be affected by the “more features, fuck performance” virus, and bloat their OS with more and more features while completly ignoring the performance. Everytime they do an “upgrade” it feels worse. Lately reboot times went up to 10 to 15 minutes - and then it still hadn’t started the virtual machines / docker containers one might run on. Another 5 to 10 minutes for those. Opening the file explorer - ages on calculating what to show. Trying to get the storage setup shown? Go get a coffee, but please fetch the beans directly from the plantation, or you are too fast.

Annoying it was. And no, no broken disks or fan or anything, it all checks out fine.

Replace QNAPs QTS system

So I started looking around what to do. More RAM may help a little bit, but I already had 48G, the system itself appears to only do 64G maximum, so not much chance of it helping enough. Hardware is all fine and working, so software needs to be changed. Sounds hard, but turns out, it is not.


And I found that multiple people replaced the QNAPs own system with a TrueNAS installation and generally had been happy. Looking further I found that TrueNAS has a variant called Scale - which is based on Debian. Doubly good, that, so I went off checking what I may need for it.


Heck, that was a step back. To install TrueNAS you need an HDMI out and a disk to put it on. The one that QTS uses is too small, so no option.

QNAPs  internal USB disk
QNAPs original internal USB drive, DOM

So either use one of the SSDs that played cache (and should do so again in TrueNAS, or get the QNAP original replaced.

HDMI out is simple, get a cheap card and put it into one of the two PCIe-4x slots, done. The disk thing looked more complicated, as QNAP uses some “internal usb stick thing”. Turns out it is “just” a USB stick that has an 8+1pin connector. Couldn’t find anything nice as replacement, but hey, there are 9-pin to USB-A adapters.

a 9pin to USB A adapter

With that adapter, one can take some random M2 SSD and an M2-to-USB case, plus some cabling, and voila, we have a nice system disk.

USB 9pin  to USB-A cable connected to Motherboard and some more cable
9pin adapter to USB-A connected with some more cable

Obviously there isn’t a good place to put this SSD case and cable, but the QNAP case is large enough to find space and use some cable ties to store it safely. Space enough to get the cable from the side, where the mainboard is to the place I mounted it, so all fine.

Mounted  SSD in external case, also shows the video card
Mounted SSD in its external case

The next best M2 SSD was a Western Digital Red with 500G - and while this is WAY too much for TrueNAS, it works. And hey, only using a tiny fraction? Oh so much more cells available internally to use when others break. Or something…

Together with the Asus card mounted I was able to install TrueNAS. Which is simple, their installer is easy enough to follow, just make sure to select the right disk to put it on.

Preserving data during the move

Switching from QNAP QTS to TrueNAS Scale means changing from MDADM Raid with LVM and ext4 on top to ZFS and as such all data on it gets erased. So a backup first is helpful, and I got myself two external Seagate USB Disks of 6TB each - enough for the data I wanted to keep.

Copying things all over took ages, especially as the QNAP backup thingie sucks, it was breaking quite often. Also, for some reason I did not investigate, the performance of it was real bad. It started at a maximum of 50MB/s, but the last terabyte of data was copied at MUCH less than that, and so it took much longer than I anticipated.

Copying back was slow too, but much less so. Of course reading things usually is faster than writing, with it going around 100MB/s most of the time, which is quite a bit more - still not what USB3 can actually do, but I guess the AMD chip doesn’t want to go that fast.

TrueNAS experience

The installation went mostly smooth, the only real trouble had been on my side. Turns out that a bad network cable does NOT help the network setup, who would have thought. Other than that it is the usual set of questions you would expect, a reboot, and then some webinterface.

And here the differences start. The whole system boots up much faster. Not even a third of the time compared to QTS.

One important thing: As TrueNAS scale is Debian based, and hence a linux kernel, it automatically detects and assembles the old RAID arrays that QTS put on. Which TrueNAS can do nothing with, so it helps to manually stop them and wipe the disks.

Afterwards I put ZFS on the disks, with a similar setup to what I had before. The spinning rust are the data disks in a RAIDZ2 setup, the two SSDs are added as cache devices. Unlike MDADM, ZFS does not have a long sync process. Also unlike the MDADM/LVM/EXT4 setup from before, ZFS works different. It manages the raid thing but it also does the volume and filesystem parts. Quite different handling, and I’m still getting used to it, so no, I won’t write some ZFS introduction now.


The two systems can not be compared completly, they are having a pretty different target audience. QNAP is more for the user that wants some network storage that offers a ton of extra features easily available via a clickable interface. While TrueNAS appears more oriented to people that want a fast but reliable storage system. TrueNAS does not offer all the extra bloat the QNAP delivers. Still, you have the ability to run virtual machines and it seems it comes with Rancher, so some kubernetes/container ability is there. It lacks essential features like assigning PCI devices to virtual machines, so is not useful right now, but I assume that will come in a future version.

I am still exploring it all, but I like what I have right now. Still rebuilding my setup to have all shares exported and used again, but the most important are working already.

19 November, 2022 12:20PM

November 18, 2022

hackergotchi for Jonathan Dowland

Jonathan Dowland


I buy a lot of music from Bandcamp, but I rarely if ever download it: I rely on streaming it from the Bandcamp website, or mobile App. The prepper/archivist/hoarder in me is a little uncomfortable about that.

During the Pandemic, Bandcamp started a scheme called Bandcamp Fridays: days on which they waive their revenue share for music purchases, meaning musicians get a larger proportion of the proceeds. It proved an enormous success, and they've kept them going: you can check when the next one is scheduled via https://isitbandcampfriday.com/.

Bandcamp appear to offer a really good deal to artists (in comparison to other platforms) so I'm not too concerned about their percentage on other days of the year. But it occurred to me that if I wanted to maintain a habit of taking a local copy of the music I buy, perhaps I should schedule doing so on Bandcamp Fridays.

18 November, 2022 11:42AM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppSpdlog 0.0.10 on CRAN: New Features

A version 0.0.10 of RcppSpdlog is now on CRAN and in Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich.

This release continues on the path started less than two weeks ago with the RcppSpdlog 0.0.9 release. We continue to support both R and C++ access by adding a (simple) variadic template formatter exposing fmt::format() (by focusing on just string arguments). This can be accessed from R via the exact same formatting strings that fmt uses, and which we have come to like for its simplicity. Of course if one prefers a different string interpolation method, or plain sprintf(), or even paste: they all work as all that matters is that a character variable gets passed on. We also added a little bit of new documentation in the vignette.

The NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.10 (2022-11-17)

  • Support variadic templates with fmt::format

  • Add R formatting helper which converts arguments to character taking advantage of variadic template logger: fmt logging from R

  • Expand vignette

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 November, 2022 02:15AM

November 17, 2022

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Nageru 2.2.0 released

I've released version 2.2.0 of Nageru, my live video mixer. The “big ticket” item this time is AV1 support (through SVT-AV1), but it's still too experimental for production—in particular, as I've written about before, the client ecosystem isn't quite there yet.

But there's also a few cleanups, in particular a dependency removal. Every dependency is a long-term burden, and in retrospect, I shouldn't have taken on so many. (I am aware that this is ironic given the previous paragraph.) I'm wondering maybe if I should try to control my own fate a bit more and get rid of some.

Anyway, here's the changelog as usual:

Nageru and Futatabi 2.2.0, November 15th, 2022

  - Support AV1 output, via SVT-AV1. Note that this is still somewhat
    experimental, not the least because SVT-AV1's streaming support
    is not as mature as x264.

  - Remove the dependency on QCustomPlot.

  - Expose BlurEffect and UnsharpMaskEffect to the theme.

  - Clean up several rarely-unused command-line flags:
    - All the 10-bit flags are now collapsed to --10-bit.
    - Remove --http-uncompressed-video.
    - Remove the x264 VBV flags.
    - Hide --x264-speedcontrol-verbose.
    - Hide --no-flush-pbos.

  - Make a workaround for audio monitoring output under PipeWire.

  - Update CEF compatibility (tested with CEF 107).

Downloads as always on nageru.sesse.net.

17 November, 2022 11:21PM

Antoine Beaupré

A ZFS migration

In my tubman setup, I started using ZFS on an old server I had lying around. The machine is really old though (2011!) and it "feels" pretty slow. I want to see how much of that is ZFS and how much is the machine. Synthetic benchmarks show that ZFS may be slower than mdadm in RAID-10 or RAID-6 configuration, so I want to confirm that on a live workload: my workstation. Plus, I want easy, regular, high performance backups (with send/receive snapshots) and there's no way I'm going to use BTRFS because I find it too confusing and unreliable.

So off we go.


Since this is a conversion (and not a new install), our procedure is slightly different than the official documentation but otherwise it's pretty much in the same spirit: we're going to use ZFS for everything, including the root filesystem.

So, install the required packages, on the current system:

apt install --yes gdisk zfs-dkms zfs zfs-initramfs zfsutils-linux

We also tell DKMS that we need to rebuild the initrd when upgrading:

echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf


This is going to partition /dev/sdc with:

  • 1MB MBR / BIOS legacy boot
  • 512MB EFI boot
  • 1GB bpool, unencrypted pool for /boot
  • rest of the disk for zpool, the rest of the data

     sgdisk --zap-all /dev/sdc
     sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/sdc
     sgdisk     -n2:1M:+512M   -t2:EF00 /dev/sdc
     sgdisk     -n3:0:+1G      -t3:BF01 /dev/sdc
     sgdisk     -n4:0:0        -t4:BF00 /dev/sdc

That will look something like this:

    root@curie:/home/anarcat# sgdisk -p /dev/sdc
    Disk /dev/sdc: 1953525168 sectors, 931.5 GiB
    Model: ESD-S1C         
    Sector size (logical/physical): 512/512 bytes
    Disk identifier (GUID): [REDACTED]
    Partition table holds up to 128 entries
    Main partition table begins at sector 2 and ends at sector 33
    First usable sector is 34, last usable sector is 1953525134
    Partitions will be aligned on 16-sector boundaries
    Total free space is 14 sectors (7.0 KiB)

    Number  Start (sector)    End (sector)  Size       Code  Name
       1              48            2047   1000.0 KiB  EF02  
       2            2048         1050623   512.0 MiB   EF00  
       3         1050624         3147775   1024.0 MiB  BF01  
       4         3147776      1953525134   930.0 GiB   BF00

Unfortunately, we can't be sure of the sector size here, because the USB controller is probably lying to us about it. Normally, this smartctl command should tell us the sector size as well:

root@curie:~# smartctl -i /dev/sdb -qnoserial
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-14-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Model Family:     Western Digital Black Mobile
Device Model:     WDC WD10JPLX-00MBPT0
Firmware Version: 01.01H01
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May 17 13:33:04 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Above is the example of the builtin HDD drive. But the SSD device enclosed in that USB controller doesn't support SMART commands, so we can't trust that it really has 512 bytes sectors.

This matters because we need to tweak the ashift value correctly. We're going to go ahead the SSD drive has the common 4KB settings, which means ashift=12.

Note here that we are not creating a separate partition for swap. Swap on ZFS volumes (AKA "swap on ZVOL") can trigger lockups and that issue is still not fixed upstream. Ubuntu recommends using a separate partition for swap instead. But since this is "just" a workstation, we're betting that we will not suffer from this problem, after hearing a report from another Debian developer running this setup on their workstation successfully.

We do not recommend this setup though. In fact, if I were to redo this partition scheme, I would probably use LUKS encryption and setup a dedicated swap partition, as I had problems with ZFS encryption as well.

Creating pools

ZFS pools are somewhat like "volume groups" if you are familiar with LVM, except they obviously also do things like RAID-10. (Even though LVM can technically also do RAID, people typically use mdadm instead.)

In any case, the guide suggests creating two different pools here: one, in cleartext, for boot, and a separate, encrypted one, for the rest. Technically, the boot partition is required because the Grub bootloader only supports readonly ZFS pools, from what I understand. But I'm a little out of my depth here and just following the guide.

Boot pool creation

This creates the boot pool in readonly mode with features that grub supports:

    zpool create \
        -o cachefile=/etc/zfs/zpool.cache \
        -o ashift=12 -d \
        -o feature@async_destroy=enabled \
        -o feature@bookmarks=enabled \
        -o feature@embedded_data=enabled \
        -o feature@empty_bpobj=enabled \
        -o feature@enabled_txg=enabled \
        -o feature@extensible_dataset=enabled \
        -o feature@filesystem_limits=enabled \
        -o feature@hole_birth=enabled \
        -o feature@large_blocks=enabled \
        -o feature@lz4_compress=enabled \
        -o feature@spacemap_histogram=enabled \
        -o feature@zpool_checkpoint=enabled \
        -O acltype=posixacl -O canmount=off \
        -O compression=lz4 \
        -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
        -O mountpoint=/boot -R /mnt \
        bpool /dev/sdc3

I haven't investigated all those settings and just trust the upstream guide on the above.

Main pool creation

This is a more typical pool creation.

    zpool create \
        -o ashift=12 \
        -O encryption=on -O keylocation=prompt -O keyformat=passphrase \
        -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
        -O compression=zstd \
        -O relatime=on \
        -O canmount=off \
        -O mountpoint=/ -R /mnt \
        rpool /dev/sdc4

Breaking this down:

  • -o ashift=12: mentioned above, 4k sector size
  • -O encryption=on -O keylocation=prompt -O keyformat=passphrase: encryption, prompt for a password, default algorithm is aes-256-gcm, explicit in the guide, made implicit here
  • -O acltype=posixacl -O xattr=sa: enable ACLs, with better performance (not enabled by default)
  • -O dnodesize=auto: related to extended attributes, less compatibility with other implementations
  • -O compression=zstd: enable zstd compression, can be disabled/enabled by dataset to with zfs set compression=off rpool/example
  • -O relatime=on: classic atime optimisation, another that could be used on a busy server is atime=off
  • -O canmount=off: do not make the pool mount automatically with mount -a?
  • -O mountpoint=/ -R /mnt: mount pool on / in the future, but /mnt for now

Those settings are all available in zfsprops(8). Other flags are defined in zpool-create(8). The reasoning behind them is also explained in the upstream guide and some also in [the Debian wiki][]. Those flags were actually not used:

  • -O normalization=formD: normalize file names on comparisons (not storage), implies utf8only=on, which is a bad idea (and effectively meant my first sync failed to copy some files, including this folder from a supysonic checkout). and this cannot be changed after the filesystem is created. bad, bad, bad.

[the Debian wiki]: https://wiki.debian.org/ZFS#Advanced_Topics

Side note about single-disk pools

Also note that we're living dangerously here: single-disk ZFS pools are rumoured to be more dangerous than not running ZFS at all. The choice quote from this article is:

[...] any error can be detected, but cannot be corrected. This sounds like an acceptable compromise, but its actually not. The reason its not is that ZFS' metadata cannot be allowed to be corrupted. If it is it is likely the zpool will be impossible to mount (and will probably crash the system once the corruption is found). So a couple of bad sectors in the right place will mean that all data on the zpool will be lost. Not some, all. Also there's no ZFS recovery tools, so you cannot recover any data on the drives.

Compared with (say) ext4, where a single disk error can recovered, this is pretty bad. But we are ready to live with this with the idea that we'll have hourly offline snapshots that we can easily recover from. It's trade-off. Also, we're running this on a NVMe/M.2 drive which typically just blinks out of existence completely, and doesn't "bit rot" the way a HDD would.

Also, the FreeBSD handbook quick start doesn't have any warnings about their first example, which is with a single disk. So I am reassured at least.

Creating mount points

Next we create the actual filesystems, known as "datasets" which are the things that get mounted on mountpoint and hold the actual files.

  • this creates two containers, for ROOT and BOOT

     zfs create -o canmount=off -o mountpoint=none rpool/ROOT &&
     zfs create -o canmount=off -o mountpoint=none bpool/BOOT

    Note that it's unclear to me why those datasets are necessary, but they seem common practice, also used in this FreeBSD example. The OpenZFS guide mentions the Solaris upgrades and Ubuntu's zsys that use that container for upgrades and rollbacks. This blog post seems to explain a bit the layout behind the installer.

  • this creates the actual boot and root filesystems:

     zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian &&
     zfs mount rpool/ROOT/debian &&
     zfs create -o mountpoint=/boot bpool/BOOT/debian

    I guess the debian name here is because we could technically have multiple operating systems with the same underlying datasets.

  • then the main datasets:

     zfs create                                 rpool/home &&
     zfs create -o mountpoint=/root             rpool/home/root &&
     chmod 700 /mnt/root &&
     zfs create                                 rpool/var
  • exclude temporary files from snapshots:

     zfs create -o com.sun:auto-snapshot=false  rpool/var/cache &&
     zfs create -o com.sun:auto-snapshot=false  rpool/var/tmp &&
     chmod 1777 /mnt/var/tmp
  • and skip automatic snapshots in Docker:

     zfs create -o canmount=off                 rpool/var/lib &&
     zfs create -o com.sun:auto-snapshot=false  rpool/var/lib/docker

    Notice here a peculiarity: we must create rpool/var/lib to create rpool/var/lib/docker otherwise we get this error:

     cannot create 'rpool/var/lib/docker': parent does not exist

    ... and no, just creating /mnt/var/lib doesn't fix that problem. In fact, it makes things even more confusing because an existing directory shadows a mountpoint, which is the opposite of how things normally work.

    Also note that you will probably need to change storage driver in Docker, see the zfs-driver documentation for details but, basically, I did:

    echo '{ "storage-driver": "zfs" }' > /etc/docker/daemon.json

    Note that podman has the same problem (and similar solution):

    printf '[storage]\ndriver = "zfs"\n' > /etc/containers/storage.conf
  • make a tmpfs for /run:

     mkdir /mnt/run &&
     mount -t tmpfs tmpfs /mnt/run &&
     mkdir /mnt/run/lock

We don't create a /srv, as that's the HDD stuff.

Also mount the EFI partition:

mkfs.fat -F 32 /dev/sdc2 &&
mount /dev/sdc2 /mnt/boot/efi/

At this point, everything should be mounted in /mnt. It should look like this:

root@curie:~# LANG=C df -h -t zfs -t vfat
Filesystem            Size  Used Avail Use% Mounted on
rpool/ROOT/debian     899G  384K  899G   1% /mnt
bpool/BOOT/debian     832M  123M  709M  15% /mnt/boot
rpool/home            899G  256K  899G   1% /mnt/home
rpool/home/root       899G  256K  899G   1% /mnt/root
rpool/var             899G  384K  899G   1% /mnt/var
rpool/var/cache       899G  256K  899G   1% /mnt/var/cache
rpool/var/tmp         899G  256K  899G   1% /mnt/var/tmp
rpool/var/lib/docker  899G  256K  899G   1% /mnt/var/lib/docker
/dev/sdc2             511M  4.0K  511M   1% /mnt/boot/efi

Now that we have everything setup and mounted, let's copy all files over.

Copying files

This is a list of all the mounted filesystems

for fs in /boot/ /boot/efi/ / /home/; do
    echo "syncing $fs to /mnt$fs..." && 
    rsync -aSHAXx --info=progress2 --delete $fs /mnt$fs

You can check that the list is correct with:

mount -l -t ext4,btrfs,vfat | awk '{print $3}'

Note that we skip /srv as it's on a different disk.

On the first run, we had:

root@curie:~# for fs in /boot/ /boot/efi/ / /home/; do
        echo "syncing $fs to /mnt$fs..." && 
        rsync -aSHAXx --info=progress2 $fs /mnt$fs
syncing /boot/ to /mnt/boot/...
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/299)  
syncing /boot/efi/ to /mnt/boot/efi/...
     16,831,437 100%  184.14MB/s    0:00:00 (xfr#101, to-chk=0/110)
syncing / to /mnt/...
 28,019,293,280  94%   47.63MB/s    0:09:21 (xfr#703710, ir-chk=6748/839220)rsync: [generator] delete_file: rmdir(var/lib/docker) failed: Device or resource busy (16)
could not make way for new symlink: var/lib/docker
 34,081,267,990  98%   50.71MB/s    0:10:40 (xfr#736577, to-chk=0/867732)    
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
syncing /home/ to /mnt/home/...
rsync: [sender] readlink_stat("/home/anarcat/.fuse") failed: Permission denied (13)
 24,456,268,098  98%   68.03MB/s    0:05:42 (xfr#159867, ir-chk=6875/172377) 
file has vanished: "/home/anarcat/.cache/mozilla/firefox/s2hwvqbu.quantum/cache2/entries/B3AB0CDA9C4454B3C1197E5A22669DF8EE849D90"
199,762,528,125  93%   74.82MB/s    0:42:26 (xfr#1437846, ir-chk=1018/1983979)rsync: [generator] recv_generator: mkdir "/mnt/home/anarcat/dist/supysonic/tests/assets/\#346" failed: Invalid or incomplete multibyte or wide character (84)
*** Skipping any contents from this failed directory ***
315,384,723,978  96%   76.82MB/s    1:05:15 (xfr#2256473, to-chk=0/2993950)    
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

Note the failure to transfer that supysonic file? It turns out they had a weird filename in their source tree, since then removed, but still it showed how the utf8only feature might not be such a bad idea. At this point, the procedure was restarted all the way back to "Creating pools", after unmounting all ZFS filesystems (umount /mnt/run /mnt/boot/efi && umount -t zfs -a) and destroying the pool, which, surprisingly, doesn't require any confirmation (zpool destroy rpool).

The second run was cleaner:

root@curie:~# for fs in /boot/ /boot/efi/ / /home/; do
        echo "syncing $fs to /mnt$fs..." && 
        rsync -aSHAXx --info=progress2 --delete $fs /mnt$fs
syncing /boot/ to /mnt/boot/...
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/299)  
syncing /boot/efi/ to /mnt/boot/efi/...
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/110)  
syncing / to /mnt/...
 28,019,033,070  97%   42.03MB/s    0:10:35 (xfr#703671, ir-chk=1093/833515)rsync: [generator] delete_file: rmdir(var/lib/docker) failed: Device or resource busy (16)
could not make way for new symlink: var/lib/docker
 34,081,807,102  98%   44.84MB/s    0:12:04 (xfr#736580, to-chk=0/867723)    
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
syncing /home/ to /mnt/home/...
rsync: [sender] readlink_stat("/home/anarcat/.fuse") failed: Permission denied (13)
IO error encountered -- skipping file deletion
 24,043,086,450  96%   62.03MB/s    0:06:09 (xfr#151819, ir-chk=15117/172571)
file has vanished: "/home/anarcat/.cache/mozilla/firefox/s2hwvqbu.quantum/cache2/entries/4C1FDBFEA976FF924D062FB990B24B897A77B84B"
315,423,626,507  96%   67.09MB/s    1:14:43 (xfr#2256845, to-chk=0/2994364)    
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

Also note the transfer speed: we seem capped at 76MB/s, or 608Mbit/s. This is not as fast as I was expecting: the USB connection seems to be at around 5Gbps:

anarcat@curie:~$ lsusb -tv | head -4
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M
        ID 0b05:1932 ASUSTek Computer, Inc.

So it shouldn't cap at that speed. It's possible the USB adapter is failing to give me the full speed though. It's not the M.2 SSD drive either, as that has a ~500MB/s bandwidth, acccording to its spec.

At this point, we're about ready to do the final configuration. We drop to single user mode and do the rest of the procedure. That used to be shutdown now, but it seems like the systemd switch broke that, so now you can reboot into grub and pick the "recovery" option. Alternatively, you might try systemctl rescue, as I found out.

I also wanted to copy the drive over to another new NVMe drive, but that failed: it looks like the USB controller I have doesn't work with older, non-NVME drives.

Boot configuration

Now we need to enter the new system to rebuild the boot loader and initrd and so on.

First, we bind mounts and chroot into the ZFS disk:

mount --rbind /dev  /mnt/dev &&
mount --rbind /proc /mnt/proc &&
mount --rbind /sys  /mnt/sys &&
chroot /mnt /bin/bash

Next we add an extra service that imports the bpool on boot, to make sure it survives a zpool.cache destruction:

cat > /etc/systemd/system/zfs-import-bpool.service <<EOF

ExecStart=/sbin/zpool import -N -o cachefile=none bpool
# Work-around to preserve zpool cache:
ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache
ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache


Enable the service:

systemctl enable zfs-import-bpool.service

I had to trim down /etc/fstab and /etc/crypttab to only contain references to the legacy filesystems (/srv is still BTRFS!).

If we don't already have a tmpfs defined in /etc/fstab:

ln -s /usr/share/systemd/tmp.mount /etc/systemd/system/ &&
systemctl enable tmp.mount

Rebuild boot loader with support for ZFS, but also to workaround GRUB's missing zpool-features support:

grub-probe /boot | grep -q zfs &&
update-initramfs -c -k all &&
sed -i 's,GRUB_CMDLINE_LINUX.*,GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian",' /etc/default/grub &&

For good measure, make sure the right disk is configured here, for example you might want to tag both drives in a RAID array:

dpkg-reconfigure grub-pc

Install grub to EFI while you're there:

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --recheck --no-floppy

Filesystem mount ordering. The rationale here in the OpenZFS guide is a little strange, but I don't dare ignore that.

mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/bpool
touch /etc/zfs/zfs-list.cache/rpool
zed -F &

Verify that zed updated the cache by making sure these are not empty:

cat /etc/zfs/zfs-list.cache/bpool
cat /etc/zfs/zfs-list.cache/rpool

Once the files have data, stop zed:

Press Ctrl-C.

Fix the paths to eliminate /mnt:

sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*

Snapshot initial install:

zfs snapshot bpool/BOOT/debian@install
zfs snapshot rpool/ROOT/debian@install

Exit chroot:



One last sync was done in rescue mode:

for fs in /boot/ /boot/efi/ / /home/; do
    echo "syncing $fs to /mnt$fs..." && 
    rsync -aSHAXx --info=progress2 --delete $fs /mnt$fs

Then we unmount all filesystems:

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

Reboot, swap the drives, and boot in ZFS. Hurray!


This is a test that was ran in single-user mode using fio and the Ars Technica recommended tests, which are:

  • Single 4KiB random write process:

     fio --name=randwrite4k1x --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
  • 16 parallel 64KiB random write processes:

     fio --name=randwrite64k16x --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
  • Single 1MiB random write process:

     fio --name=randwrite1m1x --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

Strangely, that's not exactly what the author, Jim Salter, did in his actual test bench used in the ZFS benchmarking article. The first thing is there's no read test at all, which is already pretty strange. But also it doesn't include stuff like dropping caches or repeating results.

So here's my variation, which i called fio-ars-bench.sh for now. It just batches a bunch of fio tests, one by one, 60 seconds each. It should take about 12 minutes to run, as there are 3 pair of tests, read/write, with and without async.

My bias, before building, running and analysing those results is that ZFS should outperform the traditional stack on writes, but possibly not on reads. It's also possible it outperforms it on both, because it's a newer drive. A new test might be possible with a new external USB drive as well, although I doubt I will find the time to do this.


All tests were done on WD blue SN550 drives, which claims to be able to push 2400MB/s read and 1750MB/s write. An extra drive was bought to move the LVM setup from a WDC WDS500G1B0B-00AS40 SSD, a WD blue M.2 2280 SSD that was at least 5 years old, spec'd at 560MB/s read, 530MB/s write. Benchmarks were done on the M.2 SSD drive but discarded so that the drive difference is not a factor in the test.

In practice, I'm going to assume we'll never reach those numbers because we're not actually NVMe (this is an old workstation!) so the bottleneck isn't the disk itself. For our purposes, it might still give us useful results.

Rescue test, LUKS/LVM/ext4

Those tests were performed with everything shutdown, after either entering the system in rescue mode, or by reaching that target with:

systemctl rescue

The network might have been started before or after the test as well:

systemctl start systemd-networkd

So it should be fairly reliable as basically nothing else is running.

Raw numbers, from the ?job-curie-lvm.log, converted to MiB/s and manually merged:

test read I/O read IOPS write I/O write IOPS
rand4k4g1x 39.27 10052 212.15 54310
rand4k4g1x--fsync=1 39.29 10057 2.73 699
rand64k256m16x 1297.00 20751 1068.57 17097
rand64k256m16x--fsync=1 1290.90 20654 353.82 5661
rand1m16g1x 315.15 315 563.77 563
rand1m16g1x--fsync=1 345.88 345 157.01 157

Peaks are at about 20k IOPS and ~1.3GiB/s read, 1GiB/s write in the 64KB blocks with 16 jobs.

Slowest is the random 4k block sync write at an abysmal 3MB/s and 700 IOPS The 1MB read/write tests have lower IOPS, but that is expected.

Rescue test, ZFS

This test was also performed in rescue mode.

Raw numbers, from the ?job-curie-zfs.log, converted to MiB/s and manually merged:

test read I/O read IOPS write I/O write IOPS
rand4k4g1x 77.20 19763 27.13 6944
rand4k4g1x--fsync=1 76.16 19495 6.53 1673
rand64k256m16x 1882.40 30118 70.58 1129
rand64k256m16x--fsync=1 1865.13 29842 71.98 1151
rand1m16g1x 921.62 921 102.21 102
rand1m16g1x--fsync=1 908.37 908 64.30 64

Peaks are at 1.8GiB/s read, also in the 64k job like above, but much faster. The write is, as expected, much slower at 70MiB/s (compared to 1GiB/s!), but it should be noted the sync write doesn't degrade performance compared to async writes (although it's still below the LVM 300MB/s).


Really, ZFS has trouble performing in all write conditions. The random 4k sync write test is the only place where ZFS outperforms LVM in writes, and barely (7MiB/s vs 3MiB/s). Everywhere else, writes are much slower, sometimes by an order of magnitude.

And before some ZFS zealot jumps in talking about the SLOG or some other cache that could be added to improved performance, I'll remind you that those numbers are on a bare bones NVMe drive, pretty much as fast storage as you can find on this machine. Adding another NVMe drive as a cache probably will not improve write performance here.

Still, those are very different results than the tests performed by Salter which shows ZFS beating traditional configurations in all categories but uncached 4k reads (not writes!). That said, those tests are very different from the tests I performed here, where I test writes on a single disk, not a RAID array, which might explain the discrepancy.

Also, note that neither LVM or ZFS manage to reach the 2400MB/s read and 1750MB/s write performance specification. ZFS does manage to reach 82% of the read performance (1973MB/s) and LVM 64% of the write performance (1120MB/s). LVM hits 57% of the read performance and ZFS hits barely 6% of the write performance.

Overall, I'm a bit disappointed in the ZFS write performance here, I must say. Maybe I need to tweak the record size or some other ZFS voodoo, but I'll note that I didn't have to do any such configuration on the other side to kick ZFS in the pants...

Real world experience

This section document not synthetic backups, but actual real world workloads, comparing before and after I switched my workstation to ZFS.

Docker performance

I had the feeling that running some git hook (which was firing a Docker container) was "slower" somehow. It seems that, at runtime, ZFS backends are significant slower than their overlayfs/ext4 equivalent:

May 16 14:42:52 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87\x2dinit-merged.mount: Succeeded.
May 16 14:42:52 curie systemd[5161]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87\x2dinit-merged.mount: Succeeded.
May 16 14:42:52 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:53 curie dockerd[1723]: time="2022-05-16T14:42:53.087219426-04:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10 pid=151170
May 16 14:42:53 curie systemd[1]: Started libcontainer container af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10.
May 16 14:42:54 curie systemd[1]: docker-af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10.scope: Succeeded.
May 16 14:42:54 curie dockerd[1723]: time="2022-05-16T14:42:54.047297800-04:00" level=info msg="shim disconnected" id=af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10
May 16 14:42:54 curie dockerd[998]: time="2022-05-16T14:42:54.051365015-04:00" level=info msg="ignoring event" container=af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 16 14:42:54 curie systemd[2444]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[5161]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[2444]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:54 curie systemd[5161]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:54 curie systemd[1]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.

Translating this:

  • container setup: ~1 second
  • container runtime: ~1 second
  • container teardown: ~1 second
  • total runtime: 2-3 seconds

Obviously, those timestamps are not quite accurate enough to make precise measurements...

After I switched to ZFS:

mai 30 15:31:39 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf\x2dinit.mount: Succeeded. 
mai 30 15:31:39 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf\x2dinit.mount: Succeeded. 
mai 30 15:31:40 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded. 
mai 30 15:31:40 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded. 
mai 30 15:31:41 curie dockerd[3199]: time="2022-05-30T15:31:41.551403693-04:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142 pid=141080 
mai 30 15:31:41 curie systemd[1]: run-docker-runtime\x2drunc-moby-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142-runc.ZVcjvl.mount: Succeeded. 
mai 30 15:31:41 curie systemd[5287]: run-docker-runtime\x2drunc-moby-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142-runc.ZVcjvl.mount: Succeeded. 
mai 30 15:31:41 curie systemd[1]: Started libcontainer container 42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142. 
mai 30 15:31:45 curie systemd[1]: docker-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142.scope: Succeeded. 
mai 30 15:31:45 curie dockerd[3199]: time="2022-05-30T15:31:45.883019128-04:00" level=info msg="shim disconnected" id=42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142 
mai 30 15:31:45 curie dockerd[1726]: time="2022-05-30T15:31:45.883064491-04:00" level=info msg="ignoring event" container=42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" 
mai 30 15:31:45 curie systemd[1]: run-docker-netns-e45f5cf5f465.mount: Succeeded. 
mai 30 15:31:45 curie systemd[5287]: run-docker-netns-e45f5cf5f465.mount: Succeeded. 
mai 30 15:31:45 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded. 
mai 30 15:31:45 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded.

That's double or triple the run time, from 2 seconds to 6 seconds. Most of the time is spent in run time, inside the container. Here's the breakdown:

  • container setup: ~2 seconds
  • container run: ~4 seconds
  • container teardown: ~1 second
  • total run time: about ~6-7 seconds

That's a two- to three-fold increase! Clearly something is going on here that I should tweak. It's possible that code path is less optimized in Docker. I also worry about podman, but apparently it also supports ZFS backends. Possibly it would perform better, but at this stage I wouldn't have a good comparison: maybe it would have performed better on non-ZFS as well...


While doing the offsite backups (below), the system became somewhat "sluggish". I felt everything was slow, and I estimate it introduced ~50ms latency in any input device.

Arguably, those are all USB and the external drive was connected through USB, but I suspect the ZFS drivers are not as well tuned with the scheduler as the regular filesystem drivers...

Recovery procedures

For test purposes, I unmounted all systems during the procedure:

umount /mnt/boot/efi /mnt/boot/run
umount -a -t zfs
zpool export -a

And disconnected the drive, to see how I would recover this system from another Linux system in case of a total motherboard failure.

To import an existing pool, plug the device, then import the pool with an alternate root, so it doesn't mount over your existing filesystems, then you mount the root filesystem and all the others:

zpool import -l -a -R /mnt &&
zfs mount rpool/ROOT/debian &&
zfs mount -a &&
mount /dev/sdc2 /mnt/boot/efi &&
mount -t tmpfs tmpfs /mnt/run &&
mkdir /mnt/run/lock

Offsite backup

Part of the goal of using ZFS is to simplify and harden backups. I wanted to experiment with shorter recovery times — specifically both point in time recovery objective and recovery time objective — and faster incremental backups.

This is, therefore, part of my backup services.

This section documents how an external NVMe enclosure was setup in a pool to mirror the datasets from my workstation.

The final setup should include syncoid copying datasets to the backup server regularly, but I haven't finished that configuration yet.


The above partitioning procedure used sgdisk, but I couldn't figure out how to do this with sgdisk, so this uses sfdisk to dump the partition from the first disk to an external, identical drive:

sfdisk -d /dev/nvme0n1 | sfdisk --no-reread /dev/sda --force

Pool creation

This is similar to the main pool creation, except we tweaked a few bits after changing the upstream procedure:

zpool create \
        -o cachefile=/etc/zfs/zpool.cache \
        -o ashift=12 -d \
        -o feature@async_destroy=enabled \
        -o feature@bookmarks=enabled \
        -o feature@embedded_data=enabled \
        -o feature@empty_bpobj=enabled \
        -o feature@enabled_txg=enabled \
        -o feature@extensible_dataset=enabled \
        -o feature@filesystem_limits=enabled \
        -o feature@hole_birth=enabled \
        -o feature@large_blocks=enabled \
        -o feature@lz4_compress=enabled \
        -o feature@spacemap_histogram=enabled \
        -o feature@zpool_checkpoint=enabled \
        -O acltype=posixacl -O xattr=sa \
        -O compression=lz4 \
        -O devices=off \
        -O relatime=on \
        -O canmount=off \
        -O mountpoint=/boot -R /mnt \
        bpool-tubman /dev/sdb3

The change from the main boot pool are:

Main pool creation is:

zpool create \
        -o ashift=12 \
        -O encryption=on -O keylocation=prompt -O keyformat=passphrase \
        -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
        -O compression=zstd \
        -O relatime=on \
        -O canmount=off \
        -O mountpoint=/ -R /mnt \
        rpool-tubman /dev/sdb4

First sync

I used syncoid to copy all pools over to the external device. syncoid is a thing that's part of the sanoid project which is specifically designed to sync snapshots between pool, typically over SSH links but it can also operate locally.

The sanoid command had a --readonly argument to simulate changes, but syncoid didn't so I tried to fix that with an upstream PR.

It seems it would be better to do this by hand, but this was much easier. The full first sync was:

root@curie:/home/anarcat# ./bin/syncoid -r  bpool bpool-tubman

CRITICAL ERROR: Target bpool-tubman exists but has no snapshots matching with bpool!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

          NOTE: Target bpool-tubman dataset is < 64MB used - did you mistakenly run
                `zfs create bpool-tubman` on the target? ZFS initial
                replication must be to a NON EXISTENT DATASET, which will
                then be CREATED BY the initial replication process.

INFO: Sending oldest full snapshot bpool/BOOT@test (~ 42 KB) to new target filesystem:
44.2KiB 0:00:00 [4.19MiB/s] [========================================================================================================================] 103%            
INFO: Updating new target filesystem with incremental bpool/BOOT@test ... syncoid_curie_2022-05-30:12:50:39 (~ 4 KB):
2.13KiB 0:00:00 [ 114KiB/s] [===============================================================>                                                         ] 53%            
INFO: Sending oldest full snapshot bpool/BOOT/debian@install (~ 126.0 MB) to new target filesystem:
 126MiB 0:00:00 [ 308MiB/s] [=======================================================================================================================>] 100%            
INFO: Updating new target filesystem with incremental bpool/BOOT/debian@install ... syncoid_curie_2022-05-30:12:50:39 (~ 113.4 MB):
 113MiB 0:00:00 [ 315MiB/s] [=======================================================================================================================>] 100%

root@curie:/home/anarcat# ./bin/syncoid -r  rpool rpool-tubman

CRITICAL ERROR: Target rpool-tubman exists but has no snapshots matching with rpool!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

          NOTE: Target rpool-tubman dataset is < 64MB used - did you mistakenly run
                `zfs create rpool-tubman` on the target? ZFS initial
                replication must be to a NON EXISTENT DATASET, which will
                then be CREATED BY the initial replication process.

INFO: Sending oldest full snapshot rpool/ROOT@syncoid_curie_2022-05-30:12:50:51 (~ 69 KB) to new target filesystem:
44.2KiB 0:00:00 [2.44MiB/s] [===========================================================================>                                             ] 63%            
INFO: Sending oldest full snapshot rpool/ROOT/debian@install (~ 25.9 GB) to new target filesystem:
25.9GiB 0:03:33 [ 124MiB/s] [=======================================================================================================================>] 100%            
INFO: Updating new target filesystem with incremental rpool/ROOT/debian@install ... syncoid_curie_2022-05-30:12:50:52 (~ 3.9 GB):
3.92GiB 0:00:33 [ 119MiB/s] [======================================================================================================================>  ] 99%            
INFO: Sending oldest full snapshot rpool/home@syncoid_curie_2022-05-30:12:55:04 (~ 276.8 GB) to new target filesystem:
 277GiB 0:27:13 [ 174MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/home/root@syncoid_curie_2022-05-30:13:22:19 (~ 2.2 GB) to new target filesystem:
2.22GiB 0:00:25 [90.2MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/var@syncoid_curie_2022-05-30:13:22:47 (~ 5.6 GB) to new target filesystem:
5.56GiB 0:00:32 [ 176MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/var/cache@syncoid_curie_2022-05-30:13:23:22 (~ 627.3 MB) to new target filesystem:
 627MiB 0:00:03 [ 169MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/var/lib@syncoid_curie_2022-05-30:13:23:28 (~ 69 KB) to new target filesystem:
44.2KiB 0:00:00 [1.40MiB/s] [===========================================================================>                                             ] 63%            
INFO: Sending oldest full snapshot rpool/var/lib/docker@syncoid_curie_2022-05-30:13:23:28 (~ 442.6 MB) to new target filesystem:
 443MiB 0:00:04 [ 103MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254 (~ 6.3 MB) to new target filesystem:
6.49MiB 0:00:00 [12.9MiB/s] [========================================================================================================================] 102%            
INFO: Updating new target filesystem with incremental rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254 ... syncoid_curie_2022-05-30:13:23:34 (~ 4 KB):
1.52KiB 0:00:00 [27.6KiB/s] [============================================>                                                                            ] 38%            
INFO: Sending oldest full snapshot rpool/var/lib/flatpak@syncoid_curie_2022-05-30:13:23:36 (~ 2.0 GB) to new target filesystem:
2.00GiB 0:00:17 [ 115MiB/s] [=======================================================================================================================>] 100%            
INFO: Sending oldest full snapshot rpool/var/tmp@syncoid_curie_2022-05-30:13:23:55 (~ 57.0 MB) to new target filesystem:
61.8MiB 0:00:01 [45.0MiB/s] [========================================================================================================================] 108%            
INFO: Clone is recreated on target rpool-tubman/var/lib/docker/ed71ddd563a779ba6fb37b3b1d0cc2c11eca9b594e77b4b234867ebcb162b205 based on rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254
INFO: Sending oldest full snapshot rpool/var/lib/docker/ed71ddd563a779ba6fb37b3b1d0cc2c11eca9b594e77b4b234867ebcb162b205@syncoid_curie_2022-05-30:13:23:58 (~ 218.6 MB) to new target filesystem:
 219MiB 0:00:01 [ 151MiB/s] [=======================================================================================================================>] 100%

Funny how the CRITICAL ERROR doesn't actually stop syncoid and it just carries on merrily doing when it's telling you it's "cowardly refusing to destroy your existing target"... Maybe that's because my pull request broke something though...

During the transfer, the computer was very sluggish: everything feels like it has ~30-50ms latency extra:

anarcat@curie:sanoid$ LANG=C top -b  -n 1 | head -20
top - 13:07:05 up 6 days,  4:01,  1 user,  load average: 16.13, 16.55, 11.83
Tasks: 606 total,   6 running, 598 sleeping,   0 stopped,   2 zombie
%Cpu(s): 18.8 us, 72.5 sy,  1.2 ni,  5.0 id,  1.2 wa,  0.0 hi,  1.2 si,  0.0 st
MiB Mem :  15898.4 total,   1387.6 free,  13170.0 used,   1340.8 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1319.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     70 root      20   0       0      0      0 S  83.3   0.0   6:12.67 kswapd0
4024878 root      20   0  282644  96432  10288 S  44.4   0.6   0:11.43 puppet
3896136 root      20   0   35328  16528     48 S  22.2   0.1   2:08.04 mbuffer
3896135 root      20   0   10328    776    168 R  16.7   0.0   1:22.93 zfs
3896138 root      20   0   10588    788    156 R  16.7   0.0   1:49.30 zfs
    350 root       0 -20       0      0      0 R  11.1   0.0   1:03.53 z_rd_int
    351 root       0 -20       0      0      0 S  11.1   0.0   1:04.15 z_rd_int
3896137 root      20   0    4384    352    244 R  11.1   0.0   0:44.73 pv
4034094 anarcat   30  10   20028  13960   2428 S  11.1   0.1   0:00.70 mbsync
4036539 anarcat   20   0    9604   3464   2408 R  11.1   0.0   0:00.04 top
    352 root       0 -20       0      0      0 S   5.6   0.0   1:03.64 z_rd_int
    353 root       0 -20       0      0      0 S   5.6   0.0   1:03.64 z_rd_int
    354 root       0 -20       0      0      0 S   5.6   0.0   1:04.01 z_rd_int

I wonder how much of that is due to syncoid, particularly because I often saw mbuffer and pv in there which are not strictly necessary to do those kind of operations, as far as I understand.

Once that's done, export the pools to disconnect the drive:

zpool export bpool-tubman
zpool export rpool-tubman

Raw disk benchmark

Copied the 512GB SSD/M.2 device to another 1024GB NVMe/M.2 device:

anarcat@curie:~$ sudo dd if=/dev/sdb of=/dev/sdc bs=4M status=progress conv=fdatasync
499944259584 octets (500 GB, 466 GiB) copiés, 1713 s, 292 MB/s
119235+1 enregistrements lus
119235+1 enregistrements écrits
500107862016 octets (500 GB, 466 GiB) copiés, 1719,93 s, 291 MB/s

... while both over USB, whoohoo 300MB/s!


ZFS should be monitoring your pools regularly. Normally, the [[!debman zed]] daemon monitors all ZFS events. It is the thing that will report when a scrub failed, for example. See this configuration guide.

Scrubs should be regularly scheduled to ensure consistency of the pool. This can be done in newer zfsutils-linux versions (bullseye-backports or bookworm) with one of those, depending on the desired frequency:

systemctl enable zfs-scrub-weekly@rpool.timer --now

systemctl enable zfs-scrub-monthly@rpool.timer --now

When the scrub runs, if it finds anything it will send an event which will get picked up by the zed daemon which will then send a notification, see below for an example.

TODO: deploy on curie, if possible (probably not because no RAID) TODO: this should be in Puppet

Scrub warning example

So what happens when problems are found? Here's an example of how I dealt with an error I received.

After setting up another server (tubman) with ZFS, I eventually ended up getting a warning from the ZFS toolchain.

Date: Sun, 09 Oct 2022 00:58:08 -0400
From: root <root@anarc.at>
To: root@anarc.at
Subject: ZFS scrub_finish event for rpool on tubman

ZFS has finished a scrub:

   eid: 39536
 class: scrub_finish
  host: tubman
  time: 2022-10-09 00:58:07-0400
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 00:33:57 with 0 errors on Sun Oct  9 00:58:07 2022

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb4    ONLINE       0     1     0
            sdc4    ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

This, in itself, is a little worrisome. But it helpfully links to this more detailed documentation (and props up there: the link still works) which explains this is a "minor" problem (something that could be included in the report).

In this case, this happened on a server setup on 2021-04-28, but the disks and server hardware are much older. The server itself (marcos v1) was built around 2011, over 10 years ago now. The hard drive in question is:

root@tubman:~# smartctl -i -qnoserial /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-15-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Model Family:     Seagate BarraCuda 3.5
Device Model:     ST4000DM004-2CV104
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Oct 11 11:02:32 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Some more SMART stats:

root@tubman:~# smartctl -a -qnoserial /dev/sdb | grep -e  Head_Flying_Hours -e Power_On_Hours -e Total_LBA -e 'Sector Sizes'
Sector Sizes:     512 bytes logical, 4096 bytes physical
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       12464 (206 202 0)
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10966h+55m+23.757s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       21107792664
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3201579750

That's over a year of power on, which shouldn't be so bad. It has written about 10TB of data (21107792664 LBAs * 512 byte/LBA), which is about two full writes. According to its specification, this device is supposed to support 55 TB/year of writes, so we're far below spec. Note that are still far from the "non-recoverable read error per bits" spec (1 per 10E15), as we've basically read 13E12 bits (3201579750 LBAs * 512 byte/LBA = 13E12 bits).

It's likely this disk was made in 2018, so it is in its fourth year.

Interestingly, /dev/sdc is also a Seagate drive, but of a different series:

root@tubman:~# smartctl -qnoserial  -i /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-15-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Model Family:     Seagate BarraCuda 3.5
Device Model:     ST4000DM004-2CV104
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Oct 11 11:21:35 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

It has seen much more reads than the other disk which is also interesting:

root@tubman:~# smartctl -a -qnoserial /dev/sdc | grep -e  Head_Flying_Hours -e Power_On_Hours -e Total_LBA -e 'Sector Sizes'
Sector Sizes:     512 bytes logical, 4096 bytes physical
  9 Power_On_Hours          0x0032   059   059   000    Old_age   Always       -       36240
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       33994h+10m+52.118s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       30730174438
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       51894566538

That's 4 years of Head_Flying_Hours, and over 4 years (4 years and 48 days) of Power_On_Hours. The copyright date on that drive's specs goes back to 2016, so it's a much older drive.

SMART self-test succeeded.

Remaining issues

  • TODO: move send/receive backups to offsite host, see also zfs for alternatives to syncoid/sanoid there
  • TODO: setup backup cron job (or timer?)
  • TODO: swap still not setup on curie, see zfs
  • TODO: document this somewhere: bpool and rpool are both pools and datasets. that's pretty confusing, but also very useful because it allows for pool-wide recursive snapshots, which are used for the backup system

fio improvements

I really want to improve my experience with fio. Right now, I'm just cargo-culting stuff from other folks and I don't really like it. stressant is a good example of my struggles, in the sense that it doesn't really work that well for disk tests.

I would love to have just a single .fio job file that lists multiple jobs to run serially. For example, this file describes the above workload pretty well:

# cargo-culting Salter
# no need to drop caches, done by default
# invalidate=1

# Single 4KiB random read/write process


# 16 parallel 64KiB random read/write processes:


# Single 1MiB random read/write process


... except the jobs are actually started in parallel, even though they are stonewall'd, as far as I can tell by the reports. I sent a mail to the fio mailing list for clarification.

It looks like the jobs are started in parallel, but actual (correctly) run serially. It seems like this might just be a matter of reporting the right timestamps in the end, although it does feel like starting all the processes (even if not doing any work yet) could skew the results.

Hangs during procedure

During the procedure, it happened a few times where any ZFS command would completely hang. It seems that using an external USB drive to sync stuff didn't work so well: sometimes it would reconnect under a different device (from sdc to sdd, for example), and this would greatly confuse ZFS.

Here, for example, is sdd reappearing out of the blue:

May 19 11:22:53 curie kernel: [  699.820301] scsi host4: uas
May 19 11:22:53 curie kernel: [  699.820544] usb 2-1: authorized to connect
May 19 11:22:53 curie kernel: [  699.922433] scsi 4:0:0:0: Direct-Access     ROG      ESD-S1C          0    PQ: 0 ANSI: 6
May 19 11:22:53 curie kernel: [  699.923235] sd 4:0:0:0: Attached scsi generic sg2 type 0
May 19 11:22:53 curie kernel: [  699.923676] sd 4:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
May 19 11:22:53 curie kernel: [  699.923788] sd 4:0:0:0: [sdd] Write Protect is off
May 19 11:22:53 curie kernel: [  699.923949] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 11:22:53 curie kernel: [  699.924149] sd 4:0:0:0: [sdd] Optimal transfer size 33553920 bytes
May 19 11:22:53 curie kernel: [  699.961602]  sdd: sdd1 sdd2 sdd3 sdd4
May 19 11:22:53 curie kernel: [  699.996083] sd 4:0:0:0: [sdd] Attached SCSI disk

Next time I run a ZFS command (say zpool list), the command completely hangs (D state) and this comes up in the logs:

May 19 11:34:21 curie kernel: [ 1387.914843] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=71344128 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914859] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=205565952 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914874] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=272789504 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914906] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=270336 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.914932] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=1073225728 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.914948] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=1073487872 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.915165] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=272793600 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.915183] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=339853312 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.915648] WARNING: Pool 'bpool' has encountered an uncorrectable I/O failure and has been suspended.
May 19 11:34:21 curie kernel: [ 1387.915648] 
May 19 11:37:25 curie kernel: [ 1571.558614] task:txg_sync        state:D stack:    0 pid:  997 ppid:     2 flags:0x00004000
May 19 11:37:25 curie kernel: [ 1571.558623] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.558640]  __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.558650]  schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.558670]  schedule_timeout+0x8b/0x140
May 19 11:37:25 curie kernel: [ 1571.558675]  ? __next_timer_interrupt+0x110/0x110
May 19 11:37:25 curie kernel: [ 1571.558678]  io_schedule_timeout+0x4c/0x80
May 19 11:37:25 curie kernel: [ 1571.558689]  __cv_timedwait_common+0x12b/0x160 [spl]
May 19 11:37:25 curie kernel: [ 1571.558694]  ? add_wait_queue_exclusive+0x70/0x70
May 19 11:37:25 curie kernel: [ 1571.558702]  __cv_timedwait_io+0x15/0x20 [spl]
May 19 11:37:25 curie kernel: [ 1571.558816]  zio_wait+0x129/0x2b0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.558929]  dsl_pool_sync+0x461/0x4f0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559032]  spa_sync+0x575/0xfa0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559138]  ? spa_txg_history_init_io+0x101/0x110 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559245]  txg_sync_thread+0x2e0/0x4a0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559354]  ? txg_fini+0x240/0x240 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559366]  thread_generic_wrapper+0x6f/0x80 [spl]
May 19 11:37:25 curie kernel: [ 1571.559376]  ? __thread_exit+0x20/0x20 [spl]
May 19 11:37:25 curie kernel: [ 1571.559379]  kthread+0x11b/0x140
May 19 11:37:25 curie kernel: [ 1571.559382]  ? __kthread_bind_mask+0x60/0x60
May 19 11:37:25 curie kernel: [ 1571.559386]  ret_from_fork+0x22/0x30
May 19 11:37:25 curie kernel: [ 1571.559401] task:zed             state:D stack:    0 pid: 1564 ppid:     1 flags:0x00000000
May 19 11:37:25 curie kernel: [ 1571.559404] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.559409]  __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.559412]  ? __kmalloc_node+0x141/0x2b0
May 19 11:37:25 curie kernel: [ 1571.559417]  schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.559420]  schedule_preempt_disabled+0xa/0x10
May 19 11:37:25 curie kernel: [ 1571.559424]  __mutex_lock.constprop.0+0x133/0x460
May 19 11:37:25 curie kernel: [ 1571.559435]  ? nvlist_xalloc.part.0+0x68/0xc0 [znvpair]
May 19 11:37:25 curie kernel: [ 1571.559537]  spa_all_configs+0x41/0x120 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559644]  zfs_ioc_pool_configs+0x17/0x70 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559752]  zfsdev_ioctl_common+0x697/0x870 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559758]  ? _copy_from_user+0x28/0x60
May 19 11:37:25 curie kernel: [ 1571.559860]  zfsdev_ioctl+0x53/0xe0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559866]  __x64_sys_ioctl+0x83/0xb0
May 19 11:37:25 curie kernel: [ 1571.559869]  do_syscall_64+0x33/0x80
May 19 11:37:25 curie kernel: [ 1571.559873]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 19 11:37:25 curie kernel: [ 1571.559876] RIP: 0033:0x7fcf0ef32cc7
May 19 11:37:25 curie kernel: [ 1571.559878] RSP: 002b:00007fcf0e181618 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May 19 11:37:25 curie kernel: [ 1571.559881] RAX: ffffffffffffffda RBX: 000055b212f972a0 RCX: 00007fcf0ef32cc7
May 19 11:37:25 curie kernel: [ 1571.559883] RDX: 00007fcf0e181640 RSI: 0000000000005a04 RDI: 000000000000000b
May 19 11:37:25 curie kernel: [ 1571.559885] RBP: 00007fcf0e184c30 R08: 00007fcf08016810 R09: 00007fcf08000080
May 19 11:37:25 curie kernel: [ 1571.559886] R10: 0000000000080000 R11: 0000000000000246 R12: 000055b212f972a0
May 19 11:37:25 curie kernel: [ 1571.559888] R13: 0000000000000000 R14: 00007fcf0e181640 R15: 0000000000000000
May 19 11:37:25 curie kernel: [ 1571.559980] task:zpool           state:D stack:    0 pid:11815 ppid:  3816 flags:0x00004000
May 19 11:37:25 curie kernel: [ 1571.559983] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.559988]  __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.559992]  schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.559995]  io_schedule+0x42/0x70
May 19 11:37:25 curie kernel: [ 1571.560004]  cv_wait_common+0xac/0x130 [spl]
May 19 11:37:25 curie kernel: [ 1571.560008]  ? add_wait_queue_exclusive+0x70/0x70
May 19 11:37:25 curie kernel: [ 1571.560118]  txg_wait_synced_impl+0xc9/0x110 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560223]  txg_wait_synced+0xc/0x40 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560325]  spa_export_common+0x4cd/0x590 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560430]  ? zfs_log_history+0x9c/0xf0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560537]  zfsdev_ioctl_common+0x697/0x870 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560543]  ? _copy_from_user+0x28/0x60
May 19 11:37:25 curie kernel: [ 1571.560644]  zfsdev_ioctl+0x53/0xe0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560649]  __x64_sys_ioctl+0x83/0xb0
May 19 11:37:25 curie kernel: [ 1571.560653]  do_syscall_64+0x33/0x80
May 19 11:37:25 curie kernel: [ 1571.560656]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 19 11:37:25 curie kernel: [ 1571.560659] RIP: 0033:0x7fdc23be2cc7
May 19 11:37:25 curie kernel: [ 1571.560661] RSP: 002b:00007ffc8c792478 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May 19 11:37:25 curie kernel: [ 1571.560664] RAX: ffffffffffffffda RBX: 000055942ca49e20 RCX: 00007fdc23be2cc7
May 19 11:37:25 curie kernel: [ 1571.560666] RDX: 00007ffc8c792490 RSI: 0000000000005a03 RDI: 0000000000000003
May 19 11:37:25 curie kernel: [ 1571.560667] RBP: 00007ffc8c795e80 R08: 00000000ffffffff R09: 00007ffc8c792310
May 19 11:37:25 curie kernel: [ 1571.560669] R10: 000055942ca49e30 R11: 0000000000000246 R12: 00007ffc8c792490
May 19 11:37:25 curie kernel: [ 1571.560671] R13: 000055942ca49e30 R14: 000055942aed2c20 R15: 00007ffc8c795a40

Here's another example, where you see the USB controller bleeping out and back into existence:

mai 19 11:38:39 curie kernel: usb 2-1: USB disconnect, device number 2
mai 19 11:38:39 curie kernel: sd 4:0:0:0: [sdd] Synchronizing SCSI cache
mai 19 11:38:39 curie kernel: sd 4:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
mai 19 11:39:25 curie kernel: INFO: task zed:1564 blocked for more than 241 seconds.
mai 19 11:39:25 curie kernel:       Tainted: P          IOE     5.10.0-14-amd64 #1 Debian 5.10.113-1
mai 19 11:39:25 curie kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mai 19 11:39:25 curie kernel: task:zed             state:D stack:    0 pid: 1564 ppid:     1 flags:0x00000000
mai 19 11:39:25 curie kernel: Call Trace:
mai 19 11:39:25 curie kernel:  __schedule+0x282/0x870
mai 19 11:39:25 curie kernel:  ? __kmalloc_node+0x141/0x2b0
mai 19 11:39:25 curie kernel:  schedule+0x46/0xb0
mai 19 11:39:25 curie kernel:  schedule_preempt_disabled+0xa/0x10
mai 19 11:39:25 curie kernel:  __mutex_lock.constprop.0+0x133/0x460
mai 19 11:39:25 curie kernel:  ? nvlist_xalloc.part.0+0x68/0xc0 [znvpair]
mai 19 11:39:25 curie kernel:  spa_all_configs+0x41/0x120 [zfs]
mai 19 11:39:25 curie kernel:  zfs_ioc_pool_configs+0x17/0x70 [zfs]
mai 19 11:39:25 curie kernel:  zfsdev_ioctl_common+0x697/0x870 [zfs]
mai 19 11:39:25 curie kernel:  ? _copy_from_user+0x28/0x60
mai 19 11:39:25 curie kernel:  zfsdev_ioctl+0x53/0xe0 [zfs]
mai 19 11:39:25 curie kernel:  __x64_sys_ioctl+0x83/0xb0
mai 19 11:39:25 curie kernel:  do_syscall_64+0x33/0x80
mai 19 11:39:25 curie kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
mai 19 11:39:25 curie kernel: RIP: 0033:0x7fcf0ef32cc7
mai 19 11:39:25 curie kernel: RSP: 002b:00007fcf0e181618 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
mai 19 11:39:25 curie kernel: RAX: ffffffffffffffda RBX: 000055b212f972a0 RCX: 00007fcf0ef32cc7
mai 19 11:39:25 curie kernel: RDX: 00007fcf0e181640 RSI: 0000000000005a04 RDI: 000000000000000b
mai 19 11:39:25 curie kernel: RBP: 00007fcf0e184c30 R08: 00007fcf08016810 R09: 00007fcf08000080
mai 19 11:39:25 curie kernel: R10: 0000000000080000 R11: 0000000000000246 R12: 000055b212f972a0
mai 19 11:39:25 curie kernel: R13: 0000000000000000 R14: 00007fcf0e181640 R15: 0000000000000000
mai 19 11:39:25 curie kernel: INFO: task zpool:11815 blocked for more than 241 seconds.
mai 19 11:39:25 curie kernel:       Tainted: P          IOE     5.10.0-14-amd64 #1 Debian 5.10.113-1
mai 19 11:39:25 curie kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mai 19 11:39:25 curie kernel: task:zpool           state:D stack:    0 pid:11815 ppid:  2621 flags:0x00004004
mai 19 11:39:25 curie kernel: Call Trace:
mai 19 11:39:25 curie kernel:  __schedule+0x282/0x870
mai 19 11:39:25 curie kernel:  schedule+0x46/0xb0
mai 19 11:39:25 curie kernel:  io_schedule+0x42/0x70
mai 19 11:39:25 curie kernel:  cv_wait_common+0xac/0x130 [spl]
mai 19 11:39:25 curie kernel:  ? add_wait_queue_exclusive+0x70/0x70
mai 19 11:39:25 curie kernel:  txg_wait_synced_impl+0xc9/0x110 [zfs]
mai 19 11:39:25 curie kernel:  txg_wait_synced+0xc/0x40 [zfs]
mai 19 11:39:25 curie kernel:  spa_export_common+0x4cd/0x590 [zfs]
mai 19 11:39:25 curie kernel:  ? zfs_log_history+0x9c/0xf0 [zfs]
mai 19 11:39:25 curie kernel:  zfsdev_ioctl_common+0x697/0x870 [zfs]
mai 19 11:39:25 curie kernel:  ? _copy_from_user+0x28/0x60
mai 19 11:39:25 curie kernel:  zfsdev_ioctl+0x53/0xe0 [zfs]
mai 19 11:39:25 curie kernel:  __x64_sys_ioctl+0x83/0xb0
mai 19 11:39:25 curie kernel:  do_syscall_64+0x33/0x80
mai 19 11:39:25 curie kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
mai 19 11:39:25 curie kernel: RIP: 0033:0x7fdc23be2cc7
mai 19 11:39:25 curie kernel: RSP: 002b:00007ffc8c792478 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
mai 19 11:39:25 curie kernel: RAX: ffffffffffffffda RBX: 000055942ca49e20 RCX: 00007fdc23be2cc7
mai 19 11:39:25 curie kernel: RDX: 00007ffc8c792490 RSI: 0000000000005a03 RDI: 0000000000000003
mai 19 11:39:25 curie kernel: RBP: 00007ffc8c795e80 R08: 00000000ffffffff R09: 00007ffc8c792310
mai 19 11:39:25 curie kernel: R10: 000055942ca49e30 R11: 0000000000000246 R12: 00007ffc8c792490
mai 19 11:39:25 curie kernel: R13: 000055942ca49e30 R14: 000055942aed2c20 R15: 00007ffc8c795a40

I understand those are rather extreme conditions: I would fully expect the pool to stop working if the underlying drives disappear. What doesn't seem acceptable is that a command would completely hang like this.


See the zfs documentation for more information about ZFS, and tubman for another installation and migration procedure.

17 November, 2022 09:25PM

November 16, 2022

Ian Jackson

Stop writing Rust linked list libraries!


Don’t write a Rust linked list library: they are hard to do well, and usually useless.

Use VecDeque, which is great. If you actually need more than VecDeque can do, use one of the handful of libraries that actually offer a significantly more useful API.

If you are writing your own data structure, check if someone has done it already, and consider slotmap or generation_arena, (or maybe Rc/Arc).


Survey of Rust linked list libraries

I have updated my Survey of Rust linked list libraries.


In 2019 I was writing plag-mangler, a tool for planar graph layout.

I needed a data structure. Naturally I looked for a library to help. I didn’t find what I needed, so I wrote rc-dlist-deque. However, on the way I noticed an inordinate number of linked list libraries written in Rust. Most all of these had no real reason for existing. Even the one in the Rust standard library is useless.


Now I have redone the survey. The results are depressing. In 2019 there were 5 libraries which, in my opinion, were largely useless. In late 2022 there are now thirteen linked list libraries that ought probably not ever to be used. And, a further eight libraries for which there are strictly superior alternatives. Many of these have the signs of projects whose authors are otherwise competent: proper documentation, extensive APIs, and so on.

There is one new library which is better for some applications than those available in 2019. (I’m referring to generational_token_list, which makes a plausible alternative to dlv-list which I already recommended in 2019.)

Why are there so many poor Rust linked list libraries ?

Linked lists and Rust do not go well together. But (and I’m guessing here) I presume many people are taught in programming school that a linked list is a fundamental data structure; people are often even asked to write one as a teaching exercise. This is a bad idea in Rust. Or maybe they’ve heard that writing linked lists in Rust is hard and want to prove they can do it.

Double-ended queues

One of the main applications for a linked list in a language like C, is a queue, where you put items in at one end, and take them out at the other. The Rust standard library has a data structure for that, VecDeque.

Five of the available libraries:

  • Have an API which is a subset of that of VecDeque: basically, pushing and popping elements at the front and back.
  • Have worse performance for most applications than VecDeque,
  • Are less mature, less available, less well tested, etc., than VecDeque, simply because VecDeque is in the Rust Standard Library.

For these you could, and should, just use VecDeque instead.

The Cursor concept

A proper linked list lets you identify and hold onto an element in the middle of the list, and cheaply insert and remove elements there.

Rust’s ownership and borrowing rules make this awkward. One idea that people have many times reinvented and reimplemented, is to have a Cursor type, derived from the list, which is a reference to an element, and permits insertion and removal there.

Eight libraries have implemented this in the obvious way. However, there is a serious API limitation:

To prevent a cursor being invalidated (e.g. by deletion of the entry it points to) you can’t modify the list while the cursor exists. You can only have one cursor (that can be used for modification) at a time.

The practical effect of this is that you cannot retain cursors. You can make and use such a cursor for a particular operation, but you must dispose of it soon. Attempts to do otherwise will see you losing a battle with the borrow checker.

If that’s good enough, then you could just use a VecDeque and use array indices instead of the cursors. It’s true that deleting or adding elements in the middle involves a lot of copying, but your algorithm is O(n) even with the single-cursor list libraries, because it must first walk the cursor to the desired element.

Formally, I believe any algorithm using these exclusive cursors can be rewritten, in an obvious way, to simply iterate and/or copy from the start or end (as one can do with VecDeque) without changing the headline O() performance characteristics.

IMO the savings available from avoiding extra copies etc. are not worth the additional dependency, unsafe code, and so on, especially as there are other ways of helping with that (e.g. boxing the individual elements).

Even if you don’t find that convincing, generational_token_list and dlv_list are strictly superior since they offer a more flexible and convenient API and better performance, and rely on much less unsafe code.

Rustic approaches to pointers-to-and-between-nodes data structures

Most of the time a VecDeque is great. But if you actually want to hold onto (perhaps many) references to the middle of the list, and later modify it through those references, you do need something more. This is a specific case of a general class of problems where the naive approach (use Rust references to the data structure nodes) doesn’t work well.

But there is a good solution:

Keep all the nodes in an array (a Vec<Option<T>> or similar) and use the index in the array as your node reference. This is fast, and quite ergonomic, and neatly solves most of the problems. If you are concerned that bare indices might cause confusion, as newly inserted elements would reuse indices, add a per-index generation count.

These approaches have been neatly packaged up in libraries like slab, slotmap, generational-arena and thunderdome. And they have been nicely applied to linked lists by the authors of generational_token_list. and dlv-list.

The alternative for nodey data structures in safe Rust: Rc/Arc

Of course, you can just use Rust’s “interior mutability” and reference counting smart pointers, to directly implement the data structure of your choice.

In many applications, a single-threaded data structure is fine, in which case Rc and Cell/RefCell will let you write safe code, with cheap refcount updates and runtime checks inserted to defend against unexpected aliasing, use-after-free, etc.

I took this approach in rc-dlist-deque, because I wanted each node to be able to be on multiple lists.

Rust’s package ecosystem demonstrating software’s NIH problem

The Rust ecosystem is full of NIH libraries of all kinds. In my survey, there are: five good options; seven libraries which are plausible, but just not as good as the alternatives; and fourteen others.

There is a whole rant I could have about how the whole software and computing community is pathologically neophilic. Often we seem to actively resist reusing ideas, let alone code; and are ignorant and dismissive of what has gone before. As a result, we keep solving the same problems, badly - making the same mistakes over and over again. In some subfields, working software, or nearly working software, is frequently replaced with something worse, maybe more than once.

One aspect of this is a massive cultural bias towards rewriting rather than reusing, let alone fixing and using.

Many people can come out of a degree, trained to be a programmer, and have no formal training in selecting and evaluating software; this is even though working effectively with computers requires making good use of everyone else’s work.

If one isn’t taught these skills (when and how to search for prior art, how to choose between dependencies, and so on) one must learn it on the job. The result is usually an ad-hoc and unsystematic approach, often dominated by fashion rather than engineering.

The package naming paradox

The more experienced and competent programmer is aware of all the other options that exist - after all they have evaluated other choices before writing their own library.

So they will call their library something like generational_token_list or vecdeque-stableix.

Whereas the novice straight out of a pre-Rust programming course just thinks what they are doing is the one and only obvious thing (even though it’s a poor idea) and hasn’t even searched for a previous implementation. So they call their package something obvious like “linked list”.

As a result, the most obvious names seem to refer to the least useful libraries.

Edited 2022-11-16 23:55 UTC to update numbers of libraries in various categories following updates to the survey (including updates prompted by feedback received after this post first published).

comment count unavailable comments

16 November, 2022 11:55PM

November 14, 2022

Antoine Beaupré

Looking at Wayland terminal emulators

Back in 2018, I made a two part series about terminal emulators that was actually pretty painful to write. So I'm not going to retry this here, not at all. Especially since I'm not submitting this to the excellent LWN editors so I can get away with not being very good at writing. Phew.

Still, it seems my future self will thank me for collecting my thoughts on the terminal emulators I have found out about since I wrote that article. Back then, Wayland was not quite at the level where it is now, being the default in Fedora (2016), Debian (2019), RedHat (2019), and Ubuntu (2021). Also, a bunch of folks thought they would solve everything by using OpenGL for rendering. Let's see how things stack up.


In the previous article, I touched on those projects:

Terminal Changes since review
Alacritty releases! scrollback, better latency, URL launcher, clipboard support, still not in Debian, but close
GNOME Terminal not much? couldn't find a changelog
Konsole outdated changelog, color, image previews, clickable files, multi-input, SSH plugin, sixel images
mlterm long changelog but: supports console mode (like GNU screen?!), Wayland support through libvte, sixel graphics, zmodem, mosh (!)
pterm changes: Wayland support
st unparseable changelog, suggests scroll(1) or scrollback.patch for scrollback now
Terminator moved to GitHub, Python 3 support, not being dead
urxvt no significant changes, a single release, still in CVS!
Xfce Terminal hard to parse changelog, presumably some improvements to paste safety?
xterm notoriously hard to parse changelog, improvements to paste safety (disallowedPasteControls), fonts, clipboard improvements?

After writing those articles, bizarrely, I was still using rxvt even though it did not come up as shiny as I would have liked. The colors problems were especially irritating.

I briefly played around with Konsole and xterm, and eventually switched to XTerm as my default x-terminal-emulator "alternative" in my Debian system, while writing this.

I quickly noticed why I had stopped using it: clickable links are a huge limitation. I ended up adding keybindings to open URLs in a command. There's another keybinding to dump the history into a command. Neither are as satisfactory as just clicking a damn link.


Figuring out my requirements is actually a pretty hard thing to do. In my last reviews, I just tried a bunch of stuff and collected everything, but a lot of things (like tab support) I don't actually care about. So here's a set of things I actually do care about:

  • latency
  • resource usage
  • proper clipboard support, that is:
    • mouse selection and middle button uses PRIMARY
    • control-shift-c and control-shift-v for CLIPBOARD
  • true color support
  • no known security issues
  • active project
  • paste protection
  • clickable URLs
  • scrollback
  • font resize
  • non-destructive text-wrapping (ie. resizing a window doesn't drop scrollback history)
  • proper unicode support (at least latin-1, ideally "everything")
  • good emoji support (at least showing them, ideally "nicely"), which involves font fallback

Latency is particularly something I wonder about in Wayland. Kitty seem to have been pretty dilligent at doing latency tests, claiming 35ms with a hardware-based latency tester and 7ms with typometer, but it's unclear how those would come up in Wayland because, as far as I know, typometer does not support Wayland.


Those are the projects I am considering.

  • darktile - GPU rendering, Unicode support, themable, ligatures (optional), Sixel, window transparency, clickable URLs, true color support, not in Debian
  • foot - Wayland only, daemon-mode, sixel images, scrollback search, true color, font resize, URLs not clickable, but keyboard-driven selection, proper clipboard support, in Debian
  • havoc - minimal, scrollback, configurable keybindings, not in Debian
  • sakura - libvte, Wayland support, tabs, no menu bar, original libvte gangster, dynamic font size, probably supports Wayland, in Debian
  • termonad - Haskell? in Debian
  • wez - Rust, Wayland, multiplexer, ligatures, scrollback search, clipboard support, bracketed paste, panes, tabs, serial port support, Sixel, Kitty, iTerm graphics, built-in SSH client (!?), not in Debian
  • XTerm - status quo, no Wayland port obviously
  • zutty: OpenGL rendering, true color, clipboard support, small codebase, no Wayland support, crashes on bremner's, in Debian

Candidates not considered


I would really, really like to use Alacritty, but it's still not packaged in Debian, and they haven't fully addressed the latency issues although, to be fair, maybe it's just an impossible task. Once it's packaged in Debian, maybe I'll reconsider.


Kitty is a "fast, feature-rich, GPU based", with ligatures, emojis, hyperlinks, pluggable, scriptable, tabs, layouts, history, file transfer over SSH, its own graphics system, and probably much more I'm forgetting. It's packaged in Debian.

So I immediately got two people commenting (on IRC) that they use Kitty and are pretty happy with it. I've been hesitant in directly talking about Kitty publicly, but since it's likely there will be a pile-up of similar comments, I'll just say why it's not the first in my list, even if it might, considering it's packaged in Debian and otherwise checks all the boxes.

I don't trust the Kitty code. Kitty was written by the same author as Calibre, which has a horrible security history and generally really messy source code. I have tried to do LTS work on Calibre, and have mostly given up on the idea of making that program secure in any way. See calibre for the details on that.

Now it's possible Kitty is different: it's quite likely the author has gotten some experience writing (and maintaining for so long!) Calibre over the years. But I would be more optimistic if the author's reaction to the security issues were more open and proactive.

I've also seen the same reaction play out on Kitty's side of things. As anyone who worked on writing or playing with non-XTerm terminal emulators, it's quite a struggle to make something (bug-for-bug) compatible with everything out there. And Kitty is in that uncomfortable place right now where it diverges from the canon and needs its own entry in the ncurses database. I don't remember the specifics, but the author also managed to get into fights with those people as well, which I don't feel is reassuring for the project going forward.

If security and compatibility wasn't such big of a deal for me, I wouldn't mind so much, but I'll need a lot of convincing before I consider Kitty more seriously at this point.

Next steps

It seems like Arch Linux defaults to foot in Sway, and I keep seeing it everywhere, so it is probably my next thing to try, if/when I switch to Wayland.

One major problem with foot is that it's yet another terminfo entry. They did make it into ncurses (patch 2021-07-31) but only after Debian bullseye stable was released. So expect some weird compatibility issues when connecting to any other system that is older or the same as stable (!). Thankfully, there is a foot-terminfo package that is available starting from Debian bullseye.

One question mark with all Wayland terminals, and Foot in particular, is how much latency they introduce in the rendering pipeline. The foot performance and benchmarks look excellent, but do not include latency benchmarks.

No conclusion

So I guess that's all I've got so far, I may try alacritty if it hits Debian, or foot if I switch to Wayland, but for now I'm hacking in xterm still. Happy to hear ideas in the comments.

Stay tuned for more happy days.

Update: a few months after this article was written, I did switch a laptop to Wayland, and as part of that migration, adopted foot. It's great, and latency is barely noticeable. I recommend people try it out when they switch to Wayland, and, inversely, try out Wayland if only to try out Foot, it's a great terminal emulator.

14 November, 2022 06:29PM

November 12, 2022

hackergotchi for Wouter Verhelst

Wouter Verhelst

Day 3 of the Debian Videoteam Sprint in Cape Town

The Debian Videoteam has been sprinting in Cape Town, South Africa -- mostly because with Stefano here for a few months, four of us (Jonathan, Kyle, Stefano, and myself) actually are in the country on a regular basis. In addition to that, two more members of the team (Nicolas and Louis-Philippe) are joining the sprint remotely (from Paris and Montreal).

Videoteam sprint

(Kyle and Stefano working on things, with me behind the camera and Jonathan busy elsewhere.)

We've made loads of progress! Some highlights:

  • We did a lot of triaging of outstanding bugs and merge requests against our ansible repository. Stale issues were closed, merge requests have been merged (or closed when they weren't relevant anymore), and new issues that we found while working on them were fixed. We also improved our test coverage for some of our ansible roles, and modernized as well as improved the way our documentation is built. (Louis-Philippe, Stefano, Kyle, Wouter, Nicolas)
  • Some work was done on SReview, our video review and transcode tool: I fixed up the metadata export code and did some other backend work, while Stefano worked a bit on the frontend, bringing it up to date to use bootstrap 4, and adding client-side filtering using vue. Future work on this will allow editing various things from the webinterface -- currently that requires issuing SQL commands directly. (Wouter and Stefano)
  • Jonathan explored new features in OBS. We've been using OBS for our "loopy" setup since DebConf20, which is used for the slightly more interactive sponsor loop that is shown in between talks. The result is that we'll be able to simplify and improve that setup in future (mini)DebConf instances. (Jonathan)
  • Kyle had a look at options for capturing hardware. We currently use Opsis boards, but they are not an ideal solution, and we are exploring alternatives. (Kyle)
  • Some package uploads happened! libmedia-convert-perl will now (hopefully) migrate to testing; and if all goes well, a new version of SReview will be available in unstable soon.

The sprint isn't over yet (we're continuing until Sunday), but loads of things have already happened. Stay tuned!

12 November, 2022 09:06AM

Craig Small

WordPress 6.1

Debian will soon have WordPress version 6.1 I’m not really sure of the improvements, but there is a new 2023 theme as part of the update.

They really weren’t mucking around when they said the 6.0.3 security release would be short-lived.

The updates seem to be focused on content creation and making the formatting do what content creators want it to do. For me, I need headings 1 and 2 , paragraphs and preformatted text.

12 November, 2022 07:00AM by dropbear

November 11, 2022

Reproducible Builds

Reproducible Builds in October 2022

Welcome to the Reproducible Builds report for October 2022! In these reports we attempt to outline the most important things that we have been up to over the past month.

As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Our in-person summit this year was held in the past few days in Venice, Italy. Activity and news from the summit will therefore be covered in next month’s report!

A new article related to reproducible builds was recently published in the 2023 IEEE Symposium on Security and Privacy. Titled Taxonomy of Attacks on Open-Source Software Supply Chains and authored by Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais, their paper:

[…] proposes a general taxonomy for attacks on opensource supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution.

Taking the form of an attack tree, the paper covers 107 unique vectors linked to 94 real world supply-chain incidents which is then mapped to 33 mitigating safeguards including, of course, reproducible builds:

Reproducible Builds received a very high utility rating (5) from 10 participants (58.8%), but also a high-cost rating (4 or 5) from 12 (70.6%). One expert commented that a ”reproducible build like used by Solarwinds now, is a good measure against tampering with a single build system” and another claimed this ”is going to be the single, biggest barrier”.

It was noticed this month that Solarwinds published a whitepaper back in December 2021 in order to:

[…] illustrate a concerning new reality for the software industry and illuminates the increasingly sophisticated threats made by outside nation-states to the supply chains and infrastructure on which we all rely.

The 12-month anniversary of the 2020 “Solarwinds attack” (which SolarWinds Worldwide LLC itself calls the “SUNBURST” attack) was, of course, the likely impetus for publication.

Whilst collaborating on making the Cyrus IMAP server reproducible, Ellie Timoney asked why the Reproducible Builds testing framework uses two remarkably distinctive build paths when attempting to flush out builds that vary on the absolute system path in which they were built. In the case of the Cyrus IMAP server, these happened to be:

  • /build/1st/cyrus-imapd-3.6.0~beta3/
  • /build/2/cyrus-imapd-3.6.0~beta3/2nd/

Asked why they vary in three different ways, Chris Lamb listed in detail the motivation behind to each difference.

On our mailing list this month:

The Reproducible Builds project is delighted to welcome openEuler to the Involved projects page []. openEuler is Linux distribution developed by Huawei, a counterpart to it’s more commercially-oriented EulerOS.


Colin Watson wrote about his experience towards making the databases generated by the man-db UNIX manual page indexing tool:

One of the people working on [reproducible builds] noticed that man-db’s database files were an obstacle to [reproducibility]: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn’t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs.

Colin goes on to describe his approach to solving the problem, including fixing various fits of internal caching, and he ends his post with “None of this is particularly glamorous work, but it paid off”.

Vagrant Cascadian announced on our mailing list another online sprint to help “clear the huge backlog of reproducible builds patches submitted” by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but another was held on October 6th, and another small one on October 20th. This resulted in the following progress:

41 reviews of Debian packages were added, 62 were updated and 12 were removed this month adding to our knowledge about identified issues. A number of issue types were updated too. [1][]

Lastly, Luca Boccassi submitted a patch to debhelper, a set of tools used in the packaging of the majority of Debian packages. The patch addressed an issue in the dh_installsysusers utility so that the postinst post-installation script that debhelper generates the same data regardless of the underlying filesystem ordering.

Other distributions

F-Droid is a community-run app store that provides free software applications for Android phones. This month, F-Droid changed their documentation and guidance to now explicitly encourage RB for new apps [][], and FC Stegerman created an extremely in-depth issue on GitLab concerning the APK signing block. You can read more about F-Droid’s approach to reproducibility in our July 2022 interview with Hans-Christoph Steiner of the F-Droid Project.

In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 224 and 225 to Debian:

  • Add support for comparing the text content of HTML files using html2text. []
  • Add support for detecting ordering-only differences in XML files. []
  • Fix an issue with detecting ordering differences. []
  • Use the capitalised version of “Ordering” consistently everywhere in output. []
  • Add support for displaying font metadata using ttx(1) from the fonttools suite. []
  • Testsuite improvements:

    • Temporarily allow the stable-po pipeline to fail in the CI. []
    • Rename the order1.diff test fixture to json_expected_ordering_diff. []
    • Tidy the JSON tests. []
    • Use assert_diff over get_data and an manual assert within the XML tests. []
    • Drop the ALLOWED_TEST_FILES test; it was mostly just annoying. []
    • Tidy the tests/test_source.py file. []

Chris Lamb also added a link to diffoscope’s OpenBSD packaging on the diffoscope.org homepage [] and Mattia Rizzolo fix an test failure that was occurring under with LLVM 15 [].

Testing framework

The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

  • Run the logparse tool to analyse results on the Debian Edu build logs. []
  • Install btop(1) on all nodes running Debian. []
  • Switch Arch Linux from using SHA1 to SHA256. []
  • When checking Debian debstrap jobs, correctly log the tool usage. []
  • Cleanup more task-related temporary directory names when testing Debian packages. [][]
  • Use the cdebootstrap-static binary for the 2nd runs of the cdebootstrap tests. []
  • Drop a workaround when testing OpenWrt and coreboot as the issue in diffoscope has now been fixed. []
  • Turn on an rm(1) warning into an “info”-level message. []
  • Special case the osuosl168 node for running Debian bookworm already. [][]
  • Use the new non-free-firmware suite on the o168 node. []

In addition, Mattia Rizzolo made the following changes:

  • Ensure that 2nd build has a merged /usr. []
  • Only reconfigure the usrmerge package on Debian bookworm and above. []
  • Fix bc(1) syntax in the computation of the percentage of unreproducible packages in the dashboard. [][][]
  • In the index_suite_ pages, order the package status to be the same order of the menu. []
  • Pass the --distribution parameter to the pbuilder utility. []

Finally, Roland Clobus continued his work on testing Live Debian images. In particular, he extended the maintenance script to warn when workspace directories cannot be deleted. []

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

11 November, 2022 03:56PM

Enrico Zini

Sharing argparse arguments with subcommands

argparse subcommands are great, but they have a quirk in which options are only available right after the subcommand that define them.

So, if you for example add the --verbose / -v argument to your main parser, and you have subcommands, you need to give the -v option before the subcommand name. For example, given this script:

import argparse

parser = argparse.ArgumentParser(description="test")
parser.add_argument("-v", "--verbose", action="store_true")
subparsers = parser.add_subparsers(dest="handler", required=True)

args = parser.parse_args()

You get this behaviour:

$ ./mycmd test
$ ./mycmd -v test
$ ./mycmd test -v
usage: mycmd [-h] [-v] {test} ...
mycmd: error: unrecognized arguments: -v

This sometimes makes sense, and many other times it's really annoying, since the user has to remember at which level an option was defined.

Last night some pieces clicked in my head, and I created a not-too-dirty ArgumentParser subclass that adds a shared option to arguments, that propagates them to subparsers:

from hacks import argparse

parser = argparse.ArgumentParser(description="test")
parser.add_argument("-v", "--verbose", action="store_true", shared=True)
subparsers = parser.add_subparsers(dest="handler", required=True)

args = parser.parse_args()

And finally, -v can be given at all levels:

$ ./mycmd test
$ ./mycmd -v test
$ ./mycmd test -v

It even works recursively, forwarding arguments to sub-subparsers, but at the moment it can only do it for store_true and store kind of arguments.

11 November, 2022 12:51PM

November 10, 2022

hackergotchi for Shirish Agarwal

Shirish Agarwal

The Road to Gandolfo, Webforms, Hearing Loss info & Mum’s Birthday.

The Road to Gandolfo

I think I had read this book almost 10-12 years back and somehow ended up reading it up again. Apparently, he had put this fiction, story, book under some other pen name earlier. It is possible that I might have read it under that name and hence forgotten all about it. This book/story is full of innuendo, irony, sarcasm and basically the thrill of life. There are two main characters in the book, the first is General Mackenzie who has spent almost 3 to 4 decades being a spy/a counterintelligence expert in the Queen’s service. And while he outclasses them all even at the ripe age of 50, he is thrown out under the pretext of conduct unbecoming of an officer.

The other main character is Sam Devereaux. This gentleman is an army lawyer and is basically counting the days when he completes his tour of duty as a military lawyer and start his corporate civil law with somebody he knows. It is much to his dismay that while under a week is left for his tour of duty to be left over he is summoned to try and extradite General MacKenzie who has been put on house arrest. Apparently, in China there was a sculpture of a great Chinese gentleman in the nude. For reasons unknown or rather not being shared herein, he basically breaks part of the sculpture. This of course, enrages the Chinese and they call it a diplomatic accident and try to put the General into house arrest. Unfortunately for both the General and his captors, he decides to escape/go for it. While he does succeed entering the American embassy, he finds himself to be person non-grata and is thrown back outside where the Chinese recapture him.

This is where the Embassy & the Govt. decide it would be better if somehow the General could be removed from China permanently so he doesn’t cause any further diplomatic accidents. In order to do that Sam’s services are bought.

Now in order to understand the General, Sam learns that he has 4 ex-wives. He promptly goes and meet them to understand why the general behaved as he did. He apparently also peed on the American flag. To his surprise, all the four ex-wives are still very much in with the general. During the course of interviewing the ladies he is seduced by them and also gives names to their chests in order to differentiate between each one of them. Later he is seduced by the eldest of the four wives and they spend the evening together.

Next day Sam meets and is promptly manhandled by the general and the diplomatic papers are seen by the general. After meeting the general and the Chinese counterpart, they quickly agree to extradite him as they do not know how to keep the general control. During his stay of house arrest, the General reads one of the ‘communist rags’ as he puts it and gets the idea to kidnap the pope and that forms the basis of the story.

Castel Gandolfo seems to be a real place which is in Italy and is apparently is the papal residence where s/he goes to reside every winter. The book is written in 1976 hence in the book, the General decides to form a corporation for which he would raise funds in order to make the kidnapping. The amount in 1976 was 40 million dollars and it was a big sum, to be with times, let’s think of say 40 billion dollars so gets the scale of things.

Now while a part of me wants to tell the rest of the story, the story isn’t really mine to tell. Read The Road to Gandolfo for the rest. While I can’t guarantee you much, I can say you might find yourself constantly amused by the antics of both the General, Sam and the General’s ex-wives. There are also a few minute characters that you will meet on the way, hope you discover them and enjoy it immensely as I have.

One thing I have to say, while I was reading it, I very much got vibes of ‘Not a penny more, not a penny less’ by Jeffrey Archer. As shared before, lots of twists and turns, enjoy the ride 🙂


Webforms are nothing but a form you fill on the web or www. Webforms are and were a thing from early 90s to today. I was supposed to register for https://www.swavlambancard.gov.in/ almost a month back but procrastinated till few couple of days back and with good reason. I was hoping one of my good friends would help me but they had their own thing. So finally, I tried to fill the form few days back. It took me almost 30 odd attempts to finally fill the form and was given an enrollment number. Why it took me 30 odd attempts and with what should tell you the reason –

  1. I felt like I was filling the form from 1990’s rather than today because –
  2. The form doesn’t know either its state or saves data during a session – This lesson has been learned a long time back by almost all service providers except Govt. of India. Both the browsers on a mobile as well as desktop can save data during session. If you don’t know what I mean by that go to about:preferences#privacy in Firefox and look at Manage Data. There you will find most sites do put some data along with cookies arguably to help make your web experience better. Chrome or Chromium has the same thing perhaps shared under a different name but its the same thing. But that is not all.
  3. None of the fields have any verification. The form is of 3 pages. The verification at the end of the document doesn’t tell you what is wrong and what needs to be corrected. Really think on this, I am on a 24″ LED monitor and I’m filling the form and I had to do it at least 20-30 times before it was accepted. And guess what, I have no clue even about why it was selected. The same data, the same everything and after the nth time it accepted. Now if I am facing such a problem when I have some idea how technology works somewhat how are people who are trying to fill this form on 6″ mobiles supposed to do? And many of them not at all clued in technology as I am.

I could go on outlining many of the issues that I faced but they are all similar in many ways the problems faced while filling the ‘NEW’ Income Tax forms. Of course the New Income Tax portal is a whole ball-game in itself as it gives new errors every time instead of solving them. Most C.A.’s have turned to third-party xml tools that enable you to upload xml compliant data to the ‘New’ Income tax portal but this is for businesses and those who can afford it. Again, even that is in a sort of messy state but that is a whole another tale altogether.

One of the reasons to my mind why the forms are designed the way they are so that people go to specific cybercafes or get individual people to fill and upload it and make more money. I was told to go to a specific cybercafe and meet a certain individual and he asked for INR 500/- to do the work. While I don’t have financial problems, I was more worried about my data going into the wrong hands. But I can see a very steady way to make money without doing much hard work.

Hearing Loss info.

Now because I had been both to Kamla Nehru Hospital as well as Sasoon and especially the deaf department, I saw many kids with half-formed ears. I had asked the doctors and they had shared this is due to malnutrition. We do know that women during pregnancies need more calories, more everything as they are eating for two bodies, not one. And this is large-scale, apparently more than 5 percent of population have children like this. And this number was of 2014, what is it today nobody knows. I also came to know that at least for some people like me, due to Covid they became deaf. I had asked the doctors if they knew of people who had become deaf due to Covid. They basically replied in the negative as they don’t have the resources to monitor the same. The Govt. has an idea of health ID but just like Aadhar has to many serious sinister implications. Somebody had shared with me a long time back that in India systems work inspite of Govt. machinery rather than because of it. Meaning that the Government itself ties itself into several knots and then people have to be creative to try and figure a way out to help people. I found another issue while dealing with them.

Apparently, even though I have 60% hearing loss I would be given a certificate of 40% hearing loss and they call it Temporary Progressive Loss. I saw almost all the people who had come, many of them having far severe defencies than me getting the same/similar certificate. All of them got Temporary Progressive. Some of the cases were real puzzling. For e.g. I met another Agarwal who had a severe accident few months ago and there is some kind of paralysis & bone issue. The doctors have given up but even that gentleman was given Temporary Progressive. From what little I could understand, the idea is that over period if there is possibility of things becoming better then it should be given. Another gentleman suffered a case of dwarfism. Even he was given the same certificate. Think there have been orders from above so that people even having difficulties are not helped. Another point if you look in a macro sense, it presents a somewhat rosy picture. If someone were to debunk the Govt. data either from India or abroad then from GOI perspective they have an ‘agenda’ even though the people who are suffering are our brothers and sisters 😦 And all of this is because I can read, write, articulate. Perhaps many of them may not even have a voice or a platform.

Even to get this temporary progressive disability certificate there is more than 4 months of running from one place to the other, 4 months of culmination of work. This I can share and tell from my experience, who knows how much else others might have suffered for the same. In my case a review will happen after 5 years, in most other cases they have given only 1 year. Of course, this does justify people’s jobs and perhaps partly it may be due to that. Such are times where I really miss that I am unable to hear otherwise could have fleshed out lot more other people’s sufferings.

And just so people know/understand this is happening in the heart of the city whose population easily exceeds 6 million plus and is supposed to be a progressive city. I do appreciate and understand the difficulties that the doctors are placed under.

Mum’s Birthday & Social Engineering.

While I don’t want to get into details, in the last couple of weeks mum’s birthday was there and that had totally escaped me. I have been trying to disassociate myself from her and at times it’s hard and then you don’t remember and somebody makes you remember. So, on one hand guilty, and the other do not know what to do. If she were alive I would have bought a piece of cake or something. Didn’t feel like it, hence donated some money to the local aged home. This way at least I hope they have some semblance of peace. All of them are of her similar age group.

The other thing that I began to observe in the earnest, fake identities have become the norm. Many people took elon musk’s potrait using their own names in the handles, but even then Elon “Free Speech” Musk banned them. So much for free speech. Then I saw quite a few handles that have cute women as their profile picture but they are good at social engineering. This has started only a couple of weeks back and have seen quite a few handles leaving Twitter and joining Mastodon. Also, have been hearing that many admins of Mastodon pods are unable to get on top of this. Also, lot of people complaining as it isn’t user-friendly UI as twitter is. Do they not realize that Twitter has its own IP and any competing network can’t copy or infringe on their product. Otherwise, they will be sued like how Ford was & potentially win. I am not really gonna talk much about it as the blog post has become quite long and that needs its own post to do any sort of justice to it. Till later people 🙂

10 November, 2022 01:45PM by shirishag75

Antoine Beaupré

Using the bell as modern notification

Computer terminals have traditionally had an actual bell that would ring when a certain control character (the bell character, typically control-g or \a in an C escape sequence) would come in the input stream.

That feature actually predates computers altogether, and was present in Baudot code, "an early character encoding for telegraphy invented by Émile Baudot in the 1870s", itself superseding Morse code.

Modern terminal emulators have, of course, kept that feature: if you run this command in a terminal right now:

printf '\a'

... you may hear some annoying beep. Or not. It actually depends on a lot of factors, including which terminal emulator you're using, how it's configured, whether you have headphones on, or speakers connected, or, if you're really old school, a PC speaker, even.

Typically, I have this theory that it does the exact opposite of what you want, regardless of whether you have configured it or not. That is, if you want it to make noises, it won't, and if you want it to stay silent, it will make brutal, annoying noises in moments you would the least expect. I suspect this is a law in computer science, but I'm too lazy to come up with a formal definition.

Yet something can be done with this.

Making the bell useful and silent

Many terminal emulators have this feature where they can silence the bell somehow. It can be turned into a "visual bell" which basically flashes the screen when a bell arrives. Or that can also be disabled and the bell is just completely ignored.

What I did instead is turn the bell into a "urgency hint" (part of the ICCCM standard. In xterm, this is done with this X resource entry (typically in ~/.Xresources):

XTerm*bellIsUrgent:  true
XTerm*visualBell: false

Interestingly, this doesn't clearly say "bell is muted", but it's effectively what it does. Or maybe it works because I have muted "System Sounds" in Pulseaudio. Who knows. I do have this in my startup scripts though:

xset b off

... which, according to the xset(1) manpage, means

If the dash or 'off' are given, the bell will be turned off.

Interestingly, you have the option of setting the bell "volume", "pitch, in hertz, and [...] duration in milliseconds. Note that not all hardware can vary the bell characteristics." In any case, I think that's the magic trick to turn the darn thing off.

Now this should send urgency hints to your window manager:

sleep 3 ; printf '\a'

Try it: run the command, switch to another desktop, then wait 3 seconds. You should see the previous desktop show up in red or something.

In the i3 window manager I am currently using, this is the default, although I did set the colors (client.urgent and urgent_workspace in bar.colors).

Other window managers or desktop environments may require different configurations.

Sending a bell...

Now that, on itself, will only be useful when something sets a bell. One place I had found a trick like this, long ago, is this post (dead link, archived) which has various instructions for different tools. I'll recopy some of them here since the original site is dead, but credit goes to the Netbuz blog.

(Note that the blog post also features an Awesome WM configuration for urgency hints.)


set beep=yes
set beep_new=yes


/set beep_when_window_active ON
/set beep_when_away ON
/set beep_msg_level MSGS DCC DCCMSGS HILIGHT

It was also recommending this setting, but it appears to be deprecated and gives a warning in modern irssi versions:

/set bell_beeps ON

GNU Screen

This is an important piece of the puzzle, because by default, terminal multiplexers have their own opinion of what to do with the bell as well:

# disabled: we want to propagate bell to clients, which should handle
# it in their own terminal settings. this `vbell off` is also the
# upstream and tmux's default
# see also: http://netbuz.org/blog/2011/11/x-bells-and-urgency-hints/
vbell off
# propagate bell from other windows up to the terminal emulator as well
bell_msg 'Bell in window %n^G'

The bell_msg bit is an extra from me: it uses the bell message that pops up when screen detects a bell in another window to resend the bell control character up to the running terminal.

This makes it so a bell in any multiplexed window will also propagate to the parent terminal, which is not the default.


Untested, but that is apparently how you do it:

# listen to alerts from all windows
set -g bell-action any
# notice bell in windows
set -g monitor-bell on
# only propagate bell, don't warn user, as it hangs tmux for a second
set -g visual-bell off
# send bell *and* notify when activity (if monitor-activity)
set -g visual-activity both

Note that this config goes beyond what we have in GNU screen in that inactivity or activity will trigger a bell as well. This might be useful for cases where you don't have the prompt hack (below) but it could also very well be very noisy. It will only generate noise when monitor-activity is enabled though.

bash and shell prompts

Now the icing on cake is to actually send the bell when a command completes. This is what I use this for the most, actually.

I was previously using undistract-me for this, actually. That was nice: it would send me a nice desktop notification when a command was running more than a certain amount of time, and if the window was unfocused. But configuring this was a nightmare: because it uses a complex PROMPT_COMMAND in bash, it would conflict with my (already existing and way too) complex bash prompt, and lead to odd behaviors. It would also not work for remote commands, of course, as it wouldn't have access to my local D-BUS to send notifications (thankfully!).

So instead, what I do now is systematically print a bell whenever a command terminates, in all my shells. I have this in my /root/.bashrc on all my servers, deployed in Puppet:

PROMPT_COMMAND='printf "\a"'

Or you can just put it directly in the shell prompt, with something like:


(I have the equivalent in my own .bashrc, although that thing is much more complex, featuring multi-command pipeline exit status, colors, terminal title setting, and more, which should probably warrant its own blog post.)

This sounds a little bonkers and really noisy, but remember that I turned off the audible bell. And urgency hints are going to show up only if the window is unfocused. So it's actually really nice and not really distracting.

Or, to reuse the undistract-me concept: it allows me to not lose focus too much when I'm waiting for a long process to complete.

That idea actually came from ahf, so kudos to him on that nice hack.


That is not setup on all the machines I administer for work, that said. I'm afraid that would be too disruptive for people who do not have that configuration. This implies that I don't get notifications for commands that run for a long time on remote servers, most of the time. That said, I could simply run a command with a trailing printf '\a' to get a notification.

This might not work in Wayland, your window manager, your desktop environment, your Linux console, or your telegraphy session.

10 November, 2022 01:44PM

Alastair McKinstry

Government approves T&Cs for first offshore wind auction under the ...

Government approves T&Cs for first offshore wind auction under the Renewable Electricity Support Scheme

ORESS 1 expected to secure 2.5GW of electricity generating capacity


The Government has today approved the Terms and Conditions of ORESS 1, the first auction for offshore wind under the Renewable Electricity Support Scheme.

This is a seminal moment in the delivery of offshore wind in Ireland. The offshore auction, the first in Ireland's history, is expected to provide a route to market for up to 2.5GW of offshore renewable energy to the Irish grid, enough to power 2.5 million Irish homes with clean electricity. Coming during the weeks of COP27, publishing details of the auction sends a strong international signal that Ireland is serious about offshore energy and our national climate targets and obligations.

Recognising the critical role of local hosting communities in the development of this critical infrastructure, all offshore wind projects developed via ORESS will be required to make Community Benefit Fund contributions, from construction phase and continuing for the duration of the support period, typically for a total period of 25 years. This will result in lasting, tangible benefits for these communities.

Speaking about this development, Minister Ryan said: “The publication of these ORESS 1 Terms and Conditions is another massive step forward – for offshore wind, for Irish climate leadership and towards Ireland’s future as an international green energy hub. The first stage of this transformative auction will start before Christmas and it sets us on a path to powering many more of our homes and businesses from our own green energy resources over the coming years. It follows the enactment of the Maritime Area Planning Act last year, and the announcement regarding the awarding of Maritime Area Consents to Phase One projects last month.”

A final ORESS 1 auction calendar will be published by EirGrid shortly. The pre-qualification stage will launch next month (December). The qualification stage and the auction process will take place in the first half of 2023. Final auction results will be published by June 2023.

Eligible projects

Any project that has been awarded a Maritime Area Consent is eligible to partake in the ORESS 1 auction. Seven projects – known as ‘Relevant Projects’ – were deemed ready to apply for Maritime Area Consents in Spring 2022. Once Maritime Area Consents are granted, these projects can not only compete for State support via the ORESS, but can also apply for planning permission from An Bord Pleanála.

Delivering on our broader targets

ORESS 1 is expected to procure approximately 2.5GW of electricity generating capacity.

Further auctions will be required to meet our renewable energy and climate ambitions. At least three offshore energy auctions are currently planned for this decade. The aim is to progress enough viable offshore projects through the consenting system to have competitive auctions. This will ultimately drive down cost for electricity consumers.

10 November, 2022 08:16AM by Alastair McKinstry

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo on CRAN: Updates

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1027 packages other packages on CRAN, downloaded 26.9 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 503 times according to Google Scholar.

This release reflect as new upstream bugfix release 11.4.2 made recently by Conrad. To accomodate CRAN and their preference for at most a release per month, we held it back since the 11.4.0 release early October. As we usually do, we generally update once upstream Armadillo releases are made. When we do not immediately release to CRAN (in order to lower the release cadence), we make those “interim” releases available via GitHub source and the Rcpp drat repo.

This release also brings a rearranged, and as we think, simplified layout of the header files. All existing locations are still supported but we will be starting a (very patient and slow) transition at some point.

The full set of changes (since the last CRAN release follows.

Changes in RcppArmadillo version (2022-11-08)

  • Upgraded to Armadillo release 11.4.2 (Ship of Theseus)

    • more robust handling of corner cases in multi-threaded contexts
  • Internal header organisation with new sub-directories while providing full compatibility via existing paths (#395 #396)

Changes in RcppArmadillo version (2022-10-10) (GitHub Only)

  • Upgraded to Armadillo release 11.4.1 (Ship of Theseus)

    • fix data race in Cube::slice()

    • workarounds for false warnings emitted by GCC 12 when compiling with FORTIFY_SOURCE enabled (already in RcppArmadillo too)

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 November, 2022 01:15AM

November 09, 2022

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Bidirectional Dijkstra

While I'm waiting for pull requests to be processed, enjoy some visualizations of bidirectional Dijkstra (1, 2).

09 November, 2022 11:00PM

Sven Hoexter

CentOS 9, stunnel, an openssl memory leak and a VirtualBox crash

tl;dr; OpenSSL 3.0.1 leaks memory in ssl3_setup_write_buffer(), seems to be fixed in 3.0.5 3.0.2. The issue manifests at least in stunnel and keepalived on CentOS 9. In addition I learned the hard way that running a not so recent VirtualBox version on Debian bullseye let to dh parameter generation crashing in libcrypto in bn_sqr8x_internal().

A recent rabbit hole I went down. The actual bug in openssl was nailed down and documented by Quentin Armitage on GitHub in keepalived My bugreport with all back and forth in the RedHat Bugzilla is #2128412.

Act I - Hello stunnel, this is the OOMkiller Calling

We started to use stunnel on Google Cloud compute engine instances running CentOS 9. The loadbalancer in front of those instances used a TCP health check to validate the backend availability. A day or so later the stunnel instances got killed by the OOMkiller. Restarting stunnel and looking into /proc/<pid>/smaps showed a heap segment growing quite quickly.

Act II - Reproducing the Issue

While I'm not the biggest fan of VirtualBox and Vagrant I've to admit it's quite nice to just fire up a VM image, and give other people a chance to recreate that setup as well. Since VirtualBox is no longer released with Debian/stable I just recompiled what was available in unstable at the time of the bullseye release, and used that. That enabled me now to just start a CentOS 9 VM, setup stunnel with a minimal config, grab netcat and a for loop and watch the memory grow. E.g. while true; do nc -z localhost 2600; sleep 1; done To my surprise, in addition to the memory leak, I also observed some crashes but did not yet care too much about those.

Act III - Wrong Suspect, a Workaround and Bugreporting

Of course the first idea was that something must be wrong in stunnel itself. But I could not find any recent bugreports. My assumption is that there are still a few people around using CentOS and stunnel, so someone else should probably have seen it before. Just to be sure I recompiled the latest stunnel package from Fedora. Didn't change anything. Next I recompiled it without almost all the patches Fedora/RedHat carries. Nope, no progress. Next idea: Maybe this is related to the fact that we do not initiate a TLS context after connecting? So we changed the test case from nc to openssl s_client, and the loadbalancer healthcheck from TCP to a TLS based one. Tada, a workaround, no more memory leaking. In addition I gave Fedora a try (they have Vagrant Virtualbox images in the "Cloud" Spin, e.g. here for Fedora 36) and my local Debian installation a try. No leaks experienced on both. Next I reported #2128412.

Act IV - Crash in libcrypto and a VirtualBox Bug

When I moved with the test case from the Google Cloud compute instance to my local VM I encountered some crashes. That morphed into a real problem when I started to run stunnel with gdb and valgrind. All crashes happened in libcrypto bn_sqr8x_internal() when generating new dh parameter (stunnel does that for you if you do not use static dh parameter). I quickly worked around that by generating static dh parameter for stunnel. After some back and forth I suspected VirtualBox as the culprit. Recompiling the current VirtualBox version (6.1.38-dfsg-3) from unstable on bullseye works without any changes. Upgrading actually fixed that issue.


I highly appreciate that RedHat, with all the bashing around the future of CentOS, still works on community contributed bugreports. My kudos go to Clemens Lang. :) Now that the root cause is clear, I guess RedHat will push out a fix for the openssl 3.0.1 based release they have in RHEL/CentOS 9. Until that is available at least stunnel and keepalived are known to be affected. If you run stunnel on something public it's not that pretty, because already a low rate of TCP connections will result in a DoS condition.

09 November, 2022 01:55PM

November 07, 2022

hackergotchi for Norbert Preining

Norbert Preining

KDE/Plasma for Debian – Update 2022/11

Monthly update on KDE/Plasma on Debian: Updates to Frameworks and KDE Gears

Short summary of recent changes and updates:

  • Frameworks updated to 5.99.0
  • Plasma 5.24 LTS (repo plasma524) has been updated to the latest patch level 5.24.7
  • Plasma 5.25 updated to the latest patch level 5.25.5
  • KDE Gears 22.08 updated to latest patch level 22.08.3
  • Krita updated to 5.1.3
  • (hopefully) everything recompiled against new Qt from Debian

If you see some strange behavior, please report.

Concerning Plasma 5.26

Debian unstable and testing already have (albeit outdated) packages for Plasma 5.26, and I have tried to package and build it for all the releases including Debian/stable. Unfortunately, Plasma 5.26 has a hard dependency onto a version of libDRM that is not available in Debian/stable, and thus compilation on Debian/stable does not succeed.

This makes my work regarding Plasma 5.26 far less useful, and thus I am currently not working on 5.26.

Usual reminder

I repeat (and update) instructions for all here, updated to use deb822 format (thanks to various comments on the blog here):

  • Get the key via
    curl -fsSL https://www.preining.info/obs-npreining.asc | sudo tee /usr/local/share/keyrings/obs-npreining.asc
  • Add the sources definition in /etc/apt/sources.lists.d/obs-npreining-kde.sources, replacing the DISTRIBUTION part with one of Debian_11 (for Bullseye), Debian_Testing, or Debian_Unstable:
    # deb822 source:
    # https://www.preining.info/blog/tag/kde/
    Types: deb
    Suites: /
    Signed-By: /usr/local/share/keyrings/obs-npreining.asc
    Enabled: yes

You can use also plasma524 instead of plasma525 if you prefer the LTS version of Plasma.


07 November, 2022 12:26AM by Norbert Preining

November 06, 2022

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppCCTZ 0.2.12 on CRAN: Maintenance

A new release 0.2.12 of RcppCCTZ is now on CRAN.

RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now several others packages (four the last time we counted) include its sources too. Not ideal, but beyond our control.

This version adds support for NA values when parsing, and updates GitHub Action.

Changes in version 0.2.12 (2022-11-06)

  • Support NA values in numerical or character input

  • GitHub Actions were updated to checkout version 3.

Courtesy of my CRANberries, there is also a diffstat to the previous version. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

06 November, 2022 10:22PM

hackergotchi for Marco d'Itri

Marco d'Itri

Using a custom domain as the Mastodon identity

I just did again the usual web search, and I have verified that Mastodon still does not support managing multiple domains on the same instance, and that there is still no way to migrate an account to a different instance without losing all posts (and more?).

As much as I like the idea of a federated social network, open standards and so on, I do not think that it would be wise for me to spend time developing a social network identity on somebody else's instance which could disappear at any t ime.

I have managed my own email server since the '90s, but I do not feel that the system administration effort required to maintain a private Mastodon instance would be justified at this point: there is not even a Debian package! Mastodon either needs to become much simpler to maintain or become much more socially important, and so far it is neither. Also, it would be wasteful to use so many computing resources for a single-user instance.

While it is not ideal, for the time being I compromised by redirecting WebFinger requests for md@linux.it using this Apache configuration:

<Location /.well-known/host-meta>
  Header set Access-Control-Allow-Origin: "*"
  Header set Content-Type: "application/xrd+json; charset=utf-8"
  Header set Cache-Control: "max-age=86400"

<Location /.well-known/webfinger>
  Header set Access-Control-Allow-Origin: "*"
  Header set Content-Type: "application/jrd+json; charset=utf-8"
  Header set Cache-Control: "max-age=86400"

# WebFinger (https://www.rfc-editor.org/rfc/rfc7033)
RewriteMap lc int:tolower
RewriteMap unescape int:unescape
RewriteCond %{REQUEST_URI} ^/\.well-known/webfinger$
RewriteCond ${lc:${unescape:%{QUERY_STRING}}} (?:^|&)resource=acct:([^&]+)@linux\.it(?:$|&)
RewriteRule .+ /home/soci/%1/public_html/webfinger.json [L,QSD]
# answer 404 to requests for domains != linux.it
RewriteCond ${unescape:%{QUERY_STRING}} (?:^|&)resource=
RewriteRule .+ - [L,R=404]
# answer 400 to requests without the resource parameter
RewriteCond %{REQUEST_URI} ^/\.well-known/webfinger$
RewriteRule .+ - [L,R=400]

06 November, 2022 05:15PM

Arturo Borrero González

Home network refresh: 10G and IPv6

Post header

A few days ago, my home network got a refresh that resulted in the enablement of some next-generation technologies for me and my family. Well, next-generation or current-generation, depending on your point of view. Per the ISP standards in Spain (my country), what I’ll describe next is literally the most and latest you can get.

The post title spoiled it already. I have now 10G internet uplink and native IPv6 since I changed my ISP to https://digimobil.es.

My story began a few months ago when a series of fiber deployments started in my neighborhood by a back-then mostly unknown ISP (digimobil). The workers were deploying the fiber inside the street sewers, and I noticed that they were surrounded by advertisements promoting the fastest FTTH deployment in Spain. Indeed, their website was promoting 1G and 10G fiber, so a few days later I asked the workers when would that be available for subscription. They told me to wait just a couple of months, and the wait ended this week.

I called the ISP, and a marketing person told me a lot of unnecessary information about how good service I was purchasing. I asked about IPv6 availability, but that person had no idea. They called me the next day to confirm that the home router they were installing would support both IPv6 and Wi-Fi 6. I was suspicious about nobody in the marketing department knowing anything about any of the two technologies, but I decided to proceed anyway. Just 24 hours after calling them, a technician came to my house and 45 minutes later the setup was ready.

The new home router was a ZTE ZXHN F8648P unit. By the way, it had Linux inside, but I got no GPL copyright notice or anything. It had 1x10G and 4x1G ethernet LAN ports. The optical speed tests that the technician did were giving between 8 Gbps to 9 Gbps in uplink speed, which seemed fair enough. Upon quick search, there is apparently a community of folks online which already know how to get the most out of this router by unbloking the root account (sorry, in spanish only) and using other tools.

When I plugged the RJ45 in my laptop, the magic happened: the interface got a native, public IPv6 from the router. I ran to run the now-classic IPv6 browser test at https://test-ipv6.com/. And here is the result:

IPv6 test

If you are curious, this was the IPv6 prefix whois information:

route6: 2a0c:5a80::/29
descr: Digi Spain Telecom S.L.U.
origin: AS57269

They were handing my router a prefix like 2a0c:5a80:2218:4a00::/56. I ignored if the prefix was somehow static, dynamic, just for me, or anything else. I’ve been waiting for native IPv6 at home for years. In the past, I’ve had many ideas and projects to host network services at home leveraging IPv6. But when I finally got it, I didn’t know what to do next. I had a 7 months old baby, and honestly I didn’t have the spare time to play a lot with the setup.

Actually, I had no need or use for such fast network either. But my coworker Andrew convinced me: given the price € 30 EUR / month, I didn’t have any reason not to buy it.

In fact, I didn’t have any 10G-enabled NIC at home. I had a few laptops with 2.5G ports, though, and that was enough to experience the new network speeds. Since this write-up was inspired by the now almost-legenday post by Michael Stapelberg My upgrade to 25 Gbit/s Fiber To The Home, I contacted him, and he suggested running a few speed tests using the Ookla suite against his own server. Here are the results:

$ docker run --net host --rm -it docker.io/stapelberg/speedtest:latest -s 50092
     Server: Michael Stapelberg - Zurich (id = 50092)
        ISP: Digi Spain
    Latency:    34.29 ms   (0.20 ms jitter)
   Download:  2252.42 Mbps (data used: 3.4 GB )
     Upload:  2239.27 Mbps (data used: 2.8 GB )
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/cc8d6a78-c6f8-4f71-b554-a79812e10106
$ docker run --net host --rm -it docker.io/stapelberg/speedtest:latest -s 50092
     Server: Michael Stapelberg - Zurich (id = 50092)
        ISP: Digi Spain
    Latency:    34.05 ms   (0.21 ms jitter)
   Download:  2209.85 Mbps (data used: 3.2 GB )
     Upload:  2223.45 Mbps (data used: 2.9 GB )
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/32f9158e-fc1a-47e9-bd33-130e66c25417

This is over IPv6. Very satisfying.

Bonus point: when I called my former ISP to cancel the old subscription the conversation was like:

  • I want to cancel the service.
  • What’s the reason?
  • I got upgraded to 10G by another ISP
  • The speed is measured in MB, not G.
  • Ok, I got upgraded to 10.000 MB
  • That’s not possible.
  • Well …

I didn’t even bother mentioning IPv6.


06 November, 2022 09:40AM

Russ Allbery

Review: Matrix

Review: Matrix, by Lauren Groff

Publisher: Riverhead Books
Copyright: 2021
ISBN: 0-698-40513-7
Format: Kindle
Pages: 260

Marie is a royal bastardess, a product of rape no less, and entirely out of place in the court in Westminster, where she landed after being kicked off her mother's farm. She had run the farm since her mother's untimely death, but there was no way that her relatives would let her inherit. In court, Marie is too tall, too ugly, and too strange, raised by women who were said to carry the blood of the fairy Mélusine. Eleanor of Aquitaine's solution to her unwanted house guest is a Papal commission. Marie is to become the prioress of an abbey.

I am occasionally unpleasantly reminded of why I don't read very much literary fiction. It's immensely frustrating to read a book in which the author cares about entirely different things than the reader, and where the story beats all land wrong.

This is literary historical fiction set in the 12th century. Marie is Marie de France, author of the lais about courtly love that are famous primarily due to their position as early sources for the legends of King Arthur. The lais are written on-screen very early in this book, but they disappear without any meaningful impact on the story. Matrix is, instead, about Shaftesbury Abbey and what it becomes during Marie's time as prioress and then abbess, following the theory that Marie de France was Mary of Shaftesbury.

What I thought I was getting in this book, from numerous reviews and recommendations, was a story of unexpected competence: how a wild, unwanted child of seventeen lands at a dilapidated and starving abbey, entirely against her will, and then over the next sixty years transforms it into one of the richest abbeys in England. This does happen in this book, but Groff doesn't seem to care about the details of that transformation at all.

Instead, Matrix takes the mimetic fiction approach of detailed and precise description of a few characters, with all of their flaws and complexities, and with all of the narrative's attention turned to how they are feeling and what they are seeing. It is also deeply, fully committed to a Great Man (or in this case a Great Woman) view of history.

Marie is singular. The narrative follows her alone, she makes all the significant decisions, and the development of the abbey is determined by her apparently mystical visions. (In typical mimetic fashion, these are presented as real to her, and the novel takes no position on whether that reality is objective.) She builds spy networks, maneuvers through local and church politics, and runs the abbey like her personal kingdom. The tiny amount of this that is necessarily done by other people is attributed to Marie's ability to judge character. Other people's motives are simply steamrolled over and have no effect.

Maddeningly, essentially all of this happens off-screen, and Groff is completely uninterested in the details of how any of it is accomplished. Marie decides to do something, the narrative skips forward a year, and it has happened. She decides to build something, and then it's built. She decides to collect the rents she's due, the novel gestures vaguely at how she's intimidating, and then everyone is happily paying up. She builds spy networks; who cares how? She maneuvers through crises of local and church politics that are vaguely alluded to, through techniques that are apparently too uninteresting to bother the reader with.

Instead, the narrative focuses on two things: her deeply dysfunctional, parasocial relationship with Eleanor, and her tyrannical relationship with the other nuns. I suspect that Groff would strongly disagree with my characterization of both of those narratives, and that's the other half of my problem with this book.

Marie is obsessed with and in love with Eleanor, a completely impossible love to even talk about, and therefore turns to courtly love from afar as a model into which she can fit her feelings. While this is the setup for a tragedy, it's a good idea for a story. But what undermined it for me is that Marie's obsession seems to be largely physical (she constantly dwells on Eleanor's beauty), and Eleanor is absolutely horrible to her in every way: condescending, contemptuous, dismissive, and completely uninterested. This does change a bit over the course of the book, but not enough to justify the crush that Marie maintains for this awful person through her entire life.

And Eleanor is the only person in the book who Marie treats like an equal. Everyone else is a subordinate, a daughter, a charge, a servant, or a worker. The nuns of the abbey prosper under her rule, so Marie has ample reason to justify this to herself, but no one else's opinions or beliefs matter to her in any significant way. The closest anyone can come to friendship is to be reliably obedient, perhaps after some initial objections that Marie overrules. Despite some quite good characterization of the other nuns, none of the other characters get to do anything. There is no delight in teamwork, sense of healthy community, or collaborative problem-solving. It's just all Marie, all the time, imposing her vision on the world both living and non-living through sheer force of will.

This just isn't entertaining, at least for me. The writing might be beautiful, the descriptions detailed and effective, and the research clearly extensive, but I read books primarily for characters, I read characters primarily for their relationships, and these relationships are deeply, horribly unhealthy. They are not, to be clear, unrealistic (although I do think there's way too much chosen one in Marie and way too many problems that disappear off-camera); there are certainly people in the world with dysfunctional obsessive relationships, and there are charismatic people who overwhelm everyone around them. This is just not what I want to read about.

You might think, with all I've said above, that I'm spoiling a lot of the book, but weirdly I don't think I am. Every pattern I mention above is well-established early in the novel. About the only thing that I'm spoiling is the hope that any of it is somehow going to change, a hope that I clung to for rather too long.

This is a great setup for a book, and I wish it were written by a fantasy author instead of a literary author. Perhaps I'm being too harsh on literary fiction here, but I feel like fantasy authors are more likely to write for readers who want to see the growth sequence. If someone is going to change the world, I want to see how they changed the world. The mode of fantasy writing tends to think that what people do (and how they do it) is as interesting or more interesting than what they feel or what they perceive.

If this idea, with the exact same level of (minor) mysticism and historic realism, without added magic, were written by, say, Guy Gavriel Kay or Nicola Griffith, it would be a far different and, in my opinion, a much better book. In fact, Hild is part of this book written by Nicola Griffith, and it is a much better book.

I have seen enough people rave about this book to know that this is a personal reaction that is not going to be shared by everyone, or possibly even most people. My theory is that this is due to the different reading protocols between literary fiction readers and fantasy readers. I put myself in the latter camp; if you prefer literary fiction, you may like this much better (and also I'm not sure you'll find my book reviews useful). I may be wrong, though; maybe there are fantasy readers who would like this. I will say that the sense of place is very strong and the writing has all the expected literary strengths of descriptiveness and rhythm.

But, sadly, this was not at all my thing, and I'm irritated that I wasted time on it.

Rating: 4 out of 10

06 November, 2022 01:51AM

November 04, 2022

Thorsten Alteholz

My Debian Activities in October 2022

FTP master

This month I accepted 484 and rejected 55 packages. The overall number of packages that got accepted was 492.

Debian LTS

This was my hundredth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.  Woohoo, There is a party. (yes I am old). Unfortunately there are already 101 completed month listed in the debian-lts-announce archive, so I seem to have counted wrong once. *sigh*, yes I am old.

This month my all in all workload has been 14h.

During that time I uploaded:

  • [DLA 3168-1] openvswitch security update for one CVE
  • [DLA 3167-1] ncurses security update for one CVE
  • [#1020596] bullseye-pu: mod-wsgi/4.7.1-3+deb11u1 upload
  • [graphicsmagick] debdiff for Bullseye sent to security team (update as DLA or via PU?)
  • [graphicsmagick] prepared upload for Buster
  • [libvncserver] debdiff for Buster and Bullseye sent to maintainer (no upload yet :-()

I also started to work on virglrenderer.

Last but not least I took care of NEW packages on security-master.

Debian ELTS

This month was the fifty first ELTS month.

During my allocated time I uploaded:

  • [ELA-719-1] graphicsmagick security update of Jessie and Stretch for one CVE
  • [ELA-720-1] bluez security update of Jessie and Stretch for three CVEs
  • marked two CVEs of curl as not-affected for Jessie and Stretch
  • checked that all patches for dpdk need to be backported, unfortunately that was beyond my capabilities

I also started to work on snapd.

Last but not least I finally managed to become familiar with the git workflow and imported several packages to the salsa repository.

Debian Astro

This month I uploaded new upstream versions or improved packaging of:

I also uploaded a new package pysqm. This software supports the Sky Quality Meters made by Unihedron. I was kindly given an SQM-LU for USB and SQM-LE with network adapter. I plan to put a working Python3 version of the old PySQM software into Debian, package the UDM (Unihedron Device Manager) and finally check the support within Indi.

Debian IoT

This month I uploaded new upstream versions or improved packaging of:

Debian Mobcom

This month I finished the transition of the Osmocom packages, except
osmo-mgw and osmo-msc seem to have problems. I have no idea how I can solve this, so help is appreciated.

Other stuff

This month I uploaded new packages:

04 November, 2022 03:24PM by alteholz

Alastair McKinstry

€1.3 billion announced for new Forestry Support

€1.3 billion announced for new Forestry Support

Funds to be delivered through new Forestry Programme
Premiums for planting trees to be increased by between 46% and 66% and extended to 20 years for farmers


The Taoiseach, Micheál Martin TD, Minister of State with responsibility for Forestry, Senator Pippa Hackett, and Minister for Agriculture, Food and the Marine, Charlie McConalogue T.D today announced a proposed investment by the Government of €1.3 billion in Irish forestry. The funding will be for the next national Forestry Programme and represents the largest ever investment by an Irish Government in tree-planting. The programme will now be the subject of state-aid approval by the European Commission.

The Taoiseach said:

“This commitment by the Government to such a substantial financial package reflects the seriousness with which we view the climate change and biodiversity challenges, which affect all of society. Forestry is at the heart of delivering on our sustainability goals and strong support is needed to encourage engagement from all our stakeholders in reaching our objectives. “


Minister Hackett said:

‘I’m delighted to have secured a package of €1.318 billion for forestry. This will support the biggest and best-funded Forestry Programme ever in Ireland. It comes at an appropriate time, given the urgency of taking climate mitigation measures. Planting trees is one of the most effective methods of tackling climate change as well as contributing to improved biodiversity and water quality. One of my main aims is to re-engage farmers in afforestation. I’m delighted therefore to be proposing a new 20-year premium term exclusively for farmers, as well as introducing a new small-scale native woodland scheme which will allow farmers to plant up to 1 hectare of native woodland on farmland and along watercourses outside of the forestry licensing process.”


“Minister McConalogue said:

“Today we commit to providing unprecedented incentives to encourage the planting of trees that can provide a valuable addition to farm income and help to meet national climate and biodiversity objectives. This funding guarantees continued payments to those forest owners who planted under the current scheme and who are still in receipt of premiums. It also offers new and improved financial supports to those who undertake planting and sustainable forest management under the new Programme. We intend to increase premiums for planting trees by between 46% and 66% and to extend the premium period from 15 to 20 years for farmers.


"We are approaching a new and exciting period for forestry in Ireland. The new Forestry Programme will drive a new and brighter future for forestry, for farmers and for our climate.”


The proposed new Forestry Programme is currently out to public consultation as part of the Strategic Environmental Assessment and Appropriate Assessment process. The Programme is the main implementation mechanism for the new Forest Strategy (2023 -2030) which reflects the ambitions contained in the recently published Shared National Vision for Trees, Woods and Forests in Ireland until 2050. The public consultation closes on 29th November, 2022 and any changes which result from this process will be incorporated into the Programme and the Forest Strategy.

Minister Hackett commented: “The draft Forestry Programme and Forest Strategy are the product of extensive stakeholder consultation and feedback, and both documents are open to public consultation for the next number of weeks. I would strongly encourage all interested parties to engage with the consultation in advance of the Strategy and Programme being finalised.”

The new Programme is built around the principle of right trees in the right places for the right reasons with the right management. It aims to deliver more diverse forest which will meet multiple societal objectives, economic, social and environmental. Higher grant rates for forest establishment are also proposed with increases of approximately 20% to reflect rising living costs. The new one hectare native tree area scheme will also make it easier for landowners who wish to plant small areas of trees on their farm.


The Taoiseach concluded “I welcome this milestone and I believe that this funding injection will be an important catalyst in delivering on the ambition outlined in the new Forest Strategy. Our environmental challenges are huge but so is our commitment to overcoming them and this Forestry Programme is key to delivering so many of our priorities.


The new Programme will be 100% Exchequer funded and is subject to State Aid approval from the EU Commission. The Department is in contact with the Commission in relation to this approval which is a rigorous process.

04 November, 2022 08:45AM by Alastair McKinstry

hackergotchi for Louis-Philippe Véronneau

Louis-Philippe Véronneau

Book Review: Chokepoint Capitalism, by Rebecca Giblin and Cory Doctorow

Two weeks ago, I had the chance to go see Cory Doctorow at my local independent bookstore, in Montréal. He was there to present his latest essay, co-written with Rebecca Giblin1. Titled Chokepoint Capitalism: How Big Tech and Big Content Captured Creative Labor Markets and How We'll Win Them Back, it focuses on the impact of monopolies and monopsonies (more on this later) on creative workers.

The book is divided in two main parts:

  • Part one, Culture has been captured (chapters 1 to 11), is a series of case studies that focus on different examples of market failure. The specific sectors analysed are the book market, the news media, the music industry, Hollywood, the mobile apps industry and the online video platforms.
  • Part two, Braking anticompetitive flywheels (chapters 12 to 19), looks at different solutions to try to fix these failures.

A picture of the book cover

Although Doctorow is known for his strong political stances, I have to say I'm quite surprised by the quality of the research Giblin and he did for this book. They both show a pretty advanced understanding of the market dynamics they look at, and even though most of the solutions they propose aren't new or groundbreaking, they manage to be convincing and clear.

That is to say, you certainly don't need to be an economist to understand or enjoy this book :)

As I have mentioned before, the book heavily criticises monopolies, but also monopsonies — a market structure that has only one buyer (instead of one seller). I find this quite interesting, as whereas people are often familiar with the concept of monopolies, monopsonies are frequently overlooked.

The classic example of a monopsony is a labor market with a single employer: there is a multitude of workers trying to sell their labor power, but in the end, working conditions are dictated by the sole employer, who gets to decide who has a job and who hasn't. Mining towns are good real-world examples of monopsonies.

In the book, the authors argue most of the contemporary work produced by creative workers (especially musicians and writers) is sold to monopsonies and oligopsonies, like Amazon2 or major music labels. This creates a situation where the consumers are less directly affected by the lack of competition in the market (they often get better prices), but where creators have an increasingly hard time making ends meet. Not only this, but natural monopsonies3 are relatively rare, making the case for breaking the existing ones even stronger.

Apart from the evident need to actually start applying (the quite good) antitrust laws in the USA, some of the other solutions put forward are:

  • Transparency Rights — giving creative workers a way to audit the companies that sell their work and make sure they are paid what they are due.
  • Collective Action
  • Time Limits on Copyright Contracts — making sure creators that sell their copyrights to publishers or labels can get them back after a reasonable period of time.
  • Radical Interoperability — forcing tech giants to make their walled-gardens interoperable.
  • Minimum Wages for Creative Work — enforcing minimum legal rates for workers in certain domains, like what is already done for screenplays in the US by members of the Writers Guild of America.
  • Collective Ownership

Overall, I found this book quite enjoying and well written. Since I am not a creative worker myself and don't experience first-hand the hardships presented in the book, it was the occasion for me to delve more deeply in this topic. Chances are I'll reuse some of the exposés in my classes too.

  1. Professor at the Melbourne Law School and Director of the Intellectual Property Research Institute of Australia, amongst other things. More on her here

  2. Amazon owns more than 50% of the US physical book retail market and has an even higher market share for ebooks and audiobooks (via Audible). Not only this, but with the decline of the physical book market, audiobooks are an increasingly important source of revenue for authors. 

  3. Natural monopolies happen when it does not make economic sense for multiple enterprises to compete in a market. Critical infrastructures, like water supply or electricity, make for good examples of natural monopolies. It simply wouldn't be efficient to have 10 separate electrical cables connecting your house to 10 separate electric grids. In my opinion, such monopolies are acceptable (and even desirable), as long as they are collectively owned, either by the State or by local entities (municipalities, non-profits, etc.). 

04 November, 2022 04:00AM by Louis-Philippe Véronneau

November 03, 2022

hackergotchi for Steve Kemp

Steve Kemp

Alphabetical linting ..

So this week I recycled a talk I'd given in the past, about how even using extremely simple parsers allows a lot of useful static-analysis to be done, for specific niche use-cases.

This included examples of scanning comments above classes to ensure they referred to the appropriate object, ensuring that specific function calls always included a specific (optional) parameter, etc.

Nothing too complex, but I figured I'd give a new example this time, and I remembered I'd recently written a bunch of functions for an interpreter which I'd ordered quite deliberately.

Assume you're writing a BASIC interpreter, you need to implement a bunch of built-in maths functions such as SIN, COS, TAN, then some string-related functions LEFT$, RIGHT$, MID$, etc.

When it comes to ordering there are a couple of approaches:

  • Stick them all in one package:
    • builtins/builtins.go
  • Create a package and group them:
    • builtins/maths.go
    • builtins/string.go
    • .. etc

Personal preference probably dictates the choice you make, but either way I think it would be rational and obvious that you'd put the functions in alphabetical order:

func ABS( args []primitive.Object) (primitive.Object, error) {

func COS( args []primitive.Object) (primitive.Object, error) {

func SIN( args []primitive.Object) (primitive.Object, error) {

func TAN( args []primitive.Object) (primitive.Object, error) {

I did that myself, and I wrote a perl-script to just parse the file using a simple regexp "^func\s+([^(]+)\(" but then I figured this was a good time to write a real static-analysis tool.

The golang environment is full of trivial little linters for various purposes, and the standard "go vet .." driver makes it easy to invoke them. Realizing that I was going to get driven in the same way it was obvious I'd write something called "alphaVet".

So anyway, half written for a talk, half-written because of the name:

03 November, 2022 10:00PM

hackergotchi for Sean Whitton

Sean Whitton


I’ve recently enhanced my emacsclient(1) wrapper script to make it possible to have my primary Emacs daemon always running under gdb. That way, if there’s a seemingly-random crash, I might be able to learn something about what happened. The tricky thing is that I want gdb running inside an instance of Emacs too, because Emacs has a nice interface to gdb, and gdb’s Emacs daemon – hereafter “gdbmacs” – needs to be the installed, optimised build of Emacs, such that it’s not likely to suffer the same crash. And the whole thing should be transparent: I shouldn’t have to do anything special to launch the primary session under gdb. That is, if right after booting up my machine I execute

% emacsclient foo.txt

then gdbmacs should start, it should then start the primary sesion under gdb, and finally the real emacsclient(1) should connect to the primary session and request editing foo.txt. I’ve got that all working now, and there are some nice additional features. If the primary session hits a breakpoint, for example, then emacsclient requests will be redirected to gdbmacs, so that I can still edit files etc. without losing the information in the gdb session. I’ve given gdbmacs a different background colour, so that if I request a new graphical frame and it pops up with that colour, I know that the main session is wedged and I might like to investigate.

First attempt: remote attaching

My first attempt, which was running for several weeks, had a different architecture. Instead of having gdbmacs start up the primary session, the primary session would start up gdbmacs, if necessary, and then send over its own PID, and ask gdbmacs to use gdb’s functionality for attaching to existing processes:

(defun spw/remote-gdbmacs-attach ()
  (call-process "emacsclient" nil "*gdbmacs-emacsclient*" nil
                "--socket-name=gdbmacs" "--spw/installed" "--eval"
                (prin1-to-string `(spw/gdbmacs-attach ,(emacs-pid)))))

(defun spw/maybe-remote-gdbmacs-attach ()
  (when (and (eq (daemonp) t) ; `daemonp' can return a string
             (file-in-directory-p invocation-directory "~/src/emacs/"))
(add-hook 'after-init-hook #'spw/maybe-remote-gdbmacs-attach)

So, if we are an Emacs that just started up out of my clone of Emacs’s upstream git repository in ~/src/emacs, invoke the emacsclient(1) wrapper script to have gdbmacs attach to us. The --spw/installed option asks the wrapper script to start up gdbmacs using the Emacs binary on PATH, not the one in ~/src/emacs. (We can’t use the server-eval-at function because we need the wrapper script to start up gdbmacs if it’s not already running.)

Over in gdbmacs, the spw/gdbmacs-attach function then did something like this:

(let ((default-directory (expand-file-name "~/src/emacs/")))
  (gdb (format "gdb -i=mi --pid=%d src/emacs" pid))
  (gdb-wait-for-pending (lambda () (gud-basic-call "continue"))))

Having gdbmacs attach to the existing process is more robust than having gdbmacs start up Emacs under gdb. If anything goes wrong with attaching, or with gdbmacs more generally, you’ve still got the primary session running normally – it just won’t be under a debugger. More significantly, the wrapper script doesn’t need to know anything about the relationship between the two daemons. It just needs to be able to start up both in-tree and installed daemons, via its --spw/installed option. The complexity is all in Lisp, not shell script (the wrapper is a shell script because it needs to be fast).

The disadvantage of this scheme is that the primary session’s stdout and stderr are not directly accessible to gdbmacs. There is a function redirect-debugging-output to deal with this situation, and I experimented with having spw/remote-gdbmacs-attach call this and send the new output filename to gdbmacs, but it’s much less smooth than having gdbmacs start up the primary session itself.

I think most people would probably prefer this scheme. It’s definitely cleaner to have the two daemons start up independently, and then have one attach to the other. But I decided that I was willing to complexify my wrapper script in order to have the primary session’s stdout and stderr attached to gdbmacs in the normal way.

Second attempt: daemons starting daemons

In this version, the relevant logic is shifted out of Lisp into the wrapper script. When we execute emacsclient foo.txt, the script first determines whether the primary session is already running, using something like this:

[ -e /run/user/1000/emac/server \
    -a -n "$(ss -Hplx src /run/user/1000/emacs/server)" ]

The ss(8) tool is used to determine if anything is listening on the socket. The script also uses flock(1) to have other instances of the wrapper script wait, in case they are going to cause the daemon to exit, or something. If the daemon is running, then we can just exec ~/src/emacs/lib-src/emacsclient to handle the request. If not, we first have to start up gdbmacs:

installed_emacsclient=$(PATH=$(echo "$PATH" \
                   | sed -e "s#/directory/containing/wrapper/script##") \
                command -v emacsclient)
"$installed_emacsclient" -a '' -sgdbmacs --eval '(spw/gdbmacs-attach)'

spw/gdbmacs-attach now does something like this:

(let ((default-directory (expand-file-name "~/src/emacs/")))
  (gdb "gdb -i=mi --args src/emacs --fg-daemon")
   (lambda ()
     (gud-basic-call "set cwd ~")
      (lambda ()
        (gud-basic-call "run"))))))

"$installed_emacsclient" exits as soon as spw/gdbmacs-attach returns, which is before the primary session has started listening on the socket, so the wrapper script uses inotifywait(1) to wait until /run/user/1000/server appears. Then it is finally able to exec ~/src/emacs/lib-src/emacsclient to handle the request.

A particular kind of complexity

The wrapper script must be highly reliable. I use my primary Emacs session for everything, on the same laptop that I do my academic work. The main way I get at it is via a window manager shortcut that executes emacsclient -nc to request a new frame, such that if there is a problem, I won’t see any error output until I open an xterm and tail ~/.swayerr/~/.xsession-errors. And as starting gdbmacs and only then starting up less optimised, debug in-tree builds of Emacs is not fast, I would have to wait at least ten seconds without any Emacs frame popping up before I could suppose that something was wrong.

This is where the first scheme, where the complexity is all in Lisp, really seems attractive. My emacsclient(1) wrapper script has several other facilities and convenience features, some of which are general and some of which are only for my personal usage patterns, and the code for all those is now interleaved with the special cases for gdbmacs and the primary session that I’ve described in this post. There’s a lot that could go wrong, and it’s all in shell, and its output isn’t readily visible to the user. I’ve done a lot of testing, and I’m pretty confident in the script in its current form, but if I need to change or add features, I’ll have to do a lot of testing again before I can deploy to my usual laptop.

Single-threaded, readily interactively-debuggable Emacs Lisp really shines for this sort of “do exactly what I mean, as often as possible” code, and you find a lot of it in Emacs itself, third party packages, and peoples’ init.el files. You can add all sorts of special cases to your interactive commands to make Emacs do just what is most useful, and have confidence that you can manage the resulting complexity. In this case, though, I’ve got piles of just this sort of complexity out in an opaque shell script. The ultimate goal, though, is debugging Emacs, such that one can run yet more DJWIM Emacs Lisp, which perhaps justifies it.

03 November, 2022 06:52PM

Arturo Borrero González

New OpenPGP key and new email

Post logo

I’m trying to replace my old OpenPGP key with a new one. The old key wasn’t compromised or lost or anything bad. Is still valid, but I plan to get rid of it soon. It was created in 2013.

The new key id fingerprint is: AA66280D4EF0BFCC6BFC2104DA5ECB231C8F04C4

I plan to use the new key for things like encrypted emails, uploads to the Debian archive, and more. Also, the new key includes an identity with a newer personal email address I plan to use soon: arturo.bg@arturo.bg

The new key has been uploaded to some public keyservers.

If you would like to sign the new key, please follow the steps in the Debian wiki.



If you are curious about what that long code block contains, check this https://cirw.in/gpg-decoder/

For the record, the old key fingerprint is: DD9861AB23DC3333892E07A968E713981D1515F8


03 November, 2022 04:00PM

Antoine Beaupré

Debating VPN options

In my home lab(s), I have a handful of machines spread around a few points of presence, with mostly residential/commercial cable/DSL uplinks, which means, generally, NAT. This makes monitoring those devices kind of impossible. While I do punch holes for SSH, using jump hosts gets old quick, so I'm considering adding a virtual private network (a "VPN", not a VPN service) so that all machines can be reachable from everywhere.

I see three ways this can work:

  1. a home-made Wireguard VPN, deployed with Puppet
  2. a Wireguard VPN overlay, with Tailscale or equivalent
  3. IPv6, native or with tunnels

So which one will it be?

Wireguard Puppet modules

As is (unfortunately) typical with Puppet, I found multiple different modules to talk with Wireguard.

module score downloads release stars watch forks license docs contrib issue PR notes
halyard 3.1 1,807 2022-10-14 0 0 0 MIT no requires firewall and Configvault_Write modules?
voxpupuli 5.0 4,201 2022-10-01 2 23 7 AGPLv3 good 1/9 1/4 1/61 optionnally configures ferm, uses systemd-networkd, recommends systemd module with manage_systemd to true, purges unknown keys
abaranov 4.7 17,017 2021-08-20 9 3 38 MIT okay 1/17 4/7 4/28 requires pre-generated private keys
arrnorets 3.1 16,646 2020-12-28 1 2 1 Apache-2 okay 1 0 0 requires pre-generated private keys?

The voxpupuli module seems to be the most promising. The abaranov module is more popular and has more contributors, but it has more open issues and PRs.

More critically, the voxpupuli module was written after the abaranov author didn't respond to a PR from the voxpupuli author trying to add more automation (namely private key management).

It looks like setting up a wireguard network would be as simple as this on node A:

wireguard::interface { 'wg0':
  source_addresses => ['2003:4f8:c17:4cf::1', ''],
  public_key       => $facts['wireguard_pubkeys']['nodeB'],
  endpoint         => 'nodeB.example.com:53668',
  addresses        => [{'Address' => '',},{'Address' => 'fe80::beef:1/64'},],

This configuration come from this pull request I sent to the module to document how to use that fact.

Note that the addresses used here are examples that shouldn't be reused and do not confirm to RFC5737 ("IPv4 Address Blocks Reserved for Documentation", (TEST-NET-1), (TEST-NET-2), and (TEST-NET-3)) or RFC3849 ("IPv6 Address Prefix Reserved for Documentation", 2001:DB8::/32), but that's another story.

(To avoid boostrapping problems, the resubmit-facts configuration could be used so that other nodes facts are more immediately available.)

One problem with the above approach is that you explicitly need to take care of routing, network topology, and addressing. This can get complicated quickly, especially if you have lots of devices, behind NAT, in multiple locations (which is basically my life at home, unfortunately).

Concretely, basic Wireguard only support one peer behind NAT. There are some workarounds for this, but they generally imply a relay server of some sort, or some custom registry, it's kind of a mess. And this is where overlay networks like Tailscale come in.


Tailscale is basically designed to deal with this problem. It's not fully opensource, but pretty close, and they have an interesting philosophy behind that. The client is opensource, and there is an opensource version of the server side, called headscale. They have recently (late 2022) hired the main headscale developer while promising to keep supporting it, which is pretty amazing.

Tailscale provides an overlay network based on Wireguard, where each peer basically has a peer-to-peer encrypted connexion, with automatic key rotation. They also ship a multitude of applications and features on top of that like file sharing, keyless SSH access, and so on. The authentication layer is based on an existing SSO provider, you don't just register with Tailscale with new account, you login with Google, Microsoft, or GitHub (which, really, is still Microsoft).

The Headscale server ships with many features out of that:

  • Full "base" support of Tailscale's features
  • Configurable DNS
    • Split DNS
    • MagicDNS (each user gets a name)
  • Node registration
    • Single-Sign-On (via Open ID Connect)
    • Pre authenticated key
  • Taildrop (File Sharing)
  • Access control lists
  • Support for multiple IP ranges in the tailnet
  • Dual stack (IPv4 and IPv6)
  • Routing advertising (including exit nodes)
  • Ephemeral nodes
  • Embedded DERP server (AKA NAT-to-NAT traversal)

Neither project (client or server) is in Debian (RFP 972439 for the client, none filed yet for the server), which makes deploying this for my use case rather problematic. Their install instructions are basically a curl | bash but they also provide packages for various platforms. Their Debian install instructions are surprisingly good, and check most of the third party checklist we're trying to establish. (It's missing a pin.)

There's also a Puppet module for tailscale, naturally.

What I find a little disturbing with Tailscale is that you not only need to trust Tailscale with authorizing your devices, you also basically delegate that trust also to the SSO provider. So, in my case, GitHub (or anyone who compromises my account there) can penetrate the VPN. A little scary.

Tailscale is also kind of an "all or nothing" thing. They have MagicDNS, file transfers, all sorts of things, but those things require you to hook up your resolver with Tailscale. In fact, Tailscale kind of assumes you will use their nameservers, and have suffered great lengths to figure out how to do that. And naturally, here, it doesn't seem to work reliably; my resolv.conf somehow gets replaced and the magic resolution of the ts.net domain fails.

(I wonder why we can't opt in to just publicly resolve the ts.net domain. I don't care if someone can enumerate the private IP addreses or machines in use in my VPN, at least I don't care as much as fighting with resolv.conf everywhere.)

Because I mostly have access to the routers on the networks I'm on, I don't think I'll be using tailscale in the long term. But it's pretty impressive stuff: in the time it took me to even review the Puppet modules to configure Wireguard (which is what I'll probably end up doing), I was up and running with Tailscale (but with a broken DNS, naturally).

(And yes, basic Wireguard won't bring me DNS either, but at least I won't have to trust Tailscale's Debian packages, and Tailscale, and Microsoft, and GitHub with this thing.)


IPv6 is actually what is supposed to solve this. Not NAT port forwarding crap, just real IPs everywhere.

The problem is: even though IPv6 adoption is still growing, it's kind of reaching a plateau at around 40% world-wide, with Canada lagging behind at 34%. It doesn't help that major ISPs in Canada (e.g. Bell Canada, Videotron) don't care at all about IPv6 (e.g. Videotron in beta since 2011). So we can't rely on those companies to do the right thing here.

The typical solution here is often to use a tunnel like HE's tunnelbroker.net. It's kind of tricky to configure, but once it's done, it works. You get end-to-end connectivity as long as everyone on the network is on IPv6.

And that's really where the problem lies here; the second one of your nodes can't setup such a tunnel, you're kind of stuck and that tool completely breaks down. IPv6 tunnels also don't give you the kind of security a VPN provides as well, naturally.

The other downside of a tunnel is you don't really get peer-to-peer connectivity: you go through the tunnel. So you can expect higher latencies and possibly lower bandwidth as well. Also, HE.net doesn't currently charge for this service (and they've been doing this for a long time), but this could change in the future (just like Tailscale, that said).

Concretely, the latency difference is rather minimal, Google:

--- ipv6.l.google.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 136,8ms
RTT[ms]: min = 13, median = 14, p(90) = 14, max = 15

--- google.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 136,0ms
RTT[ms]: min = 13, median = 13, p(90) = 14, max = 14

In the case of GitHub, latency is actually lower, interestingly:

--- ipv6.github.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 134,6ms
RTT[ms]: min = 13, median = 13, p(90) = 14, max = 14

--- github.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 293,1ms
RTT[ms]: min = 29, median = 29, p(90) = 29, max = 30

That is because HE.net peers directly with my ISP and Fastly (which is behind GitHub.com's IPv6, apparently?), so it's only 6 hops away. While over IPv4, the ping goes over New York, before landing AWS's Ashburn, Virginia datacenters, for a whopping 13 hops...

I managed setup a HE.net tunnel at home, because I also need IPv6 for other reasons (namely debugging at work). My first attempt at setting this up in the office failed, but now that I found the openwrt.org guide, it worked... for a while, and I was able to produce the above, encouraging, mini benchmarks.

Unfortunately, a few minutes later, IPv6 just went down again. And the problem with that is that many programs (and especially OpenSSH) do not respect the Happy Eyeballs protocol (RFC 8305), which means various mysterious "hangs" at random times on random applications. It's kind of a terrible user experience, on top of breaking the one thing it's supposed to do, of course, which is to give me transparent access to all the nodes I maintain.

Even worse, it would still be a problem for other remote nodes I might setup where I might not have acess to the router to setup the tunnel. It's also not absolutely clear what happens if you setup the same tunnel in two places... Presumably, something is smart enough to distribute only a part of the /48 block selectively, but I don't really feel like going that far, considering how flaky the setup is already.

Other options

If this post sounds a little biased towards IPv6 and Wireguard, it's because it is. I would like everyone to migrate to IPv6 already, and Wireguard seems like a simple and sound system.

I'm aware of many other options to make VPNs. So before anyone jumps in and says "but what about...", do know that I have personnally experimented with:

  • tinc: nice, automatic meshing, used for the Montreal mesh, serious design flaws in the crypto that make it generally unsafe to use; supposedly, v1.1 (or 2.0?) will fix this, but that's been promised for over a decade by now

  • ipsec, specifically strongswan: hard to configure (especially configure correctly!), harder even to debug, otherwise really nice because transparent (e.g. no need for special subnets), used at work, but also considering a replacement there because it's a major barrier to entry to train new staff

  • OpenVPN: mostly used as a client for [VPN service][]s like Riseup VPN or Mullvad, mostly relevant for client-server configurations, not really peer-to-peer, shared secrets or TLS, kind of an hassle to maintain, see also SoftEther for an alternative implementation

All of those solutions have significant problems and I do not wish to use any of those for this project.

Also note that Tailscale is only one of many projects laid over Wireguard to do that kind of thing, see this LWN review for others (basically NetbBird, Firezone, and Netmaker).

Future work

Those are options that came up after writing this post, and might warrant further examination in the future.

  • Meshbird, a "distributed private networking" with little information about how it actually works other than "encrypted with strong AES-256"

  • Nebula, "A scalable overlay networking tool with a focus on performance, simplicity and security", written by Slack people to replace IPsec, docs, runs as an overlay for Slack's 50k node network, only packaged in Debian experimental, lagging behind upstream (1.4.0, from May 2021 vs upstream's 1.6.1 from September 2022), requires a central CA, Golang, I'm in "wait and see" mode for now

  • n2n: "layer two VPN", seems packaged in Debian but inactive

  • ouroboros: "peer-to-peer packet network prototype", sounds and seems complicated

  • QuickTUN is interesting because it's just a small wrapper around NaCL, and it's in Debian... but maybe too obscure for my own good

  • unetd: Wireguard-based full mesh networking from OpenWRT, not in Debian

  • vpncloud: "high performance peer-to-peer mesh VPN over UDP supporting strong encryption, NAT traversal and a simple configuration", sounds interesting, not in Debian

  • Yggdrasil: actually a pretty good match for my use case, but I didn't think of it when starting the experiments here; packaged in Debian, with the Golang version planned, Puppet module; major caveat: nodes exposed publicly inside the global mesh unless configured otherwise (firewall suggested), requires port forwards, alpha status


Right now, I'm going to deploy Wireguard tunnels with Puppet. It seems like kind of a pain in the back, but it's something I will be able to reuse for work, possibly completely replacing strongswan.

I have another Puppet module for IPsec which I was planning to publish, but now I'm thinking I should just abort that and replace everything with Wireguard, assuming we still need VPNs at work in the future. (I have a number of reasons to believe we might not need any in the near future anyways...)

03 November, 2022 02:27PM

Alastair McKinstry

New Planning and Environment Court will reform planning appeals pro...

New Planning and Environment Court will reform planning appeals process


The Green Party has fulfilled an important Programme for Government commitment following the cabinet's decision to establish a new division of the High Court to specialise in environmental and planning issues. This is a major reform that will allow planning law to operate in a more efficient and environmentally friendly manner.

Steven Matthews TD, Green Party Spokesperson for Planning and Local Government, explained the aim of the new court;

“The goal is to ensure that our judicial system has the capacity to decide cases as quickly as possible within a growing and increasingly complex body of Irish and EU environmental law. Timely access to justice is a cornersto ne of the effective rule of law and an international legal commitment Ireland has entered under the Aarhus Convention. This is extremely pressing given the urgent need to ease the pressure on the housing system and develop the infrastructure we need to transition to a zero-carbon economy."

The Green Party is committed to ensuring the new Court will have the resources to fulfil its role. The Government will pass new legislation to allow the number of judges to be increased and provide the necessary exchequer funding to pay for these judges.

The new Court will hear all the cases currently taken by the High Court’s Commercial Planning and Strategic Infrastructure Development List and other major environmental cases. This is likely to include all major infrastructure cases as well as many cases relating to EU Environmental Law such as Environmental Impact Assessment, Strategic Environmental Assessment, Birds/Habitats, the Water Framework Directive and the Industrial Emissions Directive.

03 November, 2022 12:26PM by Alastair McKinstry

Department of Education confirms 12,544 Ukrainian pupils currently ...

Department of Education confirms 12,544 Ukrainian pupils currently enrolled in Irish schools


The Department of Education can confirm that as of 1 November, 12,544 Ukrainian pupils have been enrolled in schools across Ireland. Out of that figure, 7,948 of these pupils have been accommodated in primary schools while 4,596 pupils have enrolled in post-primary schools.

To assist with the transition of Ukrainian refugees and their families into Irish schools Regional Education and Language Teams (REALT) continue to operate, hosted by the 16 regional education and training boards around Ireland and staffed by regionally based education support personnel.

These teams are ensuring that clear, accessible information flows are in place between schools, local education support services and national support structures in relation to people arriving from Ukraine.

Information for parents in Ukrainian and Russian on accessing school places is available on gov.ie/Ukraine.

The Department will continue to publish figures on the enrolment of Ukrainian children each month.


03 November, 2022 10:55AM by Alastair McKinstry

Russ Allbery

Review: Carpe Jugulum

Review: Carpe Jugulum, by Terry Pratchett

Series: Discworld #23
Publisher: Harper
Copyright: 1998
Printing: May 2014
ISBN: 0-06-228014-7
Format: Mass market
Pages: 409

Carpe Jugulum is the 23rd Discworld novel and the 6th witches novel. I would not recommend reading it before Maskerade, which introduces Agnes.

There are some spoilers for Wyrd Sisters, Lords and Ladies, and Maskerade in the setup here and hence in the plot description below. I don't think they matter that much, but if you're avoiding all spoilers for earlier books, you may want to skip over this one. (You're unlikely to want to read it before those books anyway.)

It is time to name the child of the king of Lancre, and in a gesture of good will and modernization, he has invited his neighbors in Uberwald to attend. Given that those neighbors are vampires, an open invitation was perhaps not the wisest choice.

Meanwhile, Granny Weatherwax's invitation has gone missing. On the plus side, that meant she was home to be summoned to the bedside of a pregnant woman who was kicked by a cow, where she makes the type of hard decision that Granny has been making throughout the series. On the minus side, the apparent snub seems to send her into a spiral of anger at the lack of appreciation.

Points off right from the start for a plot based on a misunderstanding and a subsequent refusal of people to simply talk to each other. It is partly engineered, but still, it's a cheap and irritating plot.

This is an odd book.

The vampires (or vampyres, as the Count wants to use) think of themselves as modern and sophisticated, making a break from the past by attempting to overcome such traditional problems as burning up in the sunlight and fear of religious symbols and garlic. The Count has put his family through rigorous training and desensitization, deciding such traditional vulnerabilities are outdated things of the past. He has, however, kept the belief that vampires are at the top of a natural chain of being, humans are essentially cattle, and vampires naturally should rule and feed on the population. Lancre is an attractive new food source. Vampires also have mind control powers, control the weather, and can put their minds into magpies.

They are, in short, enemies designed for Granny Weatherwax, the witch expert in headology. A shame that Granny is apparently off sulking. Nanny and Agnes may have to handle the vampires on their own, with the help of Magrat.

One of the things that makes this book odd is that it seemed like Pratchett was setting up some character growth, giving Agnes a chance to shine, and giving Nanny Ogg a challenge that she didn't want. This sort of happens, but then nothing much comes of it. Most of the book is the vampires preening about how powerful they are and easily conquering Lancre, while everyone else flails ineffectively. Pratchett does pull together an ending with some nice set pieces, but that ending doesn't deliver on any of the changes or developments it felt like the story was setting up.

We do get a lot of Granny, along with an amusingly earnest priest of Om (lots of references to Small Gods here, while firmly establishing it as long-ago history). Granny is one of my favorite Discworld characters, so I don't mind that, but we've seen Granny solve a lot of problems before. I wanted to see more of Agnes, who is the interesting new character and whose dynamic with her inner voice feels like it has a great deal of unrealized potential.

There is a sharp and condensed version of comparative religion from Granny, which is probably the strongest part of the book and includes one of those Discworld quotes that has been widely repeated out of context:

"And sin, young man, is when you treat people as things. Including yourself. That's what sin is."

"It's a lot more complicated than that—"

"No. It ain't. When people say things are a lot more complicated than that, they means they're getting worried that they won’t like the truth. People as things, that's where it starts."

This loses a bit in context because this book is literally about treating people as things, and thus the observation feels more obvious when it arrives in this book than when you encounter it on its own, but it's still a great quote.

Sadly, I found a lot of this book annoying. One of those annoyances is a pet peeve that others may or may not share: I have very little patience for dialogue in phonetically-spelled dialect, and there are two substantial cases of that here. One is a servant named Igor who speaks with an affected lisp represented by replacing every ess sound with th, resulting in lots of this:

"No, my Uncle Igor thtill workth for him. Been thtruck by lightning three hundred timeth and thtill putth in a full night'th work."

I like Igor as a character (he's essentially a refugee from The Addams Family, which adds a good counterpoint to the malicious and arrogant evil of the vampires), but my brain stumbles over words like "thtill" every time. It's not that I can't decipher it; it's that the deciphering breaks the flow of reading in a way that I found not at all fun. It bugged me enough that I started skipping his lines if I couldn't work them out right away.

The other example is the Nac Mac Feegles, who are... well, in the book, they're Pictsies and a type of fairy, but they're Scottish Smurfs, right down to only having one female (at least in this book). They're entertainingly homicidal, but they all talk like this:

"Ach, hins tak yar scaggie, yer dank yowl callyake!"

I'm from the US and bad with accents and even worse with accents reproduced in weird spellings, and I'm afraid that I found 95% of everything said by Nac Mac Feegles completely incomprehensible to the point where I gave up even trying to read it. (I'm now rather worried about the Tiffany Aching books and am hoping Pratchett toned the dialect down a lot, because I'm not sure I can deal with more of this.)

But even apart from the dialect, I thought something was off about the plot structure of this book. There's a lot of focus on characters who don't seem to contribute much to the plot resolution. I wanted more of the varied strengths of Lancre coming together, rather than the focus on Granny. And the vampires are absurdly powerful, unflappable, smarmy, and contemptuous of everyone, which makes for threatening villains but also means spending a lot of narrative time with a Discworld version of Jacob Rees-Mogg. I feel like there's enough of that in the news already.

Also, while I will avoid saying too much about the plot, I get very suspicious when older forms of oppression are presented as good alternatives to modernizing, rationalist spins on exploitation. I see what Pratchett was trying to do, and there is an interesting point here about everyone having personal relationships and knowing their roles (a long-standing theme of the Lancre Discworld stories). But I think the reason why there is some nostalgia for older autocracy is that we only hear about it from stories, and the process of storytelling often creates emotional distance and a patina of adventure and happy outcomes. Maybe you can make an argument that classic British imperialism is superior to smug neoliberalism, but both of them are quite bad and I don't want either of them.

On a similar note, Nanny Ogg's tyranny over her entire extended clan continues to be played for laughs, but it's rather unappealing and seems more abusive the more one thinks about it. I realize the witches are not intended to be wholly good or uncomplicated moral figures, but I want to like Nanny, and Pratchett seems to be writing her as likable, even though she has an astonishing lack of respect for all the people she's related to. One might even say that she treats them like things.

There are some great bits in this book, and I suspect there are many people who liked it more than I did. I wouldn't be surprised if it was someone's favorite Discworld novel. But there were enough bits that didn't work for me that I thought it averaged out to a middle-of-the-road entry.

Followed by The Fifth Elephant in publication order. This is the last regular witches novel, but some of the thematic thread is picked up by The Wee Free Men, the first Tiffany Aching novel.

Rating: 7 out of 10

03 November, 2022 05:16AM

November 02, 2022

hackergotchi for Robert McQueen

Robert McQueen

Many thanks & good luck to Neil McGovern

As President of the GNOME Foundation, I wanted to post a quick note to pass on the thanks from the Board, the Foundation staff team and membership to our outgoing Executive Director, Neil McGovern. I had the pleasure of passing on GNOME’s thanks in person at the Casa Bariachi this summer at GUADEC in Guadelajara, at the most exellent mariachi celebration of GNOME’s 25th Anniversary. 🤠 Kindly they stopped the music and handed me the microphone for the whole place, although I think many of the other guests celebrating their own birthdays were less excited about Neil’s tenure as Executive Director and the Free and Open Source desktop in general. 🤣

Neil’s 6-month handover period came to an end last month and he handed over the reins to myself and Thibault Martin on the Executive Committee, and Director of Operations Rosanna Yuen has stepped up to act as Chief of Staff and interface between the Board and the staff team for the time being. Our recruitment is ongoing for a new Executive Director although the search is a little behind schedule (mostly down to me!), and we’re hugely grateful to a few volunteers who have joined our search committee to help us source, screen and interview applicants.

I have really enjoyed working closely with Neil in my time on the GNOME board, and we are hugely grateful for his contributions and achievements over the past 5 years which I posted about earlier in the year. Neil is this month starting a new role as the Executive Director of Ruby Central. Our very best wishes from the GNOME community and good luck with your new role. See you soon!

(also posted to Discourse if you wish to add any thanks or comments of your own)

02 November, 2022 12:34PM by ramcq

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, October 2022

In October I was not assigned additional time by Freexian's Debian LTS initiative, but carried over 9 hours from September and worked all those hours.

I updated the linux (4.19) package to the latest stable update, but didn't upload it. I merged the latest bullseye security update into the linux-5.10 package, uploaded that, and issued DLA-3173-1.

02 November, 2022 11:36AM