Nick Craig-Wood's articles on Nick Craig-Wood's home page

Infinity mirror with Raspberry Pi Pico, Neopixels and Micropython

nick@craig-wood.com (Nick Craig-Wood) — Sat, 11 Feb 2023 00:00:00 +0000

Infinity mirror with Raspberry Pi Pico, Neopixels and Micropython

Here is an account of how I renovated our old infinity mirror to this:

From this:

This isn't a recipe on how to do this yourself (though you could certainly use it like that), its an account of exactly what I did, including all the bits where I wasn't sure where I was going in the hope it will help less experienced constructors with the thought process.

The mirror

A long time ago my wife and I were looking in a lighting shop and we came across the infinity mirror. I was immediately captivated by it and we had it installed in our bathroom when we had it renovated.

Fast forward more than a decade and the lights in the infinity mirror had stopped working. I had never taken it apart to see how to fix it which I really should have done but you know how you get used to things not working...

Just before Christmas 2022 I decided I would like to have a project to work on over the holiday period, so after a very brief bit of research I ordered:

40 plated through hole neopixel LEDs
2 raspberry pi pico W microcontrollers
5V 2A PSU
40 x 330uF capacitors
40 x 330Ohm resistors
Some more wire since I had run out

I chose the plated through LED style neopixels because I knew they'd be visible in the mirror and I wanted them to look nicer than the neopixel strips. I knew I needed 34 LEDs but I ordered some extra for testing. I ended up breaking a couple of them so just as well.

I chose the pi pico because I'd done a project with that before with C++ but I thought it would be fun to try micropython (which I hadn't tried before).

The resistors and caps I chose after a bit of looking at the datasheet. Spoiler - I didn't need these in the end and I did need some decoupling caps instead - see later.

I made that order in the full knowledge that it wouldn't be everything I needed and that there may be stuff in there that I didn't need. When getting going on a project its much more important for me to make a start even if it is in the wrong direction than to wait until I've planned everything out to the last degree.

Later I ended up ordering:

40 x 10nF ceramic capacitors
50 x bezel mounts for the 5mm LEDs
5 x 100k pots with nice knobs
5 x switches

Neopixels

Once the stuff arrived I took some photos to see how the neopixels worked inside.

You can see the 3 LED chips and also the controller chip all connected together with very fine wire in the LED encapsulation.

Neopixels have 4 connections, Volts, Ground, Data In and Data Out. You chain them up Data In to Data Out and you connect the first Data In to your microcontroller (in this case a Raspberry Pi Pico). The microcontroller generates a serial data stream with 24 bit color for each neopixel. The first neopixel uses the first block of pixel data to color itself and passes the rest of the stream on. This means you can chain hundreds of them together with only one pin on your microcontroller and address each one individually.

Neopixels and micropython

The first thing I wanted to do was to make a neopixel work from micropython to give me some confidence that the project would work.

I made a plan like this:

Install micropython on the pi pico and make it work
Breadboard a single neopixel to see if that works
Breadboard a chain of neopixels to see if they work

I fully expected these steps to be very hard - bringing up new hardware is always troublesome.

Installing micropython

Installing micropython turned out to be very easy. I installed the micropython firmware from here

https://www.raspberrypi.com/documentation/microcontrollers/micropython.html

Using Pico W version rp2-pico-w-20230116-unstable-v1.19.1-803-g1583c1f67.uf2

I then needed to work out how to communicate with micropython. I tried thonny but I don't really like graphical clicky GUIs and eventually I found that Ubuntu packaged rshell which worked and let me use a REPL on the board and transfer files into the flash.

This was so easy compared to microcontroller development I've done in the past - well done Raspberry Pi and Micropython.

Having a REPL on the board meant I could do little experiments to see if bits of hardware were working just by typing python snippets.

First light

I soldered on 7 pins to the pico (just enough for power, IO and to stop it wobbling) then I plugged it into a breadboard with a single neopixel.

It took me a little while to discover that there is a built in neopixel library in the pico micropython. I found an external library first but decided to use the built in library after evaluating it.

I could then use the REPL from rshell to write code and try to make neopixel work.

This should set the LED to a pink color:

from neopixel import NeoPixel
import machine
pin = machine.Pin(0)
pixels = NeoPixel(pin, 1)
pixels.fill((255, 5, 20))
pixels.write()

However it didn't set it to pink at all! Eventually I worked out that the ordering of RGB seems to be wrong. I looked in the source for the neopixel python library and saw I could fix it like this:

NeoPixel.ORDER = (0, 1, 2, 3)

So a test of the LED became:

from neopixel import NeoPixel
NeoPixel.ORDER = (0, 1, 2, 3)
import machine
pin = machine.Pin(0)
pixels = NeoPixel(pin, 1)
pixels.fill((255, 5, 20))
pixels.write()

It was a relief to get over that hurdle - it seemed then that the project might work after all.

Fifth light

I then extended the breadboard with 4 more neopixels for 5 in total and tried to make that work.

I then had fun noodling around with python to try to make some interesting patterns.

Note that the neopixels are extremely bright when on full power so I did most of this testing with a bit of white paper over the neopixels. When the neopixels are in the mirror they will shine sideways not towards the viewer to make interesting internal reflections. This worked out very well.

Knobs

At this point I realised it would be nice if the user had some control over the display. A brightness knob at minimum would be nice!

The Pico has 3 ADC pins so I bought some 100k pots with knobs off eBay and wired them up and soldered them to the pico.

Later I discovered that 100k was too big a value as they are a bit noisy and can't quite pull up and down to the rails due to the internal resistance of the pico ADC. I chose the 100k to save a bit of power but it was a mistake - 10k pots would have been better. I left them at 100k though and did a bit of work in the software to make sure they could hit completely off and completely on.

I also at this point decided to add two buttons for mode up and mode down but I didn't make them until later.

Mechanical work

Now that I had proved the concept of the pico and neopixels it was time for me to disassemble the mirror. I really should have done this before I started, but I guess I enjoy the feeling of not knowing what I'm doing ;-)

You can see the mirror has what is essentially a string of Christmas lights inside. These gave a beautiful creamy light so I hope I don't destroy that with the LEDs.

Each Christmas light came out through a hole about 10mm in diameter and I had to work out how to fit the LEDs which are 5mm in diameter into those holes neatly. This is all very visible so it had to look nice.

This was probably the hardest part of the project for me. If I had a 3D printer I guess I could have printed something up.

After a lot of messing about, I eventually bought some panel mounts for the LEDs. This made them up to 8 mm in diameter. They were not quite big enough and it took a lot of force to get the LEDs far enough into the panel mounts. I broke a couple of LEDs doing it!

Then I cut about 1cm off the end of all the Christmas lights (with a saw!) so I could re-use the little bit of green plastic tube you see below and put the panel mount into those which luckily fit perfectly.

At last the LEDs were ready to fit into the 10mm holes in the frame.

I then soldered the wires to connect up the neopixels. I also soldered a 10nF decoupling capacitor across the power lines for each one. The datasheet recommended feeding each one through a resistor with a reservoir capacitor, but in my tests the 5V PSU I had bought was plenty capable of dealing with the power surges so I just went for a decoupling capacitor to minimise noise.

This was a lot of soldering - it took about 4 hours to make and solder all the wires. There are 3 wires for each LED that needed making and 4 solder joints, so 204 wire ends to strip and tin and 102 joints to make with 3 or 4 things being soldered together!

With the benefit of 20/20 hindsight and a bit more searching eBay and Amazon I could have bought a string of Christmas light neopixels which probably would have done the job with much less soldering.

I then connected the breadboard and made sure the lights worked.

Software

Now the hardware unknowns were beaten for the moment it was time to get back to the software.

You can download the software from github.

One thing I did do which I recommend you always do when working with embedded stuff is write a simulator.

I used pygame for this. This meant I could test and debug the display modes on my desktop which was a much quicker cycle than testing on the raspberry pi pico. It's also a nice thing to have - you can download the code from the github repo and try it yourself even if you don't have any hardware.

I went through a few iterations of how exactly to split the code between the simulator and micropython but in the end passing a mirror object to each mode turned out the easiest.

I experimented with different frame rates and I found that 50 Hz was too fast - some of the modes were dropping frames so I settled on 25 Hz.

I also experimented with running the refresh off a timer interrupt in micropython. This worked for a while and then locked up and I conjecture I was doing too much on the interrupt. I switched it over to a timed sleep and it became 100% reliable after that.

I experimented with lots of different modes and had the idea of using the board's temperature sensor for a temperature display mode and using the on board WiFi to pick up the current time for a clock mode. The clock mode actually works really well - with a few minutes' practice you can read it just like any analogue clock.

Final assembly

Now that the software was looking good I went back to the hardware side for the final assembly.

I needed to get the pico off the bread board and mounted into the mirror somehow.

In the end I decided to make a piece of wood which I would velcro underneath the mirror (there is a small gap so it won't be visible) and put the pico and knobs there.

This taxed my meagre carpentry skills. I made a mistake with the knobs - the bit of wood I chose was too thick to put the nuts on the knob shafts to hold them in which was a bit of a disaster. However I decided to epoxy them and the buttons in place which made a nice result provided I don't ever want to take them off again.

I fed the power (5V, GND) in to the top of the mirror and out of the bottom of the mirror comes 3 wires, 5V, GND and the input to the string of neopixels. The 5V line goes via a diode to VSYS on the pico so that the pico can be powered via the USB or by the external 5V PSU. This arrangement is outlined in the datasheet for the pico.

VBUS is the 5V input from the micro-USB port, which is fed through a Schottky diode to generate VSYS. The VBUS to VSYS diode (D1) adds flexibility by allowing power ORing of different supplies into VSYS.

and later

The simplest way to safely add a second power source to Pico is to feed it into VSYS via another Schottky diode (see Figure 15). This will 'OR' the two voltages, allowing the higher of either the external voltage or VBUS to power VSYS, with the diodes preventing either supply from back-powering the other.

We will run the LEDs directly from an external 5V power supply. Since the VSYS input range is 1.8V to 5.5V we don't need to worry about a diode drop, so any silicon diode should be fine.

So via a diode connect external 5V PSU to VSYS
Leave VBUS unconnected (since it connects directly to the USB power)

This will allow us to connect a computer and not blow up the computer with the external 5V PSU.

The neopixel drive is connected to GP0 via a 330Ohm resistor. Why the resistor? Just in case the pin gets shorted to 5V - that will save the pico in that case.

I made connectors for the 3 wires to the pico board so I could take it off and things are very nearly done.

Conclusion

That was a lot of fun. The mirror is somewhat mesmerising in the color cycling modes. It also made the wife happy to have the infinity mirror working again, so win-win-win :-)

Thanks

Dougal for helping me with the wire stripping
Dave Lawrence for writing the Matrix and ChaseRGB modes

Deadlocks: the dark side of concurrency

nick@craig-wood.com (Nick Craig-Wood) — Mon, 25 Oct 2021 00:00:00 +0000

This is a transcript of a talk I gave at Gophercon UK on 2021-10-25. You can watch it on YouTube (47 minutes) or see the slides and read the words here.

Go makes it so easy to write concurrent programs that sooner or later you will run into a deadlock. A deadlock is where your program (or part of it) just stops working for mysterious reasons.

This talk will explain what deadlocks are and why the Go deadlock detector isn't very useful in real world programs.

Deadlocks can be very difficult to find and fix and this talk will give lots of tips on how to figure out why your program deadlocked and how to fix it and avoid deadlocks in the future.

You will need an understanding of go routines, locking (sync.Mutex) and channels to get the most out of this talk. No previous deadlock experience required!

Introduction

Hello and welcome everyone.

I’m Nick Craig-Wood and I’m going to be talking to you about deadlocks.

This is an intermediate level talk, so I’m assuming you know what a go routine is and have some idea what a channel and a mutex is.

I’ve put a link to the code samples up there – you don’t need them as I’ll be putting them on the screen, but you can have them for reference during the talk or later.

In this talk we are going to talk about what a deadlock is, how to debug them and how to avoid them in future.

It will have lots of real world examples culled from dozens of painful examples in the rclone codebase!

I learned about deadlocks the hard way by having to debug lots of them and I’m aiming of for this to be the talk I would have like to listened to when I knew nothing.

Hopefully everyone, no matter how experienced will pick up a few tips.

But first a little bit about me. At the moment I’m mostly doing Open Source coding on rclone with a bit of consultancy.

I’ve been using Go for 9 years since before version 1.0. I got into it when I was a busy CTO and found I was doing nearly 100% people management and strategy with no time for coding. So I decided I’d learn Go in my spare time because it sounded interesting and I needed a programming activity to keep me sane.

After writing a few small things in Go, I started started the rclone project in 2012 and its been growing ever since, now with 30 thousand stars on GitHub and millions of downloads a month.

That’s been quite a ride but I’ll save that story for a different talk!

Since all the real world examples in this talk come from rclone I’ll tell you a little bit about it.

Rclone is a highly concurrent command line tool to sync your data to and from any cloud storage system. It supports over 40 providers including s3, dropbox, google drive, azure and lots more.

It tries to do everything it can to maximise your network and disk bandwidth and minimise the chance of data corruption.

It can also mount your cloud storage system as a network drive using its FUSE filing system.

To make it run fast rclone is riddled with Go routines, and the inevitable channels and locks which go with them and in turn the inevitable deadlocks.

Before I started on the rclone journey I knew nothing about deadlocks!

What is a deadlock?

So what is a deadlock?

Here is Wikipedia’s rather abstract definition of a deadlock – something to do with one process holding something another process needs.

I prefer to think of them as a catch 22 situation.

Rather like a newly graduated developer: you need experience to get a job, but you can’t get experience until you get a job.

In a deadlock the other process has something you want, and you have something the other process wants and you are both prepared to sit there and block forever until the other process gives you what you need.

Let’s go through a deadlock in diagram form.

Here we have two go routines (the blue circles labelled Process 1 and Process 2) and two locks (the pink squares).

Each lock could be a sync.Mutex or a Read Write Mutex or maybe even a write to a channel.

The first step is Process 1 takes Lock 1. All good so far.

Next step is process 2 takes lock 2. Again all good, nothing to see here.

Now process 1 attempts to takes lock 2. This causes Process 1 to block and wait for lock 2 to be released as lock 2 is already taken by process 2.

We haven’t reached deadlock yet as everything could be resolved as soon as Process 2 drops Lock 2.

However it doesn’t do that...

Process 2 now tries to take Lock 1.

This is unsuccessful because Lock 1 is taken by Process 1 so Process 2 blocks waiting for it to be released.

Now Process 1 and Process 2 are blocked. They will never proceed as each is waiting for the other to release a lock.

These two processes are now deadlocked.

This means that some part of your program has stopped working and will never work again.

Let’s see what that might look like in Go with a simple bit of code.

We have process 1 and process 2 just like in the previous example and they are going to sit there in a loop, taking and releasing lock1 and lock2.

The only difference between process1 and process2 is that they take and release the locks in a different order.

This code looks like it should work, but it will deadlock pretty soon.

This is the output once you run the code.

First thing to notice is that the Go runtime has detected the deadlock giving the nice friendly message “all go routines are asleep – deadlock”.

The runtime gives a backtrace of all the Go routines that are running which is very useful for working out what went wrong.

The disadvantage of this is that they can be very long – even this tiny example needed 2 slides to fit it on.

Go routine 1 is waiting on the select with no cases that we put in to stop the program ending too soon. We can ignore that.

Goroutine 17 is in state semacquire – its deadlocked calling the sync Mutex implementation.

Here is the next bit of the output.

In goroutine 19 you can see a different bit of our code in also calling the sync Mutex implementation. Again it is deadlocked in the semacquire state.

I’ve put arrows at the bits of code that the backtrace indicates the go routines are executing.

You can see process1 has taken lock1 and is blocked taking lock2, whereas process 2 has taken lock2 and is blocked taking lock1.

This is a clear deadlock. Whenever you see a trace where a single go routine takes more than one lock, you should be thinking about deadlocks.

This is exactly what we expected from our initial diagram, process 1 is blocked taking lock 2 and process 2 is blocked taking lock 1.

It is nice to see theory and practice agreeing!.

You’ll have noticed that Go can detect deadlocks.

This is very useful, but unfortunately it very rarely detects deadlocks in actual code.

This is because all the Go routines in your entire program have to be asleep for it to trigger and in my experience that almost never happens except in toy examples like these.

There is always something else going on, like log collection or statistics calculations or something like that which means the race detector does not fire.

This time if we add one extra go routine as indicated the deadlock detector does not kick in, and the program deadlocks silently.

Or rather part of the program - process 1 and process 2 - is deadlocked but the rest of it is working fine.

Unfortunately this is the usual state of affairs when debugging deadlocks – some part of your code just stops working with no other symptoms.

Enough of your code may be working to give you the idea its it isn’t broken but some important part isn’t progressing.

We’ll see in the next section of the talk what to do at this point.

A quick diversion, as I expect this question has popped into your minds already.

Can we use channels instead of those nasty Mutexes to avoid deadlocks?

Alas, no, you can cause a deadlock with channels just as easily as with Mutexes and we’ll see an example of that later.

In my my experience with rclone only about 10% of the deadlocks are with channels so they are less likely certainly.

So why do we get deadlocks?

There are two ingredients.

The first is concurrency. Without concurrency, that is things happening at the same time, we can’t have deadlocks.

This means that single threaded code can’t have deadlocks which is a point we’ll come back to later.

The next magic ingredient for deadlocks is locking.

When I say Locking here I’m including channels. Reading and writing to channels can block your program just as easily as Mutexes.

And if I wanted to be really precise I’d say shared resource acquisition instead of Locking because you can get deadlocks using other things other than Mutexes and channels, for example using file system locks.

So where did these locks come from?

My excuse is the race detector made me do it :-)

So what do we do to pacify the race detector? We put locking in, and sooner or later once you’ve put enough locking in you will cause a deadlock.

I’m joking here of course, the go race detector is wonderful.

It tells us about data races which would have caused silent corruption in our programs.

Let’s just sing the virtues of the race detector for a moment. It is a technological marvel and very close to magic.

The race detector will tell you when your program runs a risk of memory corruption due to unsafe accesses.

If you are writing concurrent programs you should run your tests with the race detector, and if you can, some of your production binaries too.

Any problems it finds are always real problems – I’ve never known it make a mistake.

However when fixing the things the race detector points out, do think about what deadlocks you are about to create.

In particular think about what locks might be already held and what locks you might want to take in this go routine.

How to debug a deadlock

Here is an example I’m going to use quite a few times.

It is a simplified example from rclone’s FUSE based filing system for mounting remote cloud storage as if it were a local disk.

On the left you can see the definition of the File struct. It has a name, size and a parent directory. It has a Path method which returns the full path of the File and it needs to ask its parent directory for its Path in order to do that. It has a Size method to return its size and a Truncate method showing how the size of the file might get changed.

On the right we see the definition of the Directory struct. This has a name, a parent directory, a slice of files it contains and a slice of directories it contains. It has a Path method for returning its full path and it needs to find out the Path of its parent here. It also has a Size method which adds up the sizes of all the files in the directory.

There is no locking in the code above, yet, but a few runs with the race detector reveals that locking is needed.

To prevent those nasty data races we add a sync.Mutex called mu into the File structure. The name, lower case mu is traditional in Go code.

Then in each of the methods we lock the mutex with mu.Lock() and defer the unlock of the mutex with the defer mu.Unlock().

This means that none of the methods will ever run concurrently and it means our instance variables are safe from data race corruption – hooray!

This is a standard pattern used in Go concurrent programming and hopefully you’ve seen it before.

Take the lock at the start of each method and defer its release to the end – what could possibly go wrong?

We do the same thing in the Directory structure, add a sync.Mutex called mu to the struct and then call mu.Lock() in each method and defer the mu.Unlock() until the method returns.

However we’ve unwittingly caused a potential deadlock by doing this.

I say potential deadlock here because when you run this code it will quite likely work just fine until just the right conditions are met.

Most deadlocks are like this – they don’t stop your code the first time you create them, they come back to bite you later when you are least expecting them.

Usually just after you’ve put your code into production.

Here is a bit of code to exercise the Directory and File abstractions and which will provoke the deadlock.

Process1 just calls File.Path in a loop continuously. Not exactly normal usage, but we’re trying to provoke the deadlock here.

Process2 just calls Directory.Size in a loop continuously.

In our main function we create a root directory and put a file into it then start go routines to call process 1 and process 2.

We also start a race detector defeating go routine which just sleeps for a minute.

And finally the empty select statement to sleep the main go routine forever. Remember if we exit the main go routine then our program will stop, and we don’t want that.

So we run the program and it is now sitting there doing a whole heap of nothing while deadlocked.

How can we find out what it is doing?

If you are running on a Unix based platform (for example Linux or macOS) then you can send the process a SIGQUIT using the kill command or press Control Backslash. This will produce a nice backtrace.

Windows platform users will have to work a little harder and install the HTTP debug handler from net/http/pprof and I’ll show how to do that a bit later.

This is what happened when I sent the File and Directory example program the QUIT signal.

I got this nice backtrace. It’s quite long but there is a lot of stuff we can ignore.

The output is divided into stanzas and at the start of each one is a little description, Here it says goroutine 0, idle.

Following that is a reference to a line of source code, plus the code itself, once for each level of the call stack.

In fact this is a system go routine doing something in the runtime – we can ignore it because it doesn’t have any of our code in the backtrace.

Next we see go routine 1, select, no cases. This is the wait forever select code which we put in to stop the main function existing. We can ignore this too.

Next we see go routine 5 semacquire. This one is important, semacquire means it is blocked when trying to acquire a lock. For most deadlocks you are searching for these semacquire lines first.

Below the semacquire you can see a couple of lines of code in the runtime – these are in the sync.Mutex code, and finally a bit of our code in Dir.Path.

We can see that Dir.Path was called by File.Path so by the time the go routine tries to acquire the Directory lock it has already acquired the File lock.

Next we see goroutine 6 – also blocked on semacquire.

It looks quite similar to the last one, two lines in sync.Mutex code and then it tring to acquire the mutex in File.Size.

Looking up the backtrace you can see that File.Size was called from Dir.Size which means that this go routine acquired the Dir lock first and is now blocked on trying to acquire the File lock.

If you have an editor you can click on those file names to jump to the code in question it is really helpful at this point so you can look up and down the call stack to see what other locks might be held.

Under that we see go routine 7 sleep which is our deadlock defeating go routine which we can ignore.

And finally the register dump which you can ignore for this purpose.

I’m sure the register dump is useful for low level debugging, maybe if you are writing assembler, but I’ve never needed it to debug deadlocks.

Here I’ve annotated the code from the backtrace showing where each go routine is blocked.

The fact that both parts of the code are blocked on acquiring a Mutex are very suggestive that there is a deadlock.

Let’s look at Dir.Path on the left hand side of the screen. We saw from examining the backtraces that this go routine already holds the file lock and is now blocked trying to acquire the directory lock.

On the right File.Size is already holding the directory lock but is blocked trying to acquire the file lock.

This means that these two functions are deadlocked and will block forever because the locks they are waiting for will never be released.

This deadlock occurs whenever the Size method on Dir is called concurrently with the Path method on File.

When thinking about deadlocks it is helpful to write out a timeline of the two processes involved. The numbers on the left of the code show the ordering of the calls to each process.

The arrows point from the blocked locks to the place the lock is held, and them crossing is very suggestive of a deadlock.

You might think that there is very little opportunity for this to happen given the short windows of opportunity – each of these instructions will be over in a few nano seconds won’t it? However, if the deadlock is possible it will happen sooner or later.

One thing to bear in mind is that locking instructions are particularly slow – they use slow memory locked instructions or schedule go routines or processes - which often turns something you thought would never happen into something which always happens.

I’ll just quickly loop back to how to install the HTTP debug handler which you can use to collect backtraces when running Windows.

Look up the docs for net/http/pprof – everything you need is in there.

You can collect the output with curl or just look at in your browser.

You can use this on any platform of course, and you might consider leaving it in your production code for when your production code has deadlocked and you want to find out why.

The output from the HTTP debug handler is similar to the output from SIGQUIT with one stanza for each go routine and a trace showing position in file and the actual line of code being executed.

It doesn’t call out the state of each go routine in the same way as the SIGQUIT output, but it is easy to see that the two go routines here are each blocked on SemaqcuireMutex which means blocked on acquiring a mutex as you might expect.

One important difference is that this output collapses goroutines with identical stacks which can very useful if you’ve got 1000s of goroutines.

When you are tracking down a deadlock in a real world program with lots of concurrency, you’ll find that the backtraces get very long, very quickly.

Deadlocks have a nasty tendency to spread through a good fraction of your running go routines until they are all waiting on a lock kind of like the way something crystallizes out of a solution.

Here is an example of an backtrace a user sent me while debugging an rclone deadlock.

It has nearly 2000 lines covering 260 go routines and is par for the course when dealing with lots of concurrency. There are 14 go routines blocked on acquiring a mutex, 123 blocked in a select statement, 121 blocked on sending or receving to a channel and 1 in a system call and 1 idle.

If that sounds like looking for a needle in a haystack, it is, we need some help here!

Luckily there is a great tool called panic parse which helps enormously here.

It condenses down that 2000 line log into a nicely coloured 142 line log with one stanza for each different type of go routine.

It shows that there are only 14 different types of go routine here which makes working out what is going on so much easier.

It also has a nice web output which I haven’t tried yet, but I’ve put up the screenshot from the github page so you can see what it is like.

This tool has saved me so much time when debugging deadlocks – thank you Marc-Antoine Ruel!

The text output of panic parse shows a stanza for each go routine. It collapses go routines with similar stacks together like the HTTP debug handler, but better.

Here is what a stanza looks like. On the left you can see a number – in this case 7 – this is how many go routines like this there were, and really usefully the time it has been paused. Looking for the first goroutine to pause can often be a big clue.

It is then reasonably easy to see that there are only a few go routines that could possible be the problem, namely the ones that are waiting on locks that have taken locks already so we can immediately ignore nearly all the go routines.

I’m not going to lie, it’s still tricky working out exactly what the cause of the deadlock from here is, but it is so much easier with panic parse.

How to avoid deadlocks

We’ve talked so far about what a deadlock is and how to find them and debug them, but a better strategy would be to avoid them completely.

I’m going to talk through a number of different techniques for avoiding deadlocks. Unfortunately there isn’t just one thing I can say – do this and you’ll never get a deadlock again, alas.

We’re going to start with some easy things to fix then move on to some harder problems.

I said there was going to be an example of deadlocks with channels a bit later and here it is!

Let’s start with a Job. You can see its struct at the top.

It has a constructor NewJob which doesn’t do a lot in this example.

The Process method for each job runs the Job. However in the process of running a Job, more Jobs might be created. In this example it picks a random number as to whether to create more jobs or not.

Those jobs are then returned in a slice for further processing.

This is simplified from an rclone example where each Job listed a directory and returned jobs to list sub directories.

Here the ProcessJobs function takes a WaitGroup and a channel of jobs. When running it takes a job out of the channel, processes it and adds any created jobs back onto the channel.

It is written like this with a channel so we can run a few copies of ProcessJobs concurrently to increase the throughput and that is what happens in the main routine - We start 3 copies of the ProcessJobs function to increase the throughput.

Its worth explaining what the sync.WaitGroup does here. We are using it to keep track of the number of active jobs – whenever a job is created it is increased with the Add(1) method and whenever a job is finished it is decreased with the Done method.

When the wait group goes to zero, we know all the jobs have finished and we wait for that moment with the Wait method.

When we run it, it works fine for a while, running jobs, creating new ones, adding them to the channel, until...

… you guessed it, we get a deadlock.

The first go routine shown, go routine 1, is our sync WaitGroup waiting for the others to finish.

Goroutine 6, 7 and 8 are all in state chan send. That is they are waiting for a channel to be emptied before they can send some data down it. You can see that they are all waiting in ProcessJobs.

The problem is that somehow all three of our go routines are blocked writing to the channel and none of them are reading from it.

Because those 3 go routines are the only place that we read from the channel, it means that the channel will never get emptied and those 3 go routines will never get unblocked, creating a deadlock.

This is the code with the deadlock positions marked on it.

The three ProcessJobs functions are all trying to write a new Job into the jobs channel. However none of them is reading from the channel, so none of those writes will ever succeed A classic deadlock, this time with channels.

We can also see the sync.Waitgroup Wait method being called to wait for the Job processing to be complete. This is now part of the deadlock too as it will never proceed.

The Waitgroup Wait isn’t the cause of the deadlock, but it is collateral damage and this is very common when looking at deadlocks. Often the collateral damage overwhelms the cause of the deadlock and that makes it very hard to work out what is going on.

So what is the solution?

We need the ProcessJobs function to be either reading from the channel or processing a Job. We don’t ever want it to be blocked on writing to the channel, because if one go routine can get blocked then all of them can.

The easiest way of achieving this is by sending into the channel in a new go routine. The original Go routine is released to receive jobs from the channel while the new go routine sends the new jobs to the channel.

This has the consequence that there might be an indefinite number of go routines with a few jobs to add to the jobs channel hanging around. If that is a problem then a different design is needed.

Note we also moved the activeJobs.Done call into the new go routine – we can’t signal a job is finished until all its Job children are safely queued. If we do then the main function will end too early.

The moral of this channel deadlock is simple. Don’t send and receive to the same channel in the same go routine – it is a recipe for deadlocks.

As I said earlier this example came straight from rclone. This dead lock took some time to appear and it took me a few iterations to a find a solution which didn’t deadlock.

Concurrency is hard!

Here is an example with read write locks.

We have two main functions RO and RW. RO takes a read lock then calls MoreRO which in turn takes the Read lock again. This is allowed – read locks can be taken multiple times.

The RW function takes the lock for Write.

This look harmless and it works most of the time until suddenly it doesn’t and you made a deadlock as we’ll see in a moment.

This example is simplified from a real world example which was caused by a bit of careless refactoring.

The next step is the Read Write lock being taken in go routine 2.

If you read the docs you’ll see a Read Write lock prevents any new Read locks happening so that the lock eventually becomes available.

There is also a note about recursive locking not being allowed if you read carefully which is exactly what this example is about.

The next step is the Read Write lock being taken in go routine 2.

If you read the docs you’ll see a Read Write lock prevents any new Read locks happening so that the lock eventually becomes available.

There is also a note about recursive locking not being allowed if you read carefully which is exactly what this example is about.

The deadlock happens when go routine 1 attempts to take the read lock again. This is called recursive locking and none of the standard library locking routines support recursive locking.

This lock is blocked because the Read Write lock in go routine 2 is in the process of being taken.

This means that MoreRO function will never return and the RO function will never release its lock, so in its turn the RW function will never acquire the mutex.

Thus, a deadlock!

And the moral of this story is Don’t take a read lock twice in the same go routine.

It might sound completely obvious but it is really easy to do by accident.

It does warn explicitly against doing this in the docs too, but of course I didn’t notice that until after I caused this deadlock!

I could show a commit message for every one of the examples in this talk, but since we haven’t got all day I’ll just put this one up so you can feel my pain.

I always find debugging deadlocks hard even with all these nice tools, so when you have to debug one yourself, just remember that!

Objects which contains locks that call each others methods are a deadlock waiting to happen.

This happens naturally in more complicated systems, for example file systems, and great care needs to be taken not to cause deadlocks.

Going back to our Directory and File example, the problem is caused by the fact that Directory and File both call each other’s methods.

The general solution to this is called Total Lock Ordering but we’re going to look at some easier mitigations first.

Here is one example fix for the deadlock. I call this technique shortening the locking Window.

What we’ve done here is released the mutex in the File Path method as soon as we can and in doing that the lock and unlock of the File and Directory mutexes no longer overlap.

At the top you can see the original function.

Below that you can see we’ve modified it to make a temporary copy of name and parent from the File then unlocked the mutex.

This is enough to fix the deadlock as shown before. It does introduce the idea that the Path function might return an out of date Path if the name or parent changes during the call of Path. Some thought is required here as to whether this is OK or not.

In order to minimize the chance of deadlocks, release locks as soon as possible to shorten the locking window.

Specifically release them before calling any other functions or methods which might take locks.

I’ve even resorted to using the defer statement to defer calls to get them outside the locking of the current method.

Another easy mitigation is to notice which members of your object are read only. These should be assigned to once in the initialization of the object and never written again.

These don’t need any kind of locking at all to access. Concurrent reads are always safe.

So, for example, here we say that the name of files and their parent directories never change.

I’ve moved them above the mutex definition to give the idea that they aren’t and don’t need to be protected by the mutex.

In this rename free world, we don’t need any locking in the Path method and we fixed the deadlock by not having any locking in the File.Path method at all.

If you can identify variables which are read only, you don’t need locking for them and they can be accessed outside the lock.

No locking means no possibility of deadlocks.

Here is a sketch of some other things you can do which I didn’t have room to make examples for.

Once way to reduce deadlock possibilities is to split the lock into multiple locks, so use more fine grained locking. This makes it less likely the locks will overlap.

Or maybe you can use atomics for some accesses (perhaps you are incrementing a counter). Atomics never block your process so can’t cause a deadlock. They come with their own caveats so use with care.

Using a Read write mutex is a great help for reducing lock contention provided you’ve got methods which only read things. Allowing many read only methods to run at once stops them blocking and minimizes the chance of deadlock.

And if you are using channels, buffered channels help avoid deadlocks because they don’t synchronize two go routines. Deadlocks can still occur when the channels get full though so it wont cure everything.

Now its time to bring out the big guns.

The ultimate solution to this kind of locking problem is to impose total lock ordering. This means that you are sure that if the locks overlap then you always take one lock first.

So if you have two locks A and B, you always take lock A before taking lock B in the same go routine. This makes sure we never have a deadlock.

This, however, is much easier said than done in a structured program with locks inside objects.

In practice with object based locks and methods, this means making one object super-ordinate and one subordinate.

For example this might mean that Directory can call File methods with the Directory lock held but File methods mustn’t call Directory methods with the File lock held.

In a perfect world we’d be able to specify what we want to a linter. I don’t think such a linter exists yet – if it does do tell! In the mean time comments like this one from rclone are what we have to live with.

It explains to the reader that the Cache and Item objects are tightly linked and that if you don’t want to cause a deadlock then don’t call Cache methods from Item with the Item lock held.

This means that when you change code you need to be very careful to uphold those invariants. If you don’t, deadlocks will ensue.

So make write of comments and do lots of code review when writing this sort of code.

Debugging deadlocks can get very hard, very quickly and I used to think I was pretty good at this computer science stuff, but debugging deadlocks frequently makes me feel very stupid.

So if the locking gets too complicated then you can bail out and use a single threaded engine to remove all the locking.

Single threaded likely means lower performance but maybe having no deadlocks is worth it.

You’d communicate with the engine over channels in a request / response style. Typically you’d send the channel for the response over the request channel.

This makes things much simpler and is very much in the classic Go style but making everything single threaded will drop your performance which may be unacceptable.

You’ll notice there’s been a lot of struggle with deadlocks in the previous slides and it would be nice if there were tools to help us find the deadlocks before they happen.

Well there are a couple of notable tools for finding deadlocks in go code.

Dingo hunter is based of a theoretical approach for finding channel based deadlocks. You may have caught the talk about it here in 2016. It only works for unbuffered channels, alas, which limits its scope.

Go deadlock is a practical tool for finding violations of total lock ordering and has been used to find deadlocks in Cockroach DB for example. I’ve used it on the rclone code base a few times.

It comes up with error messages that look rather like those from the race detector, but unlike the race detector it detects potential deadlocks before they happen so you can run this on your code which hasn’t deadlocked yet.

Go deadlock only works for mutex deadlocks, not channel deadlocks. As I said before this is probably 90% of the deadlocks I see so it is an excellent start.

Go deadlock is very easy to use. Here’s the example from its GitHub page.

You just replace your sync imports with a go deadlock import and when you run your code it will give total lock ordering errors.

This is what our File and Directory example looks like when run with go deadlock.

As you can see it nicely points out the potential deadlock. The top half is the first go routine and it’s saying the Directory lock then the File lock was taken but in the other go routine (the bottom half) the File lock was taken first, then the Directory lock. It isn’t saying a deadlock happened, it is just saying that because of this ordering of locks a deadlock could happen.

I’ve used go-deadlock on more complicated examples and I have to say I’ve found the output quite difficult to interpret. This is because deadlocks are hard to understand rather than a shortcoming of the tool. So if you do plug go deadlock into your massive concurrent program, expect a bit of head scratching while you work out what the error messages mean!

When I want to use go deadlock with rclone I run this little script.

It searches and replaces all the sync Mutex and sync RWMutex with their deadlock detector versions.

Unfortunately I find you can’t leave Go deadlock in production code as it has too much of a performance impact.

I’d like to have a neater way of enabling it for testing, but I haven’t figured that out yet.

Conclusion

This has been quite an intense talk and we’ve gone over a lot of concurrency related things, but I hope you all now have some idea of what a deadlock is, how do debug deadlocks and how to avoid them in the future.

Deadlocks are the curse of concurrent programming, however there good tools for finding and debugging them and there are some promising tools for helping to prevent them happening in any code base.

So if, or rather when, you get to debug your first deadlock, just remember, Don’t Panic, or rather...

...Do panic and collect a backtrace :-)

Thank you all very much for listening.

Rclone and Go

nick@craig-wood.com (Nick Craig-Wood) — Wed, 21 Nov 2018 00:00:00 +0000

This is a transcript of a talk I gave at the Go London User Group on 2018-11-21. You can watch it on YouTube (26 minutes) or see the slides and read the words here.

It's about rclone and how rclone uses the Go programming language.

Introduction

Good evening.

I’m Nick Craig-Wood and I’m here to talk about rclone. Rclone is an Open Source command line program to transfer files to and from cloud storage started by me as a hobby in 2012.

First a bit about me. I’ve loved programming ever since I was 13 with my ZX80 and have been doing it ever since.

Now-a-days I’m CTO of Memset Ltd who do cloud hosting and storage. I don’t get the chance to do a lot of coding at work any more so I now do Open Source coding as a hobby.

I’ve always had a keen interest in storing data, keeping it safe and transferring it about – hence rclone which is an embodiment of those things!

I used to be a data hoarder, but the house filled up with servers and disks and my wife complained about the electricity bill so I’m mostly reformed now.

Thank heavens for cloud storage!

My talk this evening is going to start with a quick summary of rclone and its history.

I’ll dive into how it works, and show a bit of code.

I’ll then discuss how you test a project like rclone and round off with a look at some of my favourite libraries.

But before I get going I don’t want you to get the impression rclone is all my work. I’d like to thank my excellent core developers, the super helpful group of people on the forum and anyone who has sent a PR or raised an issue – rclone wouldn’t be what it is today without you.. Oh and thank you for everyone who has sent a PR correcting my spelling :-)

What is rclone?

In summary rclone is a command line program to sync your data to and from cloud providers.

Rclone takes lots of care with data integrity, checking hashes at every step and preserves timestamps on your files.

It can transfer data directly between cloud providers so you can use it to back up your s3 objects to google cloud storage.

It has some optional features like encryption, but by and large rclone maps objects 1:1 from source to destination.

There is lots more info on the website.

Many of you will be familiar with rsync. Which is (see slide)

Rclone syncing works very much like rsync. I’ve tried to keep the flags the same where possible to make it familiar to rsync users.

There is one thing rclone does not do though and that is implement the extremely clever delta encoding that rsync uses for transferring objects.

That unfortunately isn’t possible when trying to copy objects 1:1 to cloud providers as they just don’t provide the API to do it.

It would be possible to implement it but you’d lose that 1:1 mapping of objects.

Rclone supports most major cloud providers. Well I think so, they keep making new ones!

Certainly all the big names are here, both for business use like s3, google cloud storage, azure blob storage and for personal like google drive and dropbox.

There are also the standard protocols like FTP, SFTP, HTTP.

I particularly like the HTTP backend – you point it file listing webpage (for example the ones apache generates) and it can pull the files off there recursively, or you can mount it as a disk.

All rclone operations are supported by all providers.

Thanks to Go’s amazing cross compiling abilities, rclone runs on nearly every platform you can think of.

Rclone is licenced under a very liberal MIT licence which means that it gets bundled into Synology NAS and other products like that.

Nearly all the platforms are cross compiled on a linux host as part of the Travis build. All except Windows and macOS which I had to build natively to fix various problems.

As I began to do less programming at work, I realised I needed to do some programming at home in order to retain my sanity.

In 2012 I decided to learn Go as it had just reached the exciting milestone of a 1.0 release.

I made a few test programs then I wrote my first major go library, an interface for the openstack swift object storage system.

I need a program to exercise that “swiftsync” which became “rclone”.

There were some things that attracted me immediately to Go.

We’d been using Python a lot at work so the single binary for a deployment was just breath of fresh air.

Go excels in concurrency which is really important for rclone too. Everything works concurrently in rclone such as directory listings and transfers of objects.

I started using Go as a new challenge and it has really brought back the joy of programming for me personally.

Lastly Go is a really easy language to pick up, especially if you’ve done any programming in a language similar to C such as Java.

From its humble beginnings rclone has grown into quite a large project.

As of today we’ve got about 11,000 stars on github, 200 contributors who’ve made 500 pull requests. Rclone get’s downloaded quarter of a million times a month and it has been packaged by the major Linux, Windows and Mac distributions.

Exactly how rclone got all those users is a mystery, but being on the front page of Hacker news added about 1,000 github stars overnight. Unfortunately at that time I was in the Outer Hebrides with no Internet so I had no idea this was happening!

Sorry, your browser doesn't support embedded videos.

Here is a visualisation of rclone’s source code made with Gource.

Rclone was created in 2012 and it got off to a slow start with just me – that is me whizzing about.

Rclone reached v1.00 in 2014 and supported 3 backends: s3, drive and swift.

Rclone gained its first external contribution in 2015. That was when it implemented Amazon Drive and it started to get popular.

In 2016 it gained some traditional protocols, SFTP and FTP and by 2017 I’d finally figured out I needed a VFS layer. Which brings us to the frenzy of 2018!

In 2017 I was surprised to find rclone in the middle of a minor media storm.

This is how...In 2015 Amazon launched their Unlimited Drive Storage product.

This, for a certain kind of person that you can find on the data hoarders subreddit was a personal challenge as you can imagine.

The screenshot on the left was sent to me from that reddit and all the data (1 Peta byte of it) was uploaded with rclone.

Rclone was banned from Amazon Cloud Drive, probably as a consequence of this, though ostensibly it was the use of encrypted keys within the binary. Amazon also shut down the unlimited Drive product in the US

I was surprised to have an interview with a journalist from The Register.

Rclone is still banned from Amazon Cloud Drive unless you have your own keys.

How does it work?

I’ll now move on from the history to how it works.

There are a number different functionalities of rclone. Each of these is implemented as a sub command otherwise known as a verb of the main program.

You write the verbs after the command so “rclone config” or “rclone sync”. More parameters can come after the verb.

For data transfer the main 3 verbs are sync, copy and move.

The first command you run after installing rclone is “rclone config”.

This starts up the old school config wizard.

I think I first saw this style of config wizard in perl when configuring CPAN.

It isn’t the prettiest in the world but it gets the job done!

When it needs to do oauth it opens up your browser for you and redirects back to a local webserver to receive the results.

The next thing you are likely to want to do is copy some files.

Here is a quick demo of some files being copied using “rclone copy” and “rclone ls” to list the files.

Always start with “rclone copy” - don’t use “rclone sync” as that can delete files in the destination.

This shows the use of rclone sync.

Rclone sync differs from copy in that it makes the source and destination identical.

This means that it copies files to the destination and deletes any files in the destination which should be there.

Always try with –dry-run first!

Here is rclone copy in diagram format.

On the left are the source and destination before the command and on the right after the command.

You can see that it can copy a file, ignore an unchanged file, overwrite a stale file and that it doesn’t touch files which exist in the destination already but aren’t in the source.

Whereas rclone sync’s job is to get the source and destination identical

You can see that it can copy a file, ignore an unchanged file, overwrite a stale file and that it deletes files which exist in the destination already but aren’t in the source.

Rclone sync, copy and move are all implemented by the same routine as most of the code is identical between them.

Rclone mount is one of my favourite features. Mount any backend supported by rclone as a FUSE mount.

This works on linux, mac and Windows.

Rclone supports seeking on the files too so you can play video quite well from your cloud storage system and I know that lots of rclone users do exactly this.

However if you want to open a file for both read and write or seek while writing then rclone will need to download the file and work on a cached copy. This is part of the VFS layer in rclone which adapts rclone objects to work more like an actual file system.

Here is another of my favourite features.

It is a blatant rip-off or homage as I prefer to call it of the ncdu tool. If you’ve never used it I recommend you install it right away

It is the best tool for answering the question “Where is all my disk space?”. It lets you explore your local disk using a really easy to use curses interface.

I had the idea of implementing that in rclone and I needed something to do on a plane journey so hence rclone ncdu was born.

It uses the excellent termbox library to implement the text based user interface.

Dive into code

Now let’s dive into a bit of code.

Rclone implements its backends using interfaces which won’t be a suprise. This shows the interface which all rclone backends must fulfil.

Rob Pike tells us: “The bigger the interface, the weaker the abstraction” - this is a big interface :-(

It is broken down into 3 sub parts in the rclone source but I’ve presented the whole interface here as it all as to be implemented for a backend.

You can see the kind of methods you might expect, List, NewObject, Put and Mkdir.

There are also some missing from that list which I’ll cover in a moment, like Copy.

This is the interface a cloud storage object must fulfil.

Again pretty straight forward. Note the Hash method which you call with one of the types the Hashes method in the Fs object returns.

Open reads the object and Update sets its contents.

You can also read the modification time and set the modification time – something that is very important for rclone syncing.

All the backends support the interfaces in the previous two slides. However some backends can do more, and rclone can use that.

For example the Purge interface deletes a directory and everything under it. Some backends can do that and some can’t.

For example, say you were doing a server side move of an object. Some backends can do that immediately with the Move method (for example Google Drive). Some support Copy but not Move (for example S3). Rclone will Copy the object then Delete it. And some don’t support either (eg B2) in which case rclone will download the object and upload it again.

Here is how you might use those optional interfaces – with an type assertion. That is straight forward go code.

However, rclone backends can wrap other backends, for example the encryption backend. The outer backend wants to expose what the inner backend can do so must implement all optional methods. The inner backend may not implement all optional methods so we’ve got a problem.

The traditional solution to this is a sentinel error which we check. Here we call Purge and check the err value against ErrCantPurge.

This works very well, provided you don’t want to know which methods are used in advance. Unfortunately rclone does need to know….

The solution to that is a good old fashioned struct full of function pointers. This is filled in with a bit of reflection so doesn’t take any manual work to keep up to date.

Backends which wrap other backends logical AND their own features with the features of the wrapped backend to make the final Features struct.

To use Purge, say, you just check to see if the pointer is nil. If it isn’t nil that means the method is supported so you can call it.

Those of you who’ve used C++ will realise this is a re-implementation of the vtable or virtual table. Go made the implementation straight forward though.

Testing

Lets go into how rclone is tested.

As you saw earlier rclone supports lots of providers, it has lots of commands, architectures OSes. That all makes it really hard to test.

Anyone think writing mocks for 27 different cloud providers would be much fun for a hobby project? No?

So rclone does lots of integration tests. These give real confidence that rclone works properly and keeps on working. After all cloud providers often change stuff.

Rclone has a full continuous integration pipeline.

I think we are amazingly lucky as open source programmers to be able to use Github, Travis and Appveyor for free – they are fantastic tools.

The CI tests run all the non integration tests and builds binaries for all the platforms, uploading them for beta release automatically for every branch pushed to the main repository.

You’ll find those at beta.rclone.org

The integration tests run on a separate server.

These run the tests, which are just normal go unit tests with some extra flags against all the backends.

Running these tests costs money and takes a long time so I don’t run them against every pull request, only against the master branch and only once per day.

The integration tests produce a report for the developers to look at.

This was a good day with only one failure out of 105 tests.

Often there are lots of failures, usually because of some cloud provider glitch, but sometimes because of a change in the rclone code.

Making a pretty report (well that is pretty by my standards) wasn’t high on my priority list. However I completely messed up one of the releases by misreading the old report, so I made a report which couldn’t be misunderstood.

The integration tests rclone uses are standard go tests. It is easy to add flags to the go tests and rclone adds a “-remote” flag to tell it to run the tests against an actual backend.

Unlike unit tests, integration tests often fail. The internet is a messy place – things go wrong. The test_all framework re-runs the tests that fails to try to get them to pass.

I should probably take the time to open source this framework as it could be useful elsewhere!

In go1.6 the concept of nested tests was introduced.

This made rclone’s life much easier. As you can see it is only a few lines of code to run the integration test suite for a backend. Some backends run the suite more than once with different parameters.

When making a new backend, you can use the integration tests as a guide for what you need to work on next and it makes the process quite enjoyable.

Pre go1.6 code generation was used to make the integration tests which was rather painful so thank you to the Go team for making nested tests.

This is something that not many people know, but can make your life much easier.

You can add flags to tests. You don’t need to do anything special, just define flags with the standard library flag package.

There are some great examples of this in the standard library, for instance using a “-golden” flag to make the tests files.

Rclone uses flags to indicate that tests should run with a real backend, and to help debug those tests.

Rclone is a reasonably large program now, approaching 100 thousand lines of Go code.

However it is built on a fantastic set of libraries some of which I’ll talk about in a moment.

And all of that in turn is built on the go standard library.

Rclone uses the vendor directory to keep that all straight, and we recently switched to “go mod” to manage that.

I don’t think we can drop the vendor directory just yet as rclone needs to support old go versions for the Linux distros that package rclone. Maybe in a year or two.

Tools

I’ll just run through a few of my favourite libraries and tools that rclone uses.

There are loads of other libraries I could have mentioned here so apologies to any authors I’ve missed out.

My first choice is goimports. Install it in your editor and never type an import statement again.

You can see me editing a small file in emacs here and the import statements being automatically added while the code is formatted for you. It works in vim too so I’m told ;-)

Cobra is excellent for organising command line programs with lots of verbs.

I’m sure many of you have used kubectl to control kubernetes or used docker – that is cobra in action.

Cobra comes with a flag library pflag which allows you to create POSIX flags. Flags made by the go flag library are somewhat old school in comparison.

Cobra also creates your bash completion script for all those people like me who use bash, but hate writing bash scripts!

Cobra helps with the documentation. When you write a command cobra makes command help for it and then has a really easy way of turning that into markdown for the website.

A lot of the help on the rclone website was generated this way and it really helps keeping everything consistent.

I then use pandoc to turn the markdown into a man page and one big HTML document for the download.

If you’ve been keeping up with go news you’ll have read the errors proposal for go 2.

The pkg/errors package was one of the motivations for the proposal, so I imagine at some time I’ll be retiring it when go 2 comes along.

You use it to annotate your errors as they bubble up through your code.

This means that rather than receiving a log message with “unexpected EOF” you receive a helpful message which gives you some idea as to what really went wrong.

Conclusion

I’m just going to talk briefly about what to do if your open source package takes off.

It is easy to get daunted by 100s of issues most of which are support requests so the first thing to do is open a forum and get the users helping themselves. I use Discourse – it is free and easy to setup and maintain.

Don’t be shy of asking users who make issues for help. You might need to give them a bit of a helping hand, but that is a positive process for both parties.

When you’ve had a few good pull requests from someone, recruit them!

Oh yes, write lots of docs – maybe that should have gone first.

Finally get octobox.io – rather than managing 100s of emails from Github manage them in a nice web interface where you can filter etc.

Thank you all for listening :-)

Snake Puzzle Solver

nick@craig-wood.com (Nick Craig-Wood) — Sun, 25 Sep 2016 00:00:00 +0000

My family know I like puzzles so they gave me this one recently:

Fork me on Github

The code for this along with a Jupyter notebook for it is on github.

When you take it out the box it looks like this:

And very soon after it looked like this (which explains why I've christened the puzzle "the snake puzzle"):

The way it works is that there is a piece of elastic running through each block. On the majority of the blocks the elastic runs straight through, but on some of the it goes through a 90 degree bend. The puzzle is trying to make it back into a cube.

After playing with it a while, I realised that it really is quite hard so I decided to write a program to solve it.

The first thing to do is find a representation for the puzzle. Here is the one I chose:

# definition - number of straight bits, before 90 degree bend
snake = [3,2,2,2,1,1,1,2,2,1,1,2,1,2,1,1,2]
assert sum(snake) == 27

If you look at the picture of it above where it is flattened you can see where the numbers came from. Start from the right hand side.

That also gives us a way of calculating how many combinations there are. At each 90 degree joint, there are 4 possible rotations (ignoring the rotations of the 180 degree blocks) so there are:

>>> 4**len(snake)
17179869184

17 billion combinations. That will include some rotations and reflections, but either way it is a big number.

However it is very easy to know when you've gone wrong with this kind of puzzle - as soon as you place a piece outside of the boundary of the 3x3x3 block you know it is wrong and should try something different.

So how to represent the solution? The way I've chosen is to represent it as a 5x5x5 cube. This is larger than it needs to be but if we fill in the edges then we don't need to do any complicated comparisons to see if a piece is out of bounds. This is a simple trick but it saves a lot of code.

I've also chosen to represent the 3d structure not as a 3d array but as a 1D array (or list in python speak) of length 5 x 5 x 5 = 125.

To move in the x direction you add 1, to move in the y direction you add 5 and to move in the z direction you move 25. This simplifies the logic of the solver considerably - we don't need to deal with vectors.

The basic definitions of the cube look like this:

N = 5
xstride=1 # number of pieces to move in the x direction
ystride=N # number of pieces to move in the y direction
zstride=N*N # number of pieces to move in the z direction

In our list we will represent empty space with 0 and space which can't be used with -1:

empty = 0

Now define the empty cube with the boundary round the edges:

# Define cube as 5 x 5 x 5 with filled in edges but empty middle for
# easy edge detection
top = [-1]*N*N
middle = [-1]*5 + [-1,0,0,0,-1]*3 + [-1]*5
cube = top + middle*3 + top

We're going to want a function to turn x, y, z co-ordinates into an index in the cube list:

def pos(x, y, z):
 """Convert x,y,z into position in cube list"""
 return x+y*ystride+z*zstride

So let's see what that cube looks like:

def print_cube(cube, margin=1):
 """Print the cube"""
 for z in range(margin,N-margin):
 for y in range(margin,N-margin):
 for x in range(margin,N-margin):
 v = cube[pos(x,y,z)]
 if v == 0:
 s = " . "
 else:
 s = "%02d " % v
 print(s, sep="", end="")
 print()
 print()

>>> print_cube(cube, margin = 0)
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1

-1 -1 -1 -1 -1
-1 . . . -1
-1 . . . -1
-1 . . . -1
-1 -1 -1 -1 -1

-1 -1 -1 -1 -1
-1 . . . -1
-1 . . . -1
-1 . . . -1
-1 -1 -1 -1 -1

-1 -1 -1 -1 -1
-1 . . . -1
-1 . . . -1
-1 . . . -1
-1 -1 -1 -1 -1

-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1
-1 -1 -1 -1 -1

Normally we'll print it without the margin.

Now let's work out how to place a segment.

Assuming that the last piece was placed at position we want to place a segment of length in direction. Note the assert to check we aren't placing stuff on top of previous things, or out of the edges:

def place(cube, position, direction, length, piece_number):
 """Place a segment in the cube"""
 for _ in range(length):
 position += direction
 assert cube[position] == empty
 cube[position] = piece_number
 piece_number += 1
 return position

Let's just try placing some segments and see what happens:

>>> cube2 = cube[:] # copy the cube
>>> place(cube2, pos(0,1,1), xstride, 3, 1)
>>> print_cube(cube2)
01 02 03
 . . .
 . . .

 . . .
 . . .
 . . .

 . . .
 . . .
 . . .

>>> place(cube2, pos(3,1,1), ystride, 2, 4)
>>> print_cube(cube2)
01 02 03
 . . 04
 . . 05

 . . .
 . . .
 . . .

 . . .
 . . .
 . . .

>>> place(cube2, pos(3,3,1), zstride, 2, 6)
>>> print_cube(cube2)
01 02 03
 . . 04
 . . 05

 . . .
 . . .
 . . 06

 . . .
 . . .
 . . 07

The next thing we'll need is to undo a place. You'll see why in a moment.

def unplace(cube, position, direction, length):
 """Remove a segment from the cube"""
 for _ in range(length):
 position += direction
 cube[position] = empty

>>> unplace(cube2, pos(3,3,1), zstride, 2)
>>> print_cube(cube2)
01 02 03
 . . 04
 . . 05

 . . .
 . . .
 . . .

 . . .
 . . .
 . . .

Now let's write a function which returns whether a move is valid given a current position and a direction and a length of the segment we are trying to place.

def is_valid(cube, position, direction, length):
 """Returns True if a move is valid"""
 for _ in range(length):
 position += direction
 if cube[position] != empty:
 return False
 return True

>>> is_valid(cube2, pos(3,3,1), zstride, 2)
True

>>> is_valid(cube2, pos(3,3,1), zstride, 3)
False

Given is_valid it is now straight forward to work out what moves are possible at a given time, given a cube with a position, a direction and a length we are trying to place.

# directions next piece could go in
directions = [xstride, -xstride, ystride, -ystride, zstride, -zstride]

def moves(cube, position, direction, length):
 """Returns the valid moves for the current position"""
 valid_moves = []
 for new_direction in directions:
 # Can't carry on in same direction, or the reverse of the same direction
 if new_direction == direction or new_direction == -direction:
 continue
 if is_valid(cube, position, new_direction, length):
 valid_moves.append(new_direction)
 return valid_moves

>>> moves(cube2, pos(3,3,1), ystride, 2)
[-1, 25]

So that is telling us that you can insert a segment of length 2 using a direction of -xstride or zstride. If you look at previous print_cube() output you'll see those are the only possible moves.

Now we have all the bits to build a recursive solver.

def solve(cube, position, direction, snake, piece_number):
 """Recursive cube solver"""
 if len(snake) == 0:
 print("Solution")
 print_cube(cube)
 return
 length, snake = snake[0], snake[1:]
 valid_moves = moves(cube, position, direction, length)
 for new_direction in valid_moves:
 new_position = place(cube, position, new_direction, length, piece_number)
 solve(cube, new_position, new_direction, snake, piece_number+length)
 unplace(cube, position, new_direction, length)

This works by being passed in the snake of moves left. If there are no moves left then it must be solved, so we print the solution. Otherwise it takes the head off the snake with length, snake = snake[0], snake[1:] and makes the list of valid moves of that length.

Then we place each move, and try to solve that cube using a recursive call to solve. We unplace the move so we can try again.

This very quickly runs through all the possible solutions:

# Start just off the side
position = pos(0,1,1)
direction = xstride
length = snake[0]
# Place the first segment along one edge - that is the only possible place it can go
position = place(cube, position, direction, length, 1)
# Now solve!
solve(cube, position, direction, snake[1:], length+1)

Wow! It came up with 2 solutions! However they are the same solution just rotated and reflected.

But how do you use the solution? Starting from the correct end of the snake, place each piece into its corresponding number. Take the first layer of the solution as being the bottom (or top - whatever is easiest), the next layer is the middle and the one after the top.

After a bit of fiddling around you'll get...

I hope you enjoyed that introduction to puzzle solving with computer.

If you want to try one yourselves, use the same technique to solve solitaire.

Download the entire code snake_puzzle_solver.py or the iPython notebook.

IOCCC 2012 Mersenne Prime Checker

nick@craig-wood.com (Nick Craig-Wood) — Wed, 03 Oct 2012 00:00:00 +0000

This was my entry for the 20^th International Obfuscated C Code Contest, a contest whose aims are

To write the most Obscure/Obfuscated C program under the rules.
To show the importance of programming style, in an ironic way.
To stress C compilers with unusual code.
To illustrate some of the subtleties of the C language.
To provide a safe forum for poor C code. :-)

I've been meaning to enter the contest for quite a few years so was very pleased to see it re-opened for 2012.

Alas my entry didn't win, but maybe you'll enjoy the entry and explanation.

Here is my entry:

#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<inttypes.h>
#define H uint32_t
#define U uint64_t
#define N void
#define T int
#define P calloc(n,sizeof(U))
#define R return
#define I for(
#define M if(
#define E >>32
#define S i=0;i<n;i++)
 U K=-(U)(H)-1,*B,*C
 ,*x,AC,h;T L,n,F,l,p,D,*_;U e(U x,U
 y){x=K-x;U r=y-x;M y<x)r+=K;R r;}U AB(U x,H
 w,U*c){U s=x+*c;H t=s<x;*c=(s>>w)+(t<<(64-w));R
 s&((((U)1)<<w)-1 ) ;} U Q(U x,U y ) {U r
 =x-y;M x<y)r+=K;R r;} U AD(U b,U
 a){H d=b E,c=b;M a >= K)a-= K;a=
 Q(a,c);a=Q(a,d) ;a=e (a
 ,((U)c)<<32);R a;}U A( U
 x,U y){H a=x,b= x E,c=y,d
 =y E;x=b*(U)c;y=a*(U)d;U e =a*(
 U)c,f=b*(U)d,t;x+=y;M x<y)f +=1LL
 <<32;t=x<<32;e+=t;M e<t)f+= 1;t=
 x E;f+=t;R AD(f,e);}U J(U a,U b
 ){U r= 1;T i;I i=63;i>=0; i--){r=A(r,r);M(b >>i)&1)r=A(r
 ,a);}R r;}U f(U a){R J(a,K -2);}N G (H n,U*x, U*y)
 {T i;I S x[i]=A(x[i],y[i ]);}N g( H n,U*x) {T i;I S x[i]
=A(x[i],x[i]);}N AA(){U d = AC; T k ;I k=
F;k>=1;k--){T m=1<<k,c=m>> 1,j, r
;U w=1;I j=0;j<c;j++){I r= 0;r<n ;r
+=m){T a=r+j,b=a+c;U u=x[a] ,v=x[b] ;x[a
]=e(u,v);x[b]=A(Q(u,v),w);}w= A( w,d );}d=A
(d,d);}}N W(){T k;I k=1;k<=F;k ++ ){T m=1<< k, c=m>>
1,j,r;U z=(U)(1<<((H)F-(H)k)), d= J(h,z),w=1;I j=0
;j<c;j++){I r=0;r<n;r+=m){T a =r+j, b=a+ c;U u=x[a], v=A (w
,x[b]);x[a]=e(u,v);x[b]=Q(u,v);}w=A(w ,d);}}}T O(){T o=0,i, w=L
/n;U s=J(7,(K-1)/192/n*5);AC=J(s,192) ;h=f (AC );M 2 *w +F
>=61)R 0;l=w;D=1<<w;p=w+1;x=P;B=P;C=P; _= P; *B=1;*C=f( n);I
i=0;i<=n;i++){U t=(U)L*(U)i,r=t%n,b=t/n ;H a;M r
E)R 0;M(H)r)b++;M b E)R 0;a=b;M i>0){U c=_[ i-1]
=a-o;M c!=w&&c!=w+1)R 0;M i<n){r=n-r;B[i]=J(s,r);C[i]= f(A(B
[i],n));}}o=a;}R 1;}U V(){T i=0,j=0;U r=0;I;i<64&&j<n;i+=_
[j],j++)r|=x[j]<<i;M r)R r;I r=j=0;j<n;j++)r|=x[j];R r;}
N X(H c,T i){while(c){I;i<n;i++){H y=1<<_[i];x[i]+=c;
M x[i]>=y)x[ i]-=y,c=1; else
R;}i=0;} }N Y(H c){T i;
while(c) {I S{H y=1<< _[i] ;x[i ]-=
c;M x[i]>=y)x[i]+=y,c=1 ;else R ;}}}N
Z(U c){T i;while(c){I S{x[ i]=AB(x[i],_[i],
&c);H t=c;M!(c E)&&t <D){M t ) X(t,i+1);R;}}}}N q()
{T i;U c=0;G(n,x, B);AA(F,x);g(n,x);W(F ,x);G(n,x
,C);I S x[i]= AB(x[i],_[i],&c);M c)Z(c);Y (2);}T main(
T w,char** v){T i,k,j;M w<2) {puts("Usage: p [n]");R 1
;}L=atoi (v[1] );j=w>2 ?atoi(v[2]):L-
2;do F++ ,n=1<<F;while(!O ());I k=0;k
<1;k++){*x=4;I i=0;i<j;i++)q();}printf("0x%016"PRIX64"\n",V());R 0;}

I'm going to explain exactly what it does and how, but first here are the hints and remarks I provided with the code. Some of the hints assume you've pre-processed the code into something vaguely legible first.

Hints

You stare at the source code... Is that a picture of a monk? What does that strange mathematical writing say? You get a subliminal message... Hunt for primes? But how?

You run the program. It comes up with a strange usage message. What could p be? You try running the program with a prime number as an argument.

./prog 23209

It returns after a few seconds with the rather cryptic message

0x0000000000000000

What could that mean? Maybe that 2²³²⁰⁹-1 (a 6,987 digit number) is prime after all?

OK If you haven't guessed by now, this program proves the primality of Mersenne primes of the form 2^p-1. The biggest known primes are of this form because there is an efficient test for primality - the Lucas Lehmer test.

If you run the program and it produces 0x0000000000000000 (0 in hex) as an answer then you've found a prime of the form 2^p-1. If it produces anything else then 2^p-1 isn't prime. The printed result is the bottom 64 bits of the last iteration of the Lucas Lehmer test.

Have a look at the source code and see if you can work out how it implements the Lucas Lehmer test...

Your first clue is to work out what the magic number -(U)(H)-1 is. (My favourite bit of obfuscation!)

Got it? It is a prime just less than 2⁶⁴, 0xFFFFFFFF00000001 = 2⁶⁴ - 2³² + 1 (p from now on), with some rather special properties.

The most important of which is that arithmetic modulo p can all be done without using divisions or modulo operations (which are really slow). See if you can spot the add, subtract and multiplication routines. There are also reciprocal and to the power operations in there somewhere.

The next important property is that p-1 has the following factors

2^32 * 3 * 5 * 17 * 257 * 65537

All those factors of 2 suggest that we can do a Fast Fourier Transform over the Galois Field GF(p) which is arithmetic modulo p. See if you can spot the FFT code!

To make a truly efficient Mersenne primality prover it is necessary to implement the IBDWT a variant of the FFT using an irrational base. This halves the size of the FFT array by combining the modulo and multiply steps from the Lucas Lehmer test into one. You might wonder how you can use an irrational base in a discrete field where every element is an integer, but as it happens the p chosen has n-th roots of 2 where n is up to 26, which means that an IBDWT can be defined for FFT sizes up to 2²⁶.

You'll find the code to generate the roots of 1 and 2 necessary for the IBDWT if you search for 7 in the code. 7 is a primitive root of GF(p) so all follows from that!

This all integer IBDWT isn't susceptible to the round off errors that plague the floating point implementation, but does suffer in the speed stakes from the fact that modern CPUs have much faster floating point units than integer units.

For example to check if 2^18000000-1 is prime (a 5,418,539 digit number) Prime95 (the current speed demon) uses a 2²⁰ FFT size and each iteration takes it about 25ms. On similar hardware it this program takes about 1.2s also using an FFT size of 2²⁰! Prime95 is written in optimised assembler to use SSE3 whereas this program was squeezed into 2k of portable C with a completely unoptimised FFT!

This program can do an FFT of up to 2²⁶ entries. Each entry can have up to 18 bits in (as 2*18+26 <= 63), which means that the biggest exponent that it can test for primality is 18*2²⁶-1 = 1,207,959,551. This would be number of 363 million decimal digits so could be used to find a 100 million digit prime and scoop the EFF $150,000 prize! It would take rather a long time though...

The second argument to the program allows the number of iterations to be put in, and using this it has been validated against the Prime95 test suite. There is a unit test for the program but I thought including it was probably against the spirit of the contest!

Speed of the FFT could be improved greatly, but unfortunately there wasn't space in the margin to do so ;-)

Remarks about the Code

This should compile on any compiler which supports inttypes.h from the C99 standard (eg gcc).

It runs a lot quicker when compiled as a 64 bit binary!

I have successfully compiled and run this program without changes or compiler warnings for:

64 bit Ubuntu Linux 11.10

32 bit Debian Linux (Lenny)

Windows using MINGW

OS X 10.7 using Apple's gcc based compiler

If you don't have a compiler which supports inttypes.h from C99 (hello it is 2012 Microsoft!) then you'll need to go platform specific:

comment out the #include <inttypes.h> line

change the #define for uint32_t to be an unsigned 32 bit type unsigned int or unsigned __int32

change the #define for uint64_t to be an unsigned 64 bit type unsigned long long or unsigned __int64

add #define PRIX64 "llX" or "lX" or "I64X"

How it was written

I wrote most of this code some time ago as a view to publishing a full description of how my ARMprime code works in easy to understand C rather than a twisty turny mess of ARM assembler macros.

However I never got round to finishing it, so I used the IOCCC as a spur to get me to finish it properly!

I've open-sourced the code and the build chain at github.

As you can imagine it isn't possible to develop the code in the form above from scratch! I created an unobfuscated version mersenne.c first and then wrote two programs, one which compresses the code (it uses a manual version of LZ77 compression with #define statements) and another which makes it into the pretty picture.

This means that I can change the code and just type make to create the final pretty code for the entry, and to run the unit tests with.

The first challenge was cutting the code down to size. It needed to be less than 2k of non-whitespace characters and less than 4k total. I went through it getting rid of all extraneous functions, compacting definitions, removing optimisations, obfuscating and it was still far too big.

With the help of crush.py and after many hours of work I managed to get it small enough. I then wrote check.py to see whether the code was within the competition rules and unit_test.py to validate the code against Prime95's test suite.

Finally I wrote artify.py which reads the crushed source code and uses it to paint the image. It treats each character (or sometimes a group of characters so the C remains valid) as a pixel and paints the image with code!

Note that none of the python programs are particularly tidy, but that python is a really good language for doing this in - I'd hate to be doing it in C!

How it works

Now I've introduced the unobfuscated version, I can show how each part of the code works.

Firstly all good obfuscated programs have lots of global variables to shorten the code down:

u64 MOD_P = -(u64)(u32)-1, /* 0xFFFFFFFF00000001 - the magic number */
 *digit_weight, /* Array of weights for IBDWT */
 *digit_unweight, /* Inverse of the above */
 *x, /* pointer to array of n 64 bit integers - the current number */
 MOD_W, /* The root of 2 for the fft so that 2**MOD_W == 1 mod p */
 MOD_INVW; /* The inverse of the above so that MOD_W * MOD_INVW == 1 mod p */

int exponent, /* The mersenne exponent - we are testing 2^exponent -1 is prime or not */
 n, /* The number of 64 bit numbers in the FFT */
 log_n, /* n = 2^log_n */
 digit_width0, /* Digit width for the IBDWT */
 digit_width1,
 digit_width_0_max,
 *digit_widths; /* Array of number of bits in each element of the FFT */

From the top down there is the main loop which contains the Lucas Lehmer algorithm:

int main(int w, char ** v)
{
 int i,k,j;
 if (w < 2)
 {
 puts("Usage:@p@[n]");
 return 1;
 }

 exponent = atoi(v[1]);
 j = w > 2 ? atoi(v[2]) : exponent - 2; /* iterations */
 /* initialise, finding correct FFT size */
 do log_n++, n = 1 << log_n;
 while(!mersenne_initialise());

 for (k = 0; k < 1; k++)
 {
 *x = 4;
 for (i = 0; i < j; i++)
 mersenne_mul();
 }
 printf("0x%016" PRIX64 "\n", mersenne_residue());
 return 0;
}

It reads the exponent and optional number of iterations from the command line, then initialises the program, starting with a small FFT size n and increasing until it finds one which fits.

It then does the Lucas Lehmer test setting the initial value to 4, iterating it for exponent times and printing the residue.

The core of the Lucas Lehmer test (and the IBDWT) is the mersenne_mul function which squares the current number and subtracts 2, returning it modulo 2^p - 1 :

void mersenne_mul()
{
 int i;
 u64 c = 0;

 /* weight the input */
 mod_vector_mul(n, x, digit_weight);

 /* transform */
 fft_fastish(log_n, x);

 /* point multiply */
 mod_sqr_vector(n, x);

 /* untransform */
 invFft_fastish(log_n, x);

 /* unweight and normalise the output */
 mod_vector_mul(n, x, digit_unweight);

 /* carry propagation */
 for (i = 0; i < n; i++)
 x[i] = mod_adc(x[i], digit_widths[i], &c);
 if (c)
 mersenne_add64(c);
 /* subtract 2 */
 mersenne_sub32(2);
}

This the FFT, point multiply and inverse FFT perform a convolution. This does the bulk of the work of the multiplication. The weight and unweight and the start and end is the IBDWT which ensures that the result is modulo 2^p - 1. The carries are propagated and 2 is subtracted from the end.

The FFT is a completely standard FFT with bit reversed output, the only unusual thing being that all the operations are mod p rather than with complex numbers. If you compare the code for the FFT with a standard implementation you will see it is very similar:

void fft_fastish()
{
 u64 d = MOD_W;
 int k;

 for (k = log_n; k >= 1; k--)
 {
 int m = 1 << k,
 c = m >> 1,
 j,
 r;
 u64 w = 1;
 for (j = 0; j < c; j++)
 {
 for (r = 0; r < n; r += m)
 {
 int a = r + j,
 b = a + c;
 u64 u = x[a],
 v = x[b];
 x[a] = mod_add(u, v);
 x[b] = mod_mul(mod_sub(u, v), w);
 }
 w = mod_mul(w, d);
 }
 d = mod_mul(d, d);
 }
}

MOD_W is the root of unity the FFT is built on. It happens to be an integer in GF(p) such that MOD_Wⁿ == 1 mod p. You can see the FFT calculating the twiddles (w). Precalculating them would be better obviously but not in 2k of code! The inner loop shows the butterfly.

Now for the fundamental operations over GF(p). First thing to note is that in the code p = MOD_P = 2⁶⁴ - 2³² + 1. MOD_P was chosen so that it has quite a few special properties, one of them being that it is possible to do modulo p operations without doing divisions (which are really slow!).

The easiest operation to define is subtraction:

u64 mod_sub(u64 x, u64 y)
{
 u64 r = x - y;
 /* if borrow generated - hopefully the compiler will optimise this! */
 if (x < y)
 r += MOD_P; /* Add back p if borrow */
 return r;
}

The C code was designed so that the if statement should be optimised into a jump on carry flag if the compiler is doing its job properly. Addition can then be defined as subtracting a negative number:

u64 mod_add(u64 x, u64 y)
{
 x = MOD_P - x; /* do addition by negating y then subtracting */
 u64 r = y - x; /* y - (-x) */
 /* if borrow generated - hopefully the compiler will optimise this! */
 if (y < x)
 r += MOD_P; /* Add back p if borrow */
 return r;
}

To do multiplication, first it is necessary to work out how to do reduce a 128 bit number mod p. We use these facts:

2⁶⁴ == 2³² -1 mod p
2⁹⁶ == -1 mod p
2¹²⁸ == -2³² mod p
2¹⁹² == 1 mod p
2ⁿ * 2^(192-n) = 1 mod p

Thus to reduce a 128 bit number mod p (split into 32 bit chunks x3,x2,x1,x0):

x3 * 2⁹⁶ + x2 * 2⁶⁴ + x1 * 2³² + x0 mod p
= x2 * 2⁶⁴ + x1*2³² + x0-x3) [using 2⁹⁶ mod p = -1]
= (x1+x2) * 2³² + x0 - x3 - x2 [using 2⁶⁴ mod p = 2³² -1]

This is explained in more detail in my ARMprime pages. This division free code then looks like this:

u64 mod_reduce(u64 b, u64 a)
{
 u32 d = b >>32,
 c = b;
 if (a >= MOD_P) /* (x1, x0) */
 a -= MOD_P;
 a = mod_sub(a, c);
 a = mod_sub(a, d);
 a = mod_add(a, ((u64)c)<<32);
 return a;
}

Given mod_reduce, multiplication is relatively straight forward:

u64 mod_mul(u64 x, u64 y)
{
 u32 a = x,
 b = x >>32,
 c = y,
 d = y >>32;

 /* first synthesise the product using 32*32 -> 64 bit multiplies */
 x = b * (u64)c;
 y = a * (u64)d;
 u64 e = a * (u64)c,
 f = b * (u64)d,
 t;

 x += y; /* b*c + a*d */
 /* carry? */
 if (x < y)
 f += 1LL << 32; /* carry into upper 32 bits - can't overflow */

 t = x << 32;
 e += t; /* a*c + LSW(b*c + a*d) */
 /* carry? */
 if (e < t)
 f += 1; /* carry into upper 64 bits - can't overflow*/
 t = x >>32;
 f += t; /* b*d + MSW(b*c + a*d) */
 /* can't overflow */

 /* now reduce: (b*d + MSW(b*c + a*d), a*c + LSW(b*c + a*d)) */
 return mod_reduce(f, e);
}

The definitions for the other arithmetic operations are straight forward now. Power is just repeated multiplication and squaring. Inverse is to the power of -1.

All those things come together in the mersenne_mul function above!

If you want to look at the complete (mostly) unobfuscated code then take look through mersenne.c.

Conclusion

The IOCCC was a fun challenge and it got me to finally complete my C conversion of a lot of really difficult ARM assembly code. Hopefully I'll find the time to demonstrate some more optimised FFTs and an FFT with a factor of 3 in too.

Getting the code small enough was really, really difficult, but helped immensely by Python. It started off life as pretty obscure so that process made it more so! I'm afraid even as explained above it probably isn't accessible to very many people - my apologies for that.

Happy Prime Hunting!

How I Solved the GCHQ challenge

nick@craig-wood.com (Nick Craig-Wood) — Sun, 04 Dec 2011 20:00:00 +0000

This is an unsanitised account of how I solved the GCHQ challenge at http://www.canyoucrackit.co.uk/. According to the BBC the competition began in secret on the 3rd of November 2011 and will continue until the 12th of December. I was going to hold back this publication until the contest ended but a solution has just made it to the front page of slashdot so I think the jig is up!

This writeup includes the wrong turnings I took and the bad assumptions I made along the way so any reader can see the kind of thought processes necessary. I very deliberately did no searching on the Internet about the puzzle so all the work below (and the mistakes) are mine alone!

The programs linked to in the article are the final versions, I didn't keep the intermediate versions which I talk about in the text, so you'll have to imagine what they looked like.

Warning

This article contains spoilers on how to do the challenge - don't read any further if you want to solve it yourself!

Stage 1

The challenge opens showing a single image with a load of hex digits in and a form to submit your answer. The image looks like this:

The first thing I did was to decode the hex data. Using a python program of course! I played around with trying to get it to display an image. It is obviously low entropy but what is it? Not an image anyway.

I used cyber.py to save it to a binary file and ran the unix "file" utility on it which told me it was x86 code:

$ file cyber.bin
cyber.bin: DOS executable (COM)

Interesting. A disassembly was required cyber.disassembly.asm:

x86dis -r 0 160 -s intel < cyber.bin > cyber.disassembly.asm

The code appeared to be linux flavour, exiting politely with the correct int 0x80 call.

The obvious next step is to run the code. It is bare code which you can't just run on any modern OS. I could have added headers to it but I decided to write cyber.c to load it into memory instead. I used mmap to map a padded version of the code so I could get the code and the stack under my control and examine it after the code had exited:

gcc -g -m32 -Wall -static -o cyber cyber.c

Running the code, it seemed to be early terminating - needing 0x42424242 or "BBBB" on the end to continue according to this bit of code:

0000004A 58 pop eax
0000004B 3D 42 42 42 42 cmp eax, 0x42424242
00000050 75 3B jnz 0x0000008D ; exit

I tried padding with "BBBB" and it core dumped this time. After studying the disassembly some more and experimenting I noted that it needed "BBBB" and a 4 byte length. Running that it appears to decrypt something on the stack this time, so I'm getting somewhere.

But what to decrypt? I needed a string starting with "BBBB". I recursively downloaded the entire website and grepped it for "BBBB" without success. However on really close examination of a hex dump of cyber.png I discovered this:

00000050: 1233 7e39 c170 0000 005d 6954 5874 436f .3~9.p...]iTXtCo
00000060: 6d6d 656e 7400 0000 0000 516b 4a43 516a mment.....QkJCQj
00000070: 4941 4141 4352 3250 4674 6343 4136 7132 IAAACR2PFtcCA6q2
00000080: 6561 4338 5352 2b38 646d 442f 7a4e 7a4c eaC8SR+8dmD/zNzL
00000090: 5143 2b74 6433 7446 5134 7178 384f 3434 QC+td3tFQ4qx8O44
000000a0: 3754 4465 755a 7735 502b 3053 7362 4563 7TDeuZw5P+0SsbEc
000000b0: 5952 0a37 386a 4b4c 773d 3d32 cabe f100 YR.78jKLw==2....

That string "QkJCQj...78jKLw==" with upper and lower case letters, '+' and '/' and the trailing '==' screams base64 encoding to me. I decoded it in python and it decodes to BBBB2\x00\x00\x00\x91... -hooray a string starting with "BBBB" and a sensible looking length! I then modified cyber.py to add that on the end of the code and ran the cyber.c binary. After it had ran I examined the stack in the cyber.bin.padded file originally written by cyber.py. I saw this:

00002c0: 00 00 00 00 00 00 00 00 00 00 47 45 54 20 2f 31 ..........GET /1
00002d0: 35 62 34 33 36 64 65 31 66 39 31 30 37 66 33 37 5b436de1f9107f37
00002e0: 37 38 61 61 64 35 32 35 65 35 64 30 62 32 30 2e 78aad525e5d0b20.
00002f0: 6a 73 20 48 54 54 50 2f de e6 fb 2f ef ae 5d aa js HTTP/.../..].

Which looks very like an HTTP transaction, ie a coded instruction to fetch a file, so I did:

$ wget http://www.canyoucrackit.co.uk/15b436de1f9107f3778aad525e5d0b20.js

Stage 2

The downloaded file 15b436de1f9107f3778aad525e5d0b20.js is a description of a VM with an initial state, but no code to implement the VM. It has a very realistic and amusing initial commment! It also says "stage 2 of 3" which is the first indication how long this challenge is going to be.

Otherwise it seems a reasonably straightforward job to implement the VM and I got cracking on vm.py after reading that. Note that it has 8 bytes of 'firmware' which don't seem to fit in anywhere which is a bit puzzling.

Wasted a lot of time trying to get the VM to work. Tried poking the firmware in various imaginitive places. Found a few bugs then finally re-read the doc again - Ah-ha it is 16 byte segment size, not a 16 bit register... Duh!

Fixed that and it runs much better now actually decrypting stuff and halts properly.

Found this in the memory in final.core after vm.py had run:

00001c0: 47 45 54 20 2f 64 61 37 35 33 37 30 66 65 31 35 GET /da75370fe15
00001d0: 63 34 31 34 38 62 64 34 63 65 65 63 38 36 31 66 c4148bd4ceec861f
00001e0: 62 64 61 61 35 2e 65 78 65 20 48 54 54 50 2f 31 bdaa5.exe HTTP/1
00001f0: 2e 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .0..............

That seems like an instruction to fetch something from the web site again:

$ wget http://www.canyoucrackit.co.uk/da75370fe15c4148bd4ceec861fbdaa5.exe

Haven't used the firmware - wonder where that fits in...

Stage 3

I looked in da75370fe15c4148bd4ceec861fbdaa5.exe

and found it to be a windows x86 executable. It seems to be using the cygcrypt dll from cygwin and the crypt() function. It has a string in it which looks exactly like a DES password crypt "hqDTK7b8K2rvw". I then set John the Ripper and crack off on it for good measure to find the encrypted password.

John the Ripper found the password cyberwin in 2 hours. The easy one was my test to make sure it was working:

Loaded 2 password hashes with 2 different salts (Traditional DES [128/128 BS SSE2-16])
easy (trivial)
cyberwin (test)
guesses: 2 time: 0:02:01:42 (3) c/s: 1537K trying: cufqnm5! - cyberwen
Use the "--show" option to display all of the cracked passwords reliably

And double checking with python:

>>> crypt("cyberwin", "hq") == "hqDTK7b8K2rvw"
True

Meanwhile I tried running the exe on windows but it doesn't run without that dll.

After installing cygwin with the "crypt" package which has the correct dll in, I copied cygcrypt-0.dll and cygwin1.dll into my working directory. The exe now runs and gives:

>da75370fe15c4148bd4ceec861fbdaa5.exe
keygen.exe
usage: keygen.exe hostname
>da75370fe15c4148bd4ceec861fbdaa5.exe localhost
keygen.exe
error: license.txt not found

I then tried it with "cyberwin" in license.txt:

>da75370fe15c4148bd4ceec861fbdaa5.exe localhost
keygen.exe
error: license.txt invalid

Hmm, I was expecting that to work.

Looking through the keygen.edit.asm (my annotated version) I discovered that the password should be prefixed with "gchq". The first hint as to who set this puzzle! :

cmp [ebp+var_38], 71686367h ; gchq
jnz short invalid_license

Putting "gchqcyberwin" into the license.txt and running the program goes better this time. It fetches a page this time, but it is a 404 not found. Note that this isn't the normal 404 page because the exe uses HTTP/1.0 rather than HTTP/1.1:

>da75370fe15c4148bd4ceec861fbdaa5.exe www.canyoucrackit.co.uk
keygen.exe
loading stage1 license key(s)...
loading stage2 license key(s)...
request:
GET /hqDTK7b8K2rvw/0/0/0/key.txt HTTP/1.0
response:
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Sat, 03 Dec 2011 09:29:29 GMT
Connection: close
Content-Length: 315
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>

Trying the above in the web browser gives the normal 404 message. Trying with telnet see that it needs HTTP/1.0. HTTP/1.1 with host header gives normal page. Trying "GET / HTTP/1.0" gives the same message so probably a red herring to do with the server (or not see later!).

The fact that the page isn't found means that there is something missing. But what? The code is executing the equivalent of this to make the GET string to fetch the page:

sprintf(buffer, "GET /%s/%x/%x/%x/key.txt HTTP/1.0\r\n\r\n", crypted_string, key1, key2, key2);

However key1, key2 and key3 are all 0 which doesn't look right. I tried some things for those missing %x parameters:

>>> t = "gchqcyberwin"
>>> from struct import *
>>> [ "%x" %x for x in unpack(">iii", t) ]
['67636871', '63796265', '7277696e']
>>> [ "%x" %x for x in unpack("<iii", t) ]
['71686367', '65627963', '6e697772']

Try:

wget http://www.canyoucrackit.co.uk/hqDTK7b8K2rvw/71686367/65627963/6e697772/key.txt
wget http://www.canyoucrackit.co.uk/hqDTK7b8K2rvw/67636871/63796265/7277696e/key.txt

But no luck.

More study of the keygen.edit.asm disassembly reveals that key1, key2, key3 come from the end of the license.txt file after "gchqcyberwin". So that makes 24 bytes of secrets read from the licence file which is the size of the buffer the code allocates.

Ah-Ha! The clue is in the "loading stageX license key(s)..." messages.

This bit of assembler code gives it away (annotations by me):

loc_4011A5: ; "loading stage1 license key(s)...\n"
mov [esp+78h+var_78], offset aLoadingStage1L
call printf
mov eax, [ebp+var_2C] ; copy 4 bytes of the licence file
mov [ebp+var_48], eax
mov [esp+78h+var_78], offset aLoadingStage2L ; "loading stage2 license key(s)...\n\n"
call printf
mov eax, [ebp+var_28] ; ...and another 4 bytes
mov [ebp+var_44], eax
mov eax, [ebp+var_24] ; ..and another 4 bytes
mov [ebp+var_40], eax

It prints "loading stage1 license keys(s)..." loads 4 bytes, then "loading stage2 license key(s)..." and loads 8 bytes. Stage 1 is the first stage of the puzzle - need 4 bytes from this - how about the unused 4 bytes at the start of the code that is jumped over "af c2 bf a3". Stage2 is the second stage from which we need 8 bytes - the mysteriously unused firmware seems appropriate.

I wrote keyfetch.py to fiddle with the endianess and after a bit of trial and error it worked:

GET 'http://www.canyoucrackit.co.uk/hqDTK7b8K2rvw/a3bfc2af/d2ab1f05/da13f110/key.txt'
Pr0t3ct!on

Fetched using HTTP/1.1 and the GET program. Which looks like it could be the password! But it doesn't work :-(

The headers showed nothing interesting. I then tried using keygen.exe with a corrected license file - it didn't work as I expected as the webserver doesn't support HTTP/1.0 (or maybe I did something wrong).

However trying it by hand using telnet and HTTP/1.0 does do something different:

$ telnet www.canyoucrackit.co.uk 80
Trying 31.222.164.161...
Connected to www.canyoucrackit.co.uk.
Escape character is '^]'.
GET http://www.canyoucrackit.co.uk/hqDTK7b8K2rvw/a3bfc2af/d2ab1f05/da13f110/key.txt HTTP/1.0
HTTP/1.1 200 OK
Content-Type: text/plain
Last-Modified: Wed, 26 Oct 2011 08:40:14 GMT
Accept-Ranges: bytes
ETag: "bc46bae1ba93cc1:0"
Server: Microsoft-IIS/7.5
Date: Sun, 04 Dec 2011 11:14:54 GMT
Connection: close
Content-Length: 37
Pr0t3ct!on#cyber_security@12*12.2011+Connection closed by foreign host.
$

I reworked keyfetch.py to make the fetch using HTTP/1.0 to double check.

Trying Pr0t3ct!on#cyber_security@12*12.2011+ as the key does indeed work and produces this page: http://www.canyoucrackit.co.uk/soyoudidit.asp:

So you did it. Well done! Now this is where it gets interesting. Could you use your skills and ingenuity to combat terrorism and cyber threats? As one of our experts, you'll help protect our nation's security and the lives of thousands. Every day will bring new challenges, new solutions to find -- and new ways to prove that you're one of the best.

I'm not going to apply for a job as I'm rather fully employed elsewhere, but it was a fun challenge!

Postmortem

I didn't see this until the 1st December 2011 when a colleague at work (thanks Tom!) mentioned it to me and I didn't have time to work on it until Friday the 2nd of December. It took me one very late night on Friday and intermittent hacking on Saturday and Sunday to complete the challenge - about 12 hours in total. I spent 3 hours tracking down that mis-understanding in vm.py about 16 byte segments and 2 hours trying to work out what the missing 12 bytes were in the URL in stage 3.

I think Stage 1 was very hard - perhaps deliberately so. Stage 2 was much easier - there was a defined goal and any competent computer scientist would be able to knock up the VM code. Stage 3 involved an awful lot of reading C compiler created x86 assembler which was painful. I suspect I could have done it a lot quicker if I'd had some better tools. An interactive disassembling debugger would have been useful - I used to have such a thing when I did 68000 programming and it was a wonder.

I note that I didn't actually have to crack the encrypted password in Stage 3. I could have just changed one byte and have it ignore the check but I was expecting that the code would actually need to use it. Alas no and so much for John the Ripper!

Finally thanks to GCHQ for making the challenge - it was a good one!

Pi - Chudnovsky

nick@craig-wood.com (Nick Craig-Wood) — Sat, 08 Oct 2011 00:00:00 +0000

In Part 3 we managed to calculate 1,000,000 decimal places of π with Machin's arctan formula. Our stated aim was 100,000,000 places which we are going to achieve now!

Fun with Maths and Python

This is a having fun with maths and python article. See the introduction for important information!

We have still got a long way to go though, and we'll have to improve both our algorithm (formula) for π and our implementation.

The current darling of the π world is the Chudnovsky algorithm which is similar to the arctan formula but it converges much quicker. It is also rather complicated. The formula itself is derived from one by Ramanjuan who's work was extraordinary in the extreme. It isn't trivial to prove, so I won't! Here is Chudnovsky's formula for π as it is usually stated:

$$ \frac{1}{\pi} = 12 \sum^\infty_{k=0} \frac{(-1)^k (6k)! (13591409 + 545140134k)}{(3k)!(k!)^3 640320^{3k + 3/2}} $$

That is quite a complicated formula, we will make more comprehensible in a moment. If you haven't seen the Σ notation before it just like a sum over a for loop in python. For example:

$$ \sum^{10}_{k=1} k^2 $$

Is exactly the same as this python fragment:

sum(k**2 for k in range(1,11))

First let's get rid of that funny fractional power:

$$ \begin{align} \frac{1}{\pi} &= 12 \sum^\infty_{k=0} \frac{(-1)^k (6k)! (13591409 + 545140134k)}{(3k)!(k!)^3 640320^{3k + 3/2}} \\ &= \frac{12}{640320 \sqrt{640320}} \sum^\infty_{k=0} \frac{(-1)^k (6k)! (13591409 + 545140134k)}{(3k)!(k!)^3 640320^{3k}} \\ &= \frac{1}{426880 \sqrt{10005}} \sum^\infty_{k=0} \frac{(-1)^k (6k)! (13591409 + 545140134k)}{(3k)!(k!)^3 640320^{3k}} \\ \end{align} $$

Now let's split it into two independent parts. See how there is a + sign inside the Σ? We can split it into two series which we will call a and b and work out what π is in terms of a and b:

$$ \begin{align} a &= \sum^\infty_{k=0} \frac{(-1)^k (6k)!}{(3k)!(k!)^3 640320^{3k}} \\ &= 1 - \frac{6\cdot5\cdot4}{(1)^3 640320^3} + \frac{12\cdot11\cdot10\cdot9\cdot8\cdot7}{(2\cdot1)^3 640320^6} - \frac{18\cdot17\cdots13}{(3\cdot2\cdot1)^3 640320^{9}} + \cdots \\ b &= \sum^\infty_{k=0} \frac{(-1)^k (6k)!k}{(3k)!(k!)^3 640320^{3k}} \\ \frac{1}{\pi} &= \frac{13591409a + 545140134b}{426880 \sqrt{10005}} \\ \pi &= \frac{426880 \sqrt{10005}}{13591409a + 545140134b} \end{align} $$

Finally note that we can calculate the next a term from the previous one, and the b terms from the a terms which simplifies the calculations rather a lot.

$$ \begin{align} a_k &= \frac{(-1)^k (6k)!}{(3k)!(k!)^3 640320^{3k}} \\ b_k &= k \cdot a_k \\ \frac{a_{k}}{a_{k-1}} &= - \frac{(6k-5)(6k-4)(6k-3)(6k-2)(6k-1)6k}{3k(3k-1)(3k-2)k^3 640320^3} \\ &= - \frac{24(6k-5)(2k-1)(6k-1)}{k^3 640320^3} \\ \end{align} $$

OK that was a lot of maths! But it is now in an easy to compute state, which we can do like this:

def pi_chudnovsky(one=1000000):
 """
 Calculate pi using Chudnovsky's series

 This calculates it in fixed point, using the value for one passed in
 """
 k = 1
 a_k = one
 a_sum = one
 b_sum = 0
 C = 640320
 C3_OVER_24 = C**3 // 24
 while 1:
 a_k *= -(6*k-5)*(2*k-1)*(6*k-1)
 a_k //= k*k*k*C3_OVER_24
 a_sum += a_k
 b_sum += k * a_k
 k += 1
 if a_k == 0:
 break
 total = 13591409*a_sum + 545140134*b_sum
 pi = (426880*sqrt(10005*one, one)*one) // total
 return pi

We need to be able to take the square root of long integers which python doesn't have a built in for. Luckily this is easy to provide. This uses a square root algorithm devised by Newton which doubles the number of significant places in the answer (quadratic convergence) each iteration:

 def sqrt(n, one):
 """
 Return the square root of n as a fixed point number with the one
 passed in. It uses a second order Newton-Raphson convergence. This
 doubles the number of significant figures on each iteration.
 """
 # Use floating point arithmetic to make an initial guess
 floating_point_precision = 10**16
 n_float = float((n * floating_point_precision) // one) / floating_point_precision
 x = (int(floating_point_precision * math.sqrt(n_float)) * one) // floating_point_precision
 n_one = n * one
 while 1:
 x_old = x
 x = (x + n_one // x) // 2
 if x == x_old:
 break
 return x

It uses normal floating point arithmetic to make an initial guess then refines it using Newton's method.

See pi_chudnovsky.py for the complete program. When I run it I get this:

Digits	Time (seconds)
10	4.696e-05
100	0.0001530
1000	0.002027
10000	0.08685
100000	8.453
1000000	956.3

Which is nearly 3 times quicker than the best result in part 3, and gets us 1,000,000 places of π in about 15 minutes.

Amazingly there are still two major improvements we can make to this. The first is to recast the calculation using something called binary splitting. Binary splitting is a general purpose technique for speeding up this sort of calculation. What it does is convert the sum of the individual fractions into one giant fraction. This means that you only do one divide at the end of the calculation which speeds things up greatly, because division is slow compared to multiplication.

Consider the general infinite series

$$ S(0,\infty) = \frac{a_0p_0}{b_0q_0} + \frac{a_1p_0p_1}{b_1q_0q_1} + \frac{a_2p_0p_1p_2}{b_2q_0q_1q_2} + \frac{a_3p_0p_1p_2p_3}{b_3q_0q_1q_2q_3} + \cdots $$

This could be used as a model for all the infinite series we've looked at so far. Now lets consider the partial sum of that series from terms a to b (including a, not including b).

$$ \begin{align} S(a,b) &= \frac{a_ap_a}{b_aq_a} + \frac{a_{a+1}p_ap_{a+1}}{b_{a+1}q_aq_{a+1}} + \frac{a_{a+2}p_ap_{a+1}p_{a+2}}{b_{a+2}q_aq_{a+1}q_{a+2}} + \cdots + \frac{a_{b-1}p_ap_{a+1}p_{a+2}\cdots p_{b-1}}{b_{b-1}q_aq_{a+1}\cdots p_{b-1}} \\ \end{align} $$

This is a part of the infinite series, and if a=0 and b=∞ it becomes the infinite series above.

Now lets define some extra functions:

$$ \begin{align} P(a,b) &= p_ap_{a+1}\cdots p_{b-1} \\ Q(a,b) &= q_aq_{a+1}\cdots q_{b-1} \\ B(a,b) &= b_ab_{a+1}\cdots b_{b-1} \\ T(a,b) &= B(a,b)Q(a,b)S(a,b) \\ \end{align} $$

Let's define m which is a <= m <= b. Making m as near to the middle of a and b will lead to the quickest calculations, but for the proof, it has to be somewhere between them. Lets work out what happens to our variables P, Q, R and T when we split the series in two:

$$ \begin{align} P(a,b) &= P(a,m)P(b,m) \\ Q(a,b) &= Q(a,m)Q(b,m) \\ B(a,b) &= B(a,m)B(b,m) \\ T(a,b) &= B(m,b)Q(m,b)T(a,m) + B(a,m)P(a,m)T(m,b) \\ \end{align} $$

The first three of those statements are obvious from the definitions, but the last deserves proof.

$$ \begin{align} & B(m,b)Q(m,b)T(a,m) + B(a,m)P(a,m)T(m,b) && \left[T(a,b) = B(a,b)Q(a,b)S(a,b)\right] \\ &= B(m,b)Q(m,b)B(a,m)Q(a,m)S(a,m) + B(a,m)P(a,m)B(m,b)Q(m,b)S(m,b) && \left[Q(a,b) = Q(a,m)Q(b,m)\right] \\ & && \left[B(a,b) = B(a,m)B(b,m)\right] \\ &= B(a,b)Q(a,b)S(a,m) + B(a,b)P(a,m)Q(m,b)S(m,b) \\ &= B(a,b)(Q(a,b)S(a,m) + P(a,m)Q(m,b)S(m,b)) && \left[Q(m,b) = \frac{Q(a,b)}{Q(a,m)}\right] \\ &= B(a,b)\left(Q(a,b)S(a,m) + \frac{Q(a,b)P(a,m)}{Q(a,m)}S(m,b)\right) \\ &= B(a,b)Q(a,b)\left(S(a,m) + \frac{P(a,m)}{Q(a,m)}S(m,b)\right) \\ &= B(a,b)Q(a,b)\left( \frac{a_ap_a}{b_aq_a} + \frac{a_{a+1}p_ap_{a+1}}{b_{a+1}q_aq_{a+1}} + \frac{a_{a+2}p_ap_{a+1}p_{a+2}}{b_{a+2}q_aq_{a+1}q_{a+2}} + \cdots + \frac{a_{m-1}p_ap_{a+1}p_{a+2}\cdots p_{m-1}}{b_{m-1}q_aq_{a+1}\cdots p_{m-1}} + \frac{p_ap_{a+1}\cdots p_{m-1}}{q_aq_{a+1}\cdots q_{m-1}} + \left(\frac{a_mp_m}{b_mq_m} + \frac{a_{m+1}p_mp_{m+1}}{b_{m+1}q_mq_{m+1}} + \frac{a_{m+2}p_mp_{m+1}p_{m+2}}{b_{m+2}q_mq_{m+1}q_{m+2}} + \cdots + \frac{a_{b-1}p_mp_{m+1}p_{m+2}\cdots p_{b-1}}{b_{b-1}q_mq_{m+1}\cdots p_{b-1}} \right)\right) \\ &= B(a,b)Q(a,b)\left( \frac{a_ap_a}{b_aq_a} + \frac{a_{a+1}p_ap_{a+1}}{b_{a+1}q_aq_{a+1}} + \frac{a_{a+2}p_ap_{a+1}p_{a+2}}{b_{a+2}q_aq_{a+1}q_{a+2}} + \cdots + \frac{a_{m-1}p_ap_{a+1}p_{a+2}\cdots p_{m-1}}{b_{m-1}q_aq_{a+1}\cdots p_{m-1}} + \frac{p_ap_{a+1}\cdots p_{m-1}}{q_aq_{a+1}\cdots q_{m-1}} + \frac{a_mp_ap_{a+1}\cdots p_m}{b_mq_aq_{a+1}\cdots q_m} + \frac{a_{m+1}p_ap_{a+1}\cdots p_{m+1}}{b_{m+1}q_aq_{a+1}\cdots q_{m+1}} + \frac{a_{m+2}p_ap_{a+1}\cdots p_{m+2}}{b_{m+2}q_aq_{a+1}\cdots q_{m+2}} + \cdots + \frac{a_{b-1}p_ap_{a+1}\cdots p_{b-1}}{b_{b-1}q_aq_{a+1}\cdots p_{b-1}} \right) \\ &= B(a,b)Q(a,b)S(a,b) \\ &= T(a,b) \\ \end{align} $$

We can use these relations to expand the series recursively, so if we want S(0,8) then we can work out S(0,4) and S(4,8) and combine them. Likewise to calculate S(0,4) and S(4,8) we work out S(0,2), S(2,4), S(4,6), S(6,8) and combine them, and to work out those we work out S(0,1), S(1,2), S(2,3), S(3,4), S(4,5), S(5,6), S(6,7), S(7,8). Luckily we don't have to split them down any more as we know what P(a,a+1), Q(a,a+1) etc is from the definition above.

$$ \begin{align} P(a,a+1) &= p_a \\ Q(a,a+1) &= q_a \\ B(a,a+1) &= b_a \\ S(a,a+1) &= \frac{a_ap_a}{b_aq_a} \\ T(a,a+1) &= B(a,a+1)Q(a,a+1)S(a,a+1) \\ &= b_aq_a\frac{a_ap_a}{b_aq_a} \\ &= a_ap_a \\ \end{align} $$

And when you've finally worked out P(0,n),Q(0,n),B(0,n),T(0,n) you can work out S(0,n) with

$$ \begin{align} S(0,n) &= \frac{T(0,n)}{B(0,n)Q(0,n)} \\ \end{align} $$

If you want a more detailed and precise treatment of binary splitting then see Bruno Haible and Thomas Papanikolaou's paper.

So back to Chudnovksy's series. We can now set these parameters in the above general formula:

$$ \begin{align} p_0 &= 1 \\ p_a &= (6a-5)(2a-1)(6a-1) \\ q_0 &= 1 \\ q_a &= a^3\cdot640320^3/24 \\ b_a &= 1 \\ a_a &= (13591409 + 545140134a) \\ \end{align} $$

This then makes our Chudnovksy pi function look like this:

def pi_chudnovsky_bs(digits):
 """
 Compute int(pi * 10**digits)

 This is done using Chudnovsky's series with binary splitting
 """
 C = 640320
 C3_OVER_24 = C**3 // 24
 def bs(a, b):
 """
 Computes the terms for binary splitting the Chudnovsky infinite series

 a(a) = +/- (13591409 + 545140134*a)
 p(a) = (6*a-5)*(2*a-1)*(6*a-1)
 b(a) = 1
 q(a) = a*a*a*C3_OVER_24

 returns P(a,b), Q(a,b) and T(a,b)
 """
 if b - a == 1:
 # Directly compute P(a,a+1), Q(a,a+1) and T(a,a+1)
 if a == 0:
 Pab = Qab = 1
 else:
 Pab = (6*a-5)*(2*a-1)*(6*a-1)
 Qab = a*a*a*C3_OVER_24
 Tab = Pab * (13591409 + 545140134*a) # a(a) * p(a)
 if a & 1:
 Tab = -Tab
 else:
 # Recursively compute P(a,b), Q(a,b) and T(a,b)
 # m is the midpoint of a and b
 m = (a + b) // 2
 # Recursively calculate P(a,m), Q(a,m) and T(a,m)
 Pam, Qam, Tam = bs(a, m)
 # Recursively calculate P(m,b), Q(m,b) and T(m,b)
 Pmb, Qmb, Tmb = bs(m, b)
 # Now combine
 Pab = Pam * Pmb
 Qab = Qam * Qmb
 Tab = Qmb * Tam + Pam * Tmb
 return Pab, Qab, Tab
 # how many terms to compute
 DIGITS_PER_TERM = math.log10(C3_OVER_24/6/2/6)
 N = int(digits/DIGITS_PER_TERM + 1)
 # Calclate P(0,N) and Q(0,N)
 P, Q, T = bs(0, N)
 one = 10**digits
 sqrtC = sqrt(10005*one, one)
 return (Q*426880*sqrtC) // T

Hopefully you'll see how the maths above relates to this as I've used the same notation in each. Note that we calculate the number of digits of pi we expect per term of the series (about 14.18) to work out how many terms of the series we need to compute, as the binary splitting algorithm needs to know in advance how many terms to calculate. We also don't bother calculating B(a) as it is always 1. Defining a function inside a function like this makes what is known as a closure. This means that the inner function can access the variables in the outer function which is very convenient in this recursive algorithm as it stops us having to pass the constants to every call of the function.

See pi_chudnovsky_bs.py for the complete program. When I run it I get this:

Digits	Time (seconds)
10	1.096e-05
100	3.194e-05
1000	0.0004899
10000	0.03403
100000	3.625
1000000	419.1

This is a bit more than twice as fast as pi_chudnovsky.py giving us our 1,000,000 places in just under 7 minutes. If you profile it you'll discover that almost all the time spent in the square root calculations (86% of the time) whereas only 56 seconds is spent in the binary splitting part. We could spend time improving the square root algorithm, but it is time to bring out the big guns: gmpy.

Gmpy is a python interface to the gmp library which is a C library for arbitrary precision arithmetic. It is very fast, much faster than the built in int type in python for large numbers. Luckily gmpy provides a type (mpz) which works exactly like normal python int types, so we have to make hardly any changes to our code to use it. These are using initialising variables with the mpz type, and using the sqrt method on mpz rather than our own home made sqrt algorithm:

import math
from gmpy2 import mpz
from time import time

def pi_chudnovsky_bs(digits):
 """
 Compute int(pi * 10**digits)

 This is done using Chudnovsky's series with binary splitting
 """
 C = 640320
 C3_OVER_24 = C**3 // 24
 def bs(a, b):
 """
 Computes the terms for binary splitting the Chudnovsky infinite series

 a(a) = +/- (13591409 + 545140134*a)
 p(a) = (6*a-5)*(2*a-1)*(6*a-1)
 b(a) = 1
 q(a) = a*a*a*C3_OVER_24

 returns P(a,b), Q(a,b) and T(a,b)
 """
 if b - a == 1:
 # Directly compute P(a,a+1), Q(a,a+1) and T(a,a+1)
 if a == 0:
 Pab = Qab = mpz(1)
 else:
 Pab = mpz((6*a-5)*(2*a-1)*(6*a-1))
 Qab = mpz(a*a*a*C3_OVER_24)
 Tab = Pab * (13591409 + 545140134*a) # a(a) * p(a)
 if a & 1:
 Tab = -Tab
 else:
 # Recursively compute P(a,b), Q(a,b) and T(a,b)
 # m is the midpoint of a and b
 m = (a + b) // 2
 # Recursively calculate P(a,m), Q(a,m) and T(a,m)
 Pam, Qam, Tam = bs(a, m)
 # Recursively calculate P(m,b), Q(m,b) and T(m,b)
 Pmb, Qmb, Tmb = bs(m, b)
 # Now combine
 Pab = Pam * Pmb
 Qab = Qam * Qmb
 Tab = Qmb * Tam + Pam * Tmb
 return Pab, Qab, Tab
 # how many terms to compute
 DIGITS_PER_TERM = math.log10(C3_OVER_24/6/2/6)
 N = int(digits/DIGITS_PER_TERM + 1)
 # Calclate P(0,N) and Q(0,N)
 P, Q, T = bs(0, N)
 one_squared = mpz(10)**(2*digits)
 sqrtC = (10005*one_squared).sqrt()
 return (Q*426880*sqrtC) // T

See pi_chudnovsky_bs_gmpy.py for the complete program. When I run it I get this:

Digits	Time (seconds)
10	1.597e-05
100	3.409e-05
1000	0.003403
10000	0.003571
100000	0.09120
1000000	1.760
10000000	30.11
100000000	542.2

So we have achieved our goal of calculating 100,000,000 places of π in just under 10 minutes! What is limiting the program now is memory... 100,000,000 places takes about 600MB of memory to run. With 6 GB of free memory it could probably calculate one billion places in a few hours.

What if we wanted to go faster? Well you could use gmp-chudnovsky.c which is a C program which implements the Chudnovsky Algorithm. It is heavily optimised and rather difficult to understand, but if you untangle it you'll see it does exactly the same as above, with one extra twist. The twist is that it factors the fraction in the binary splitting phase as it goes along. If you examine the gmp-chudnovsky.c results page you'll see that the little python program above acquits itself very well - the python program only takes 75% longer than the optimised C program on the same hardware.

One algorithm which has the potential to beat Chudnovsky is the Arithmetic Geometric Mean algorithm which doubles the number of decimal places each iteration. It involves square roots and full precision divisions which makes it tricky to implement well. In theory it should be faster than Chudnovsk but, so far, in practice Chudnovsky is faster.

Here is a chart of all the different π programs we've developed in Part 1, Part 2, Part 3 and Part 4 with their timings on my 2010 Intel® Core™ i7 CPU Q820 @ 1.73GHz running 64 bit Linux:

Source spreadsheet for the timings.

Pi - Machin

nick@craig-wood.com (Nick Craig-Wood) — Tue, 19 Jul 2011 00:00:00 +0000

In Part 2 we managed to calculate 100 decimal places of π with Archimedes' recurrence relation for π which was a fine result, but we can do bigger and faster.

Fun with Maths and Python

This is a having fun with maths and python article. See the introduction for important information!

To go faster we'll have to use a two pronged approach - a better algorithm for π, and a better implementation.

In 1706 John Machin came up with the first really fast method of calculating π and used it to calculate π to 100 decimal places. Quite an achievement considering he did that entirely with pencil and paper. Machin's formula is:

π/4 = 4cot⁻¹(5) - cot⁻¹(239)
= 4tan⁻¹(1/5) - tan⁻¹(1/239)
= 4 * arctan(1/5) - arctan(1/239)

We can prove Machin's formula using some relatively easy math! Hopefully you learnt this trigonometric formula at school:

$$ tan(a + b) = \frac{tan(a) + tan(b)}{1 - tan(a) \cdot tan(b)} $$

Substitute a = arctan(x) and b = arctan(y) into it, giving:

$$ tan(arctan(x) + arctan(y)) = \frac{tan(arctan(x)) + tan(arctan(y)}{1 - tan(arctan(x)) \cdot tan(arctan(y))} $$

Note that tan(arctan(x)) = x which gives:

$$ tan(arctan(x) + arctan(y)) = \frac{x + y}{1 - xy} $$

arctan both sides to get:

$$ arctan(x) + arctan(y) = arctan\left(\frac{x + y}{1 - xy}\right) $$

This gives us a method for adding two arctans. Using this we can prove Machin's formula with a bit of help from python's fractions module:

>>> from fractions import Fraction
>>> def add_arctan(x, y): return (x + y) / (1 - x * y)
...
>>> a = add_arctan(Fraction(1,5), Fraction(1,5))
>>> a # 2*arctan(1/5)
Fraction(5, 12)
>>> b = add_arctan(a, a)
>>> b # 4*arctan(1/5)
Fraction(120, 119)
>>> c = add_arctan(b, Fraction(-1,239))
>>> c # 4*arctan(1/5) - arctan(1/239)
Fraction(1, 1)
>>>

So Machin's formula is equal to arctan(1/1) = π/4 QED! Or if you prefer it written out in mathematical notation:

$$ \begin{align} 4arctan\left(\frac{1}{5}\right) - arctan\left(\frac{1}{239}\right) &= 2\left(arctan\left(\frac{1}{5}\right) + arctan\left(\frac{1}{5}\right)\right) - arctan\left(\frac{1}{239}\right) \\ &= 2arctan\left(\frac{5}{12}\right) - arctan\left(\frac{1}{239}\right) \\ &= arctan\left(\frac{5}{12}\right) + arctan\left(\frac{5}{12}\right) - arctan\left(\frac{1}{239}\right) \\ &= arctan\left(\frac{120}{119}\right) - arctan\left(\frac{1}{239}\right) \\ &= arctan\left(\frac{1}{1}\right) \\ &= \frac{\pi}{4} \\ \end{align} $$

So how do we use Machin's formula to calculate π? Well first we need to calculate arctan(x) which sounds complicated, but we've seen the infinite series for it already in Part 1 :

$$ arctan(x) = x - \frac{x^3}{3} + \frac{x^5}{5} - \frac{x^7}{7} + \frac{x^9}{9} - \cdots ( -1 <= x <= 1 ) $$

We can see that if x=1/5 then the terms get smaller very quickly, which means that the series will converge quickly. Let's substitute x = 1/x and get:

$$ arctan(1/x) = \frac{1}{x} - \frac{1}{3x^3} + \frac{1}{5x^5} - \frac{1}{7x^7} + \frac{1}{9x^9} - \cdots ( x >= 1 ) $$

Each term in the series is created by dividing the previous term by a small integer. Dividing two 100 digit numbers is hard work as I'm sure you can imagine from your experiences with long division at school! However it is much easier to divide a short number (a single digit) into a 100 digit number. We called this short division at school, and that is the key to making a much faster π algorithm. In fact if you are dividing an N digit number by an M digit number it takes rougly N*M operations. That square root we did in Part 2 did dozens of divisions of 200 digit numbers. So if we could somehow represent the current term in a int¹ then we could use this speedy division to greatly speed up the calculation.

The way we do that is to multiply everything by a large number, lets say 10¹⁰⁰. We then do all our calculations with integers, knowing that we should shift the decimal place 100 places to the left when done to get the answer. This needs a little bit of care, but is a well known technique known as fixed point arithmetic.

The definition of the arctan(1/x) function then looks like this:

def arctan(x, one=1000000):
 """
 Calculate arctan(1/x)

 arctan(1/x) = 1/x - 1/3*x**3 + 1/5*x**5 - ... (x >= 1)

 This calculates it in fixed point, using the value for one passed in
 """
 power = one // x # the +/- 1/x**n part of the term
 total = power # the total so far
 x_squared = x * x # precalculate x**2
 divisor = 1 # the 1,3,5,7 part of the divisor
 while 1:
 power = - power // x_squared
 divisor += 2
 delta = power // divisor
 if delta == 0:
 break
 total += delta
 return total

The value one passed in is the multiplication factor as described above. You can think of it as representing 1 in the fixed point arithmetic world. Note the use of the // operator which does integer divisions. If you don't use this then python will make floating point values². In the loop there are two divisions, power // x_squared and power // divisor. Both of these will be dividing by small numbers, x_squared will be 5² = 25 or 239² = 57121 and divisor will be 2 * the number of iterations which again will be small.

So how does this work in practice? On my machine it calculates 100 digits of π in 0.18 ms which is over 30,000 times faster than the previous calculation with the decimal module in Part 2!

Can we do better?

Well the answer is yes! Firstly there are better arctan formulae. Amazingly there are other formulae which will calculate π too, like these, which are named after their inventors:

def pi_machin(one):
 return 4*(4*arctan(5, one) - arctan(239, one))

def pi_ferguson(one):
 return 4*(3*arctan(4, one) + arctan(20, one) + arctan(1985, one))

def pi_hutton(one):
 return 4*(2*arctan(3, one) + arctan(7, one))

def pi_gauss(one):
 return 4*(12*arctan(18, one) + 8*arctan(57, one) - 5*arctan(239, one))

It turns out that Machin's formula is really very good, but Gauss's formula is slightly better.

The other way we can do better is to use a better formula for arctan(). Euler came up with this accelerated formula for arctan:

$$ arctan(1/x) = \frac{x}{1+x^2} + \frac{2}{3}\frac{x}{(1+x^2)^2} + \frac{2\cdot4}{3\cdot5}\frac{x}{(1+x^2)^3} + \frac{2\cdot4\cdot6}{3\cdot5\cdot7}\frac{x}{(1+x^2)^4} + \cdots $$

This converges to arctan(1/x) at the roughly the same rate per term than the formula above, however each term is made directly from the previous term by multiplying by 2n and dividing by (2n+1)(1+x²). This means that it can be implemented with only one (instead of two) divisions per term, and hence runs roughly twice as quickly. If we implement this in python in fixed point, then it looks like this:

def arctan_euler(x, one=1000000):
 """
 Calculate arctan(1/x) using euler's accelerated formula

 This calculates it in fixed point, using the value for one passed in
 """
 x_squared = x * x
 x_squared_plus_1 = x_squared + 1
 term = (x * one) // x_squared_plus_1
 total = term
 two_n = 2
 while 1:
 divisor = (two_n+1) * x_squared_plus_1
 term *= two_n
 term = term // divisor
 if term == 0:
 break
 total += term
 two_n += 2
 return total

Notice how we pre-calculate as many things as possible (like x_squared_plus_1) to get them out of the loop.

See the complete program: pi_artcan_integer.py. Here are some timings³ of how the Machin and the Gauss arctan formula fared, with and without the accelerated arctan.

Formula	Digits	Normal Arctan (seconds)	Euler Arctan (seconds)
Machin	10	3.004e-05	2.098e-05
Gauss	10	2.098e-05	1.907e-05
Machin	100	0.0001869	0.0001471
Gauss	100	0.0001869	0.0001468
Machin	1000	0.005599	0.003520
Gauss	1000	0.005398	0.003445
Machin	10000	0.3995	0.2252
Gauss	10000	0.3908	0.2186
Machin	100000	38.33	21.69
Gauss	100000	37.44	21.12
Machin	1000000	3954.1	2396.8
Gauss	1000000	3879.4	2402.1

There are several interesting things to note. Firstly that we just calculated 1,000,000 decimal places of π in 40 minutes! Secondly that to calculate 10 times as many digits it takes 100 times as long. This means that our algorithm scales as O(n²) where n is the number of digits. So if 1,000,000 digits takes an hour, 10,000,000 would take 100 hours (4 days) and 100,000,000 would take 400 days (13 months). Your computer might run out of memory, or explode in some other fashion when calculating 100,000,000 digits, but doesn't seem impossible any more.

The Gauss formula is slightly faster than the Machin for nearly all the results. The accelerated arctan formula is 1.6 times faster than the normal arctan formula so that is a definite win.

A billion (1,000,000,000) digits of pi would take about 76 years to calculate using this program which is a bit out of our reach! There are several ways we could improve this. If we could find an algorithm for calculating π which was quicker than O(n²), or if we could find an O(n²) algorithm which just ran a lot faster. We could also throw more CPUs at the problem. To see how this might be done, we could calculate all the odd terms on one processor, and all the even terms on another processor for the arctan formula and add them together at the end.. That would run just about twice as quickly. That could easily be generalised to lots of processors:

arctan(1/x) = 1/x - 1/3x³ + 1/5x⁵ - 1/7x⁷ + ...
= 1/x + 1/5x⁵ - + ... # run on cpu 1
+ - 1/3x³ - 1/7x⁷ + ... # run on cpu 2

So if we had 76 CPU cores, we could probably calculate π to 1,000,000,000 places in about a year.

There are better algorithms for calculating π though, and there are faster ways of doing arithmetic than python's built in long integers. We'll explore both in Part 4!

Python's integer type was called called long in python 2, just int in python 3 ↩︎
Unless you are using python 2 ↩︎
All timings generated on my 2010 Intel® Core™ i7 CPU Q820 @ 1.73GHz running 64 bit Linux ↩︎

Pi - Archimedes

nick@craig-wood.com (Nick Craig-Wood) — Mon, 18 Jul 2011 00:00:00 +0000

It is clear from Part 1 that in order to calculate π we are going to need a better method than evaluating Gregory's Series.

Fun with Maths and Python

This is a having fun with maths and python article. See the introduction for important information!

Here is one which was originally discovered by Archimedes in 200BC. Most of the proofs for series for calculating π require calculus or other advanced math, but since Archimedes didn't know any of those you can prove it without! In fact I thought I had discovered it for the first time when I was 13 or so, but alas I was 2000 years too late!

Archimedes knew that the ratio of the circumference of a circle to its diameter was π. He had the idea that he could draw a regular polygon inscribed within the circle to approximate π, and the more sides he drew on the polygon, the better approximation to π he would get. In modern maths speak, we would say as the number of sides of the polygon tends to infinity, the circumference tends to 2π.

Instead of drawing the polygon though, Archimedes calculated the length using a geometric argument, like this.

Draw a circle of radius 1 unit centered at A. Inscribe an N sided polygon within it. Our estimate for π is half the circumference of the polygon (circumference of a circle is 2πr, r = 1, giving 2π). As the sides of the polygon get smaller and smaller the circumference gets closer and closer to 2π.

The diagram shows one segment of the polygon ACE. The side of the polygon CE has length d_n. Assuming we know d for an N sided polygon, If we can find an expression for the length CD = d_2n, the edge length of a polygon with 2N sides, in terms of d_n only, then we can improve our estimate for π. Let's try to do that.

We bisect the triangle CAE to make CAD and DAE the two new equal segments of the 2N sided polygon.

Using Pythagoras's theorem on the right angled triangle ABC, we can see:

$$ \begin{align} AB^2 + BC^2 &= 1^2 = 1 \\ AB &= \sqrt{1 - BC^2} \\ \end{align} $$

Given that

$$ BC = \frac{d_n}{2} $$

Substituting gives

$$ \begin{align} AB &= \sqrt{1-\left(\frac{d_n}{2}\right)^2} \\ BD &= 1 - AB \\ &= 1 - \sqrt{1-\frac{d_n^2}{4}} \\ \end{align} $$

Using Pythagoras's theorem on the right angled triangle CBD

$$ \begin{align} CD^2 &= BC^2 + BD^2 \\ &= \left(\frac{d_n}{2}\right)^2 + \left(1 - \sqrt{1-\left(\frac{d_n}{2}\right)^2}\right)^2 \\ &= \frac{d_n^2}{4} + \left(1 - 2\sqrt{1-\frac{d_n^2}{4}} + \left(1 - \frac{d_n^2}{4}\right)\right) \\ &= 2 - 2\sqrt{1-\frac{d_n^2}{4}} \\ CD &= d_{2n} \\ &= \sqrt{2 - 2\sqrt{1-\frac{d_n^2}{4}}} \\ \end{align} $$

d_2n is the length of one side of the polygon with 2N sides.

This means that if we know the length of the sides of a polygon with N sides, then we can calculate the length of the sides of a polygon with 2N sides.

What does this mean? Lets start with a square. Inscribing a square in a circle looks like this, with the side length √2. This gives an estimate for π as 2√2, which is poor (at 2.828) but it is only the start of the process.

We can the calculate the side length of an octagon, from the side length of a square, and the side length of a 16-gon from an octagon etc.

It's almost time to break out python, but before we do so, we'll just note that to calculate d_2n from d_n, the first thing you do is calculate d² and at the very end you take the square root. This kind of suggests that it would be better to keep track of d² rather than d, which we do in the program below (pi_archimedes.py):

import math

def pi_archimedes(n):
 """
 Calculate n iterations of Archimedes PI recurrence relation
 """
 polygon_edge_length_squared = 2.0
 polygon_sides = 4
 for i in range(n):
 polygon_edge_length_squared = 2 - 2 * math.sqrt(1 - polygon_edge_length_squared / 4)
 polygon_sides *= 2
 return polygon_sides * math.sqrt(polygon_edge_length_squared) / 2

def main():
 """
 Try the series
 """
 for n in range(16):
 result = pi_archimedes(n)
 error = result - math.pi
 print("%8d iterations %.10f error %.10f" % (n, result, error))

if __name__ == "__main__":
 main()

If you run this, then it produces:

Iterations	Sides	Result	Error
0	4	2.8284271247	-0.3131655288
1	8	3.0614674589	-0.0801251947
2	16	3.1214451523	-0.0201475013
3	32	3.1365484905	-0.0050441630
4	64	3.1403311570	-0.0012614966
5	128	3.1412772509	-0.0003154027
6	256	3.1415138011	-0.0000788524
7	512	3.1415729404	-0.0000197132
8	1024	3.1415877253	-0.0000049283
9	2048	3.1415914215	-0.0000012321
10	4096	3.1415923456	-0.0000003080
11	8192	3.1415925765	-0.0000000770
12	16384	3.1415926335	-0.0000000201
13	32768	3.1415926548	0.0000000012
14	65536	3.1415926453	-0.0000000083
15	131072	3.1415926074	-0.0000000462

Hooray! We calculated π to 8 decimal places in only 13 iterations. Iteration 0 for the square shows up the expected 2.828 estimate for π. You can see after iteration 13 that the estimate of π starts getting worse. That is because we only calculated all our calculations to the limit of precision of python's floating point numbers (about 17 digits), and all those errors start adding up.

We can easily calculate more digits of π using Pythons excellent decimal module. This allows you to do arbitrary precision arithmetic on numbers. It isn't particularly quick, but it is built in and easy to use.

Let's calculate π to 100 decimal places now. That sounds like a significant milestone! (pi_archimedes_decimal.py):

from decimal import Decimal, getcontext

def pi_archimedes(n):
 """
 Calculate n iterations of Archimedes PI recurrence relation
 """
 polygon_edge_length_squared = Decimal(2)
 polygon_sides = 2
 for i in range(n):
 polygon_edge_length_squared = 2 - 2 * (1 - polygon_edge_length_squared / 4).sqrt()
 polygon_sides *= 2
 return polygon_sides * polygon_edge_length_squared.sqrt()

def main():
 """
 Try the series
 """
 places = 100
 old_result = None
 for n in range(10*places):
 # Do calculations with double precision
 getcontext().prec = 2*places
 result = pi_archimedes(n)
 # Print the result with single precision
 getcontext().prec = places
 result = +result # do the rounding on result
 print("%3d: %s" % (n, result))
 if result == old_result:
 break
 old_result = result

You'll see if you look at the pi_archimedes function that not a lot has changed. Instead of using the math.sqrt function we use the sqrt method of the Decimal object. The edge gets initialised to Decimal(2) rather than 2.0 but otherwise the methods are the same. The main method has changed a bit. You'll see that we set the precision of the decimal calculations using the getcontext().prec = ... call. This sets the precision for all following calculations. There are other ways to do this which you can see in the decimal module docs. We do the Archimedes calculations with 200 decimal places precision, then print the result out with 100 decimal places precision by changing the precision and using the result = +result trick. When the result stops changing we end, because that must be π!

If you run this, you'll find at iteration 168 it produces 100 accurately rounded decimal places of π. So far so good!

There are two downsides to this function though. One is that the decimal arithmetic is quite slow. On my computer it takes about 6 seconds to calculate 100 digits of π which sounds fast, but if you were to try for 1,000,000 places you would be waiting a very very long time! The other problem is the square root. Square roots are expensive operations, they take lots of multiplications and divisions and we need to do away with that to go faster.

In Part 3 we'll be getting back to the infinite series for arctan and on to much larger numbers of digits of π.

Pi - Gregory's Series

nick@craig-wood.com (Nick Craig-Wood) — Sun, 17 Jul 2011 00:00:00 +0000

Lets calculate π (or Pi if you prefer)! π is an irrational number (amongst other things) which means that it isn't one whole number divided by another whole number. In fact the digits of π are extremely random - if you didn't know they were the digits of π they would be perfectly random.

Fun with Maths and Python

This is a having fun with maths and python article. See the introduction for important information!

π has been calculated to billions of decimal places, but in this series of articles, we're going to aim for the more modest target of 100,000,000 decimal places calculated by a python program in a reasonable length of time. We are going to try some different formulae and algorithms following a roughly historical path.

You were probably taught at school that 22/7 is a reasonable approximation to π. Maybe you were even taught that 355/113 was a better one. You were unlikely to have been taught how π is actually calculated. You could draw a circle and measure the circumference, but that is very hard to do accurately.

Luckily maths comes to the rescue. There are 100s of formulae for calculating π, but the simplest one I know of is this:

$$ \frac{\pi}{4} = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \frac{1}{11} + \frac{1}{13} - \frac{1}{15} + \frac{1}{17} - \cdots $$

This is known as Gregory's Series (or sometimes the Leibnitz formula) for π. That ... stands for keep going forever. To make the next term in the series, alternate the sign, and add 2 to the denominator of the fraction, so + 1/19 is next then - 1/21 etc.

To see why this is true, we need to look at the Leibnitz formula for arctan (or tan⁻¹). This is:

$$ arctan(x) = x - \frac{x^3}{3} + \frac{x^5}{5} - \frac{x^7}{7} + \frac{x^9}{9} - \cdots ( -1 <= x <= 1 ) $$

Note that the result of arctan(x) is in radians. I'm not going to prove that formula here as it involves calculus, but you can see a proof of the arctan formula if you want.

arctan is the inverse of tan, so arctan(tan(x)) = x. tan(x) is the familiar (hopefully) ratio of the opposite and adjacent sides of a right angled triangle and how that relates to the angle.

If we set the angle to 45°, which is π/4 radians, then:

$$ \begin{align} tan(45°) &= 1 \\ ⇒ tan(\pi/4) &= 1 \\ ⇒ arctan(1) &= \pi/4 \\ \end{align} $$

which when substituted in the above becomes:

$$ \frac{\pi}{4} = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \cdots $$

This may be the simplest infinite series formula for π but it has the disadvantage that you'll need to add up a great number of terms to get an acceptably accurate answer.

Here is a bit of python to calculate it (pi_gregory.py):

import math

def pi_gregory(n):
 """
 Calculate n iterations of Gregory's series
 """
 result = 0
 divisor = 1
 sign = 1
 for i in range(n):
 result += sign / divisor
 divisor += 2
 sign = -sign
 return 4 * result

def main():
 """
 Try Gregory's series
 """
 for log_n in range(1,8):
 n = 10**log_n
 result = pi_gregory(n)
 error = result - math.pi
 print("%8d iterations %.10f error %.10f" % (n, result, error))

if __name__ == "__main__":
 main()

All the important stuff happens in pi_gregory and hopefully you'll see how that relates to the infinite series for arctan. Note the trick of keeping the sign of the term in a variable as 1 or -1 and then multiplying by this. This is a common trick, easier to write than an if statement and probably faster.

The program above prints:

Iterations	Result	Error
10	3.0418396189	-0.0997530347
100	3.1315929036	-0.0099997500
1000	3.1405926538	-0.0009999997
10000	3.1414926536	-0.0001000000
100000	3.1415826536	-0.0000100000
1000000	3.1415916536	-0.0000010000
10000000	3.1415925536	-0.0000001000

That looks like our old friend π doesn't it! However you can see that it takes 10 times as many iterations to add another decimal place of accuracy to the result. That isn't going to get us even 10 decimal places in a reasonable length of time... We'll have to get a bit cleverer. (If the program printed out something very different then you need to run it with python 3 not python 2 see the introduction.) You can also see something very odd in the error terms - the result is correct to 10 decimal places, all except for one digit. This is a known oddity and you can read up about it The Leibniz formula for pi .

Those of you paying attention will have noted that we used math.pi from the python standard library in the above program to calculate the difference of the result from π. So python already knows the value of π! However that is only a double precison value (17 digits or so) and we are aiming for much more. We are going to leave the world of double precision floating point behind, and calculate a lot more digits of π, much quicker, in the next exciting episode (Part 2)!

Fun with Maths and Python - Introduction

nick@craig-wood.com (Nick Craig-Wood) — Sat, 16 Jul 2011 00:00:00 +0000

This is going to be a series of articles which are all about having fun with maths and Python. I'm going to show you a bit of maths and then how to apply it with a python program, then work out how to improve it.

I decided to write these after looking in my doodles folder on my computer and finding 100s of programs to calculate this bit of maths, or explore that bit of maths, and I thought it would be interesting for me to share them. Hopefully you'll find it interesting too!

I'm going to use python 3 for all python code. This enables me to future proof these articles and encourage everyone who hasn't tried Python 3 to have a go. The programs will run with minor modifications on python 2 - just watch out for division which has changed from python 2 to 3. In fact if you run any of these programs under python 2 then put from __future__ import division at the top of them.

Python is a really excellent language for investigations into maths. It is easy to read and has excellent long number support with infinite precision integers and the decimal module. I'm going to attempt to demonstrate what I want to demonstrate not using any third party modules, but I may occasionally use the gmpy module.

I'm going to pitch the level of maths below calculus, no more than the maths that everyone should have learnt at school. If I want to use some bit of maths outside that range I'll explain it first. This does means that sometimes I won't be able to show a proof for things, but I'll add a link to an online proof if I can find one.

You are going to see maths written both in traditional style:

$$ x=\frac{-b \pm \sqrt {b^2-4ac}}{2a} $$

And in computer style which every programmer knows anyway. This involves a few more brackets, but it is much easier to translate to computer code. For example:

(-b +/- sqrt(b**2 - 4*a*c)) / (2*a)

In Python computer maths we write:

* to mean multiply
/ to mean divide
** to mean to the power
// to mean divide but give the result as an integer
% to mean the same as the modulo or mod operator, ie do the division and take the remainder.

I'm going to make free with unicode fonts so you'll see:

π - pi
x² - x squared
√x - square root of x
± - plus or minus

If these aren't appearing properly, then you'll need to try a different browser, or install some more fonts.

As for the programming level, anyone who knows a little bit of python should be able to work out what is happening - I won't be doing anything too fancy!

I'll tag all the articles with pymath so you can find them or subscribe to the rss.

Happy Calculating!

Nick Craig-Wood

nick@craig-wood.com

Technology changes in 2010

nick@craig-wood.com (Nick Craig-Wood) — Wed, 22 Dec 2010 14:56:39 +0000

Technology that has come and gone from my life in 2010, and new stuff I've learnt. More than I expected!

These notes are really for me to look back at over the years, but you might find them interesting.

Old

Things gone from my life. Gone but not forgotten :-

Mutt - finally replaced with Thunderbird. Mutt is a terminal based email client, just in case you haven't heard of it. It works really well for power users, is really configurable, but in these days of HTML emails and attachments it was no longer keeping up. I loved him - he was a faithful mutt, but he had to go. (Actually he is still there at the command line ready to go if needed - but keep it under your hat!)
Courier V34 modem - years overdue, I finally disconnected it and consigned it to the pit of oblivion. It used to receive faxes, but I don't think anyone has sent one for several years. It also used to run a BBS and an Internet node, which was useful at times, but now GPRS/3G/Wifi is everywhere I want to go it is no longer needed.
ISDN - 128k of digital goodness just wasn't good enough any more and became too expensive. Great in its hey-day with 2 channels, 10 numbers and a whopping 64k digitial Internet connection. Obsoleted by ADSL and VOIP.
Nokia E90 & Symbian - sorry... we've had good times, but I've found someone else
KDE - you worked really well until version 4 where the bugs and the slowness really kicked in.
Usenet - This was my first hint of online communities on the Internet in 1994. Unfortunately the cool kids have moved on, and so, reluctantly, have I!

New

New technology :-

Thunderbird - for people with complicated email lives the plugin system makes life bearable! Here are my top plugins
- Nostalgy - adds keyboard shortcuts to save, copy, goto folders
- Quickfolders - Put your favourite folders along the top
- Copy Sent to Current - copy sent mail to the current folder not the Sent folder - makes a new and better way of doing email
- Display Mail User Agent - snoop on what other people use for their mail client
- Enigmail - Use PGP to sign/encrypt your emails seamlessly
- External Editor - edit emails in your favourite editor (eg emacs)
- Identity Chooser - If you use multiple identities you need this!
- Mail Redirect - proper bounce of email (not forward)
IMAP - makes it easy to migrate your email setup. You can still use it with procmail so some things haven't changed ;-) I can use it with my Android phone and the excellent K9Mail.
Blogofile - which makes this (fully static) website. It replaces a pile of crufty hand built python scripts which did the same thing, just not nearly as well!
Ubuntu on the desktop - I finally gave in and replaced Debian testing with Ubuntu on my main laptop so I can spend less time fixing stuff and more time doing real work! Debian still rules on the server though.
VOIP - I've signed up with a 100% Voip provider and migrated my 10 telephone numbers to them. It went reasonably smoothly! I took the ISDN card out of my Asterisk PBX. We've been doing this at work for ages - it seemed like time to migrate at home too.
Android - is Google's new phone operating system. It runs on Linux and is mostly open source which are big positives in my view. I've been enjoying my Android phone greatly and have written a number of applications - Oxo3d is one.
Gnome - well not new actually, I last gave Gnome up in about 2005 when I got irritated at no longer being able to edit my menus. Unfortunately KDE 4 isn't shaping up for me, so it is hello again!

Books

Tech books I've read this year. No python books this year - I think I've read them all now!

Java in a Nutshell (again) programming to write Android programs. I learnt original Java ages ago, but is is quite a different experience with generic types. Still not my favourite language - reminds me too much of C++, but bearable.
Programming In Scala - just for fun - maybe I'll use it with Android one day. Scala is one of the new breed of languages which runs on the Java VM. Scala is a (mostly) functional language created by one of the creators of Java itself.
Version Control with Git - distributed revision control system. Very clever idea with lots and lots of command lines to explore. If it scales to the Linux kernel then surely it is good enough for my projects?
BGP: Building Reliable Networks with the Border Gateway Protocol - how the Internet really works!
Unlocking Android: A Developer's Guide - how to make Android applications
Running IPv6 - yes it is coming - IPv4 addresses are running out very soon.

These are affiliate links, so if you click then buy I'll earn a few pence towards my next tech book!