The Need for SPEAD

I've been absent for a good while now as a result of participating in a (successful) deployment of our PAPER experiment in South Africa. The Karoo desert in SA, where we were stationed, was very reminiscent of Rangely, CO where I grew up, except for the occasional baboon or kudu in the road. Though it came at a price of a lot of work piled up for me when I got back, and an awfully long time away from J, the isolation from all but our experiment helped ferment some ideas I'd been having about migrating the AIPY toolkit I've been developing to use a streaming data format that would avoid unnecessary disk accesses, would allow AIPY to be integrated directly with the correlators developed by our CASPER project, and would help our experiment develop a real-time analysis pipeline for compensating for ionospheric distortion in our data.

After chatting with a lot of guys working on the Karoo Array Telescope in Cape Town, we came up with a concrete protocol build on something already being used for CASPER correlator output. I just got done writing my first grant proposal to the NSF, funding a graduate student to work on this protocol--the Streaming Protocol for Exchanging Astronomical Data (SPEAD, pronounced "speed"). The process of writing a grant myself was a learning process, and helped me understand where a lot of the questions I got asked by my previous advisors were coming from.

A lesson I got to take away from SA was this: the reason we were in SA (as opposed to Australia) for PAPER was because we had been working with the KAT team, sharing correlator development. The reason we were working with the KAT team was because CASPER and KAT started up a collaboration a few years before. And that collaboration was started up because Dan Werthimer went down to visit SA some years ago to help advise them in a review of the design of their telescope electronics. Dan was invited there because he struck up a fast friendship with Alan Langman (the KAT director) at an earlier conference. The moral of this chain of causes and effects being that sometimes large projects go in new directions because of personal friendships, and sometimes those friendships end up making the difference in the success of a project.