The Voynich Manuscript: Bootstrapping Language

The internet is a powerful and dangerous thing. It all started when I read the latest xkcd, which warned me that visiting cracked.com was dangerous. Then I read about the "6 phenomena that science can't explain" (which was a very dramatic title for some underwhelming mysteries), and next thing I knew, I was reading about the Voynich Manuscript. I learned about cryptography, glossolalia, the Manchu language, among other things. Then I took a look at the manuscript and before I knew it, I had a transcribed version of the manuscript in electronic form using the European Voynich Alphabet. And it just went downhill from there.

To summarize, the Voynich Manuscript (hereafter VMS) is a handwritten text with some illustrations some 500 years old. It uses glyphs no one knows how to read, it is not clear if it corresponds to a known language, it may or may not be encrypted, and little progress has been made in deciphering any of it, despite the fact that some bright people have tried. So I decided to have a crack at it.

The reason I got interested is because of the similaries to SETI. Arecibo, back in 1974, transmitted a message off into space that had been designed to be decrypted. We might receive a message like that some day. Or we might intercept something much like the VMS--a bunch of data in a language that we have no prior knowledge of--and we may be finding ourself trying to figure out how to bootstrap a language. That is to say, to learn the grammar and semantics of a language from a static example, without outside help.

Is this possible? For grammar, I'm pretty sure of it. I can imagine an algorithm (maybe Maximum Entropy Modeling and Bayesian learning applied to grouping and parsing) that uses correlations in the appearances of language elements (starting with letters and building up) and correlations in the behaviors of these elements relative to one another to build a model for parsing a language. For the VMS, I used something similar to this (not the MEM and Bayesian part) to show that spaces, line breaks, and paragraph breaks show similar grouping correlations relative to other VMS letters, and so can probably be considered one grammatical element of whitespace. That's a pretty simple thing to deduce, but it was actually something I was worried about in getting started with the VMS.

Sematics is another issue. Once upon a time, I would have had an optomistic answer to bootstrapping sematics from text that was not written for that purpose. However, after watching my child mysteriously acquire language, illustrating how hard-wired the human brain is for learning language from another human, and how much it relies on shared experience and feedback, I'm less sure.

I would be interested to know if there is a field of mathematics that studies sematics and the properties that a self-contained system needs to have to be able to generate sematical relationships. The Arecibo message relied on a shared physical environment to try to bootstrap sematics. I wonder if it would be enough to describe the rules of the grammar of a language in the language itself. That one, once the reader had deduced the relationships between elements, you would have a shared knowledge of that subject that might enable a reader to correlate the structure of the descriptions with the grammatical structure and thereby establish the first sematical relationships.

Anyway, after preliminary analysis of the VMS, I'm pretty sure that it's not random gibberish (there are correlations between elements on levels ranging from letters to words), and if it's encrypted, it's a weak form of encryption that preserves these correlations. My pet theory, extended from the glossalalia idea, is that this is actually plaintext in a natural language with an invented set of symbols, but that the natural language might be the accidental or intentional creation of a savant or scholar.

Playing, and Why The Fast Track Wasn't the Best Track

A NY Times article on playing today got me thinking about the indirect path I took to being an astronomer. I went to school at Harvard with a lot of very bright people (and even managed to marry one of them). The undergraduate academic experience at Harvard was a little hard on me, though it took me several years after I graduated to fully understand why. The first reason is pretty common to undergraduates at Harvard--intelligent and accomplished people who are used to being the best at what they do are suddenly brought into contact with quite a few people who are better than they are. For driven students, this blow to the ego can undercut some of the self-assurance necessary to work productively.

What was harder on me than turning in my "big fish" status as I moved to a larger pond was the cultural mismatch that existed between myself and the faculty with whom I came into contact. Harvard physics (or at least physics instruction) has a strongly theoretical bent to it, and while some modicum of application is maintained through the 2 lab courses we were required to take, I was always given the impression that applied fields were a cop-out for theoreticians who couldn't make the cut. The culture of disdain for experimentalists kept me on a theoretical track throughout college, long past the point at which I was "having fun". I can tell when I'm having fun, because I play. Playing, as defined in the article above, is "apparently purposeless activity." For me, that means trying to answer questions that aren't on the homework, just out of curiosity. It means starting projects, building things, and enjoying it. The farther I went down the theory track, I less I played with what I was learning. The undergraduate curriculum left little time for doing anything that wasn't strictly required, which was one problem, but the larger problem was that the path I was taking wasn't supporting the kind of playing I like to do.

I got lucky when I enrolled in an introductory electronics class with Paul Horowitz. I found myself, outside of class, trying to teach the computer I'd built to shoot a dart at a mechanical dinosaur. I modified a remote sensing, squawking penguin to spit water at passers-by. I didn't realize it at the time, but I was playing. After I graduated, somewhat at a loss for what to do, and burnt out with physics, I asked Paul if he knew anyone I could work for. He introduced me to Dan Werthimer at Berkeley, where I started designing and building electronics for SETI. Out of school, I suddenly had a lot more time for diversions, and I began learning Python and using it to write evolving programs that mutated their own source code. I tried writing speech recognition (I'll post someday about language acquisition, one of my favorite diversions). I made a guitar website to learn cgi programming. Most telling, I (mostly) gave up video games for computer programming, which indicates the degree to which this really was playing for me.

Eventually I stumbled into radio astronomy, where the physics that I learned (and really did love), met with the electronics and programming that I loved playing with. An incredible number of skills that I currently use were developed during my diversions, including Python programming, soldering, web programming, and signal processing. I never took classes in any of these things, I just learned them from my projects. What I didn't understand as an undergraduate was that working can really be "playing" if you find the right job, and that if you don't play with what you're doing, you might be barking up the wrong tree. Moreover, I was able to learn and accomplish much more when I was in a laboratory environment, playing with what I was learning, than in a classroom listening to lectures. Of course, you can't learn everything from playing--you need people to take you beyond what you have immediately at hand--but for me at least, I would rather this be the exception to the rule. Playing shouldn't just be for kids.