Artificial Intelligence, Language Recognition, and Babies

As you may or may not know, I have son who was born 6 (almost 7) months ago. He is just the most incredible thing. Besides excelling in the normal metrics of cuteness and snugglability, he is also doing something that all babies do, but is probably the most incredible of all--he is is learning. He is learning how to move, how to recognize patterns, how to read expressions, how to form phonemes, how parse sounds. He is training the most incredible neural network on the face of the earth to solve problems that the best minds in the world have been working on for decades and haven't come close to solving. And he's making it look easy.

How is he doing it? Why can a baby, who knows nothing of the world, who lacks a fundamental understanding of physics, biology, optics, machine learning, linguistics, language, categorization, (the list goes on...) succeed at these tasks when very bright people using supercomputers cannot? I have some theories--some from an intro linguistics class I once took, some from personal experience trying to code speech recognition, some from learning foreign languages, some from my signal processing background, and some from watching this little ball of wonder over here. For what it's worth, here are my thoughts on the matter.

First, some observations:

1. The problems listed above (motor control, image/speech recognition, etc.) are HARD. Just because the human brain is extremely adept at solving them, let's not make the mistake of underestimating their complexity.
2. Babies don't come out with the answers. They pretty much can't do anything in the beginning. On the other hand...
3. Babies have a definite propensity for arriving at a solution. They may not consciously know what they are doing, but they have a pre-programmed "boot sequence" that gets them walking, talking, and causing trouble by age 2. This boot sequence is remarkably consistent between babies (no baby walks before babbling, etc.)
4. Babies have trainers (parents) who are instrumental in their development. However, babies work on problems of their choosing--a parent aids in language acquisition, for example, but cannot get a baby to start babbling before they come to it themselves. You can see this all the time when you watch kids. They have incredible attention spans for the skills they are working on, but things outside of that range are summarily ignored.
5. Babies do not reason their way to solutions. Reasoning comes later.
6. Changing gears... large neural networks don't work. Small neural networks are very good at discriminating between patterns on a few set of inputs, but one can't throw 1e4 pixels into a neural network and expect to train it to recognize any old picture of a cat.
7. A lot of skills that we think of as a single skill (say, speech recognition) are actually many interrelated skills. For example, it is certainly my experience in learning foreign languages that: a) without adequate vocabulary, I have trouble hearing the sounds that are being spoken, b) without an understanding of what a conversation is about at a high level, I have trouble knowing what words to expect, and c) without being able to hear the sounds that are being spoken, I have no clue what a conversation is about. I'm not just being silly here. There's a real, circular dependence to speech recognition that requires several skills to be developed in parallel (phoneme recognition, vocabulary, grammar, cultural expectations) in order to advance.
8. And finally, humans have an incredible knack for categorization--grouping things by common traits, and defining groups at all sorts of levels of generality.

I believe the above observations are only consist with the idea of a modular mind with a very strong hierarchy. In order to overcome the fact that large neural networks are untrainable, the brain has to be divided into modules that trained at particular sub-tasks that require fewer inputs. The mere fact that the brain is built of neurons and that these neurons only have several inputs suggests that this must be so.

Furthermore, the division of the brain into these modules must be pre-programmed. CAT scans reveal that the same physical locations in everyone's brain are responsible for certain functions, and it's clear that reason, cognition, and other general-purpose processing in our brain are not primarily responsible for language, image recognition, or motor control (although in adults, sometimes skills like language acquisition and motor control are augmented by reasoning and cognition). Humans have a natural language instinct, and capacity for image processing and motor control that belies an underlying, inherent cerebral architecture addressing these skills.

So what is the upshot of all of this? I think that work on artificial intelligence needs to reflect strong modularity and hierarchy. While designing hierarchical processing is not hard, training the kind of multi-tiered, cross-linked, sometimes circularly dependent system that AI requires is. Why do babies go through the same boot sequence? Do the stages of child development reflect the trained of different tiers of neural networks in the brain's hierarchy? It seems possible to me that the wonder of the human brain might more than this incredible hard-wired signal processing architecture--a fundamental component might be this incredible boot sequence, taking 10-20 years to complete, that trains neural networks ranging from simple movement and stimulus response to abstract thought and language.

Attiyah and His Theories

I guess when you're an astronomer, you have to expect to be the target of the occasional crackpot with their personal theory of the universe. My antagonist is Attiyah Zahdeh. I don't know where he's from (although devious research indicates he's on central time, so my current theory is Chicago), or how he got my email, but for a couple of years now I have been getting emails of which the following is the most recent example:

Attiyah's Planetary Motion I introduce this hypothesis in order to be discussed by scientists. I do not claim that I now have any mathematical proof or practical model to support Attiyah's Planetary Motion. I consider that the Kepler's second and third laws themselves support my hypothesis. It seems to me that Kepler failed to conclude that, relative to the Sun, the motion of the planets is the same as of the pendulum.Thus, he coined his second and third laws as alternative statements to express the laws of the simple harmonic motion of the planets. I'm inclined to say that Kepler (1571-1630 A.D.) was not aware of the work of Galileo (1564.1642 A.D.) on the pendulum and the laws of its motion he discovered. The hypothesis of Attiyah's Planetary Motion is four propositions: 1. The planets move not around the Sun but in front of it. 2. The planetary motion in front of the Sun is of the simple harmonic type. 3. The planet (the bob), gravitational force (the length, the line between the Earth's gravity center and the solar gravity center) and the Sun (the solar gravity center as the pivot point), altogether form a pendulum. 4. The planet oscillates in front of the Sun in a hemiellipse. Notes: 1. This hypothesis is an alternative of Kepler's first law only. 2. This hypothesis does not apply to the motion of the satellites.

I have also been spammed with Attiyah's Sun Theory, which I think says that daylight is caused by charged particles (or X-rays, or whatever) hitting our atmosphere--much the same as the mechanism causing the northern and southern lights, and with Attiyah's Hologeomagnetosphere, which asserts that the northern and southern lights are generated by electrical currents in the earth's molten core turning our ionosphere into a giant CRT monitor.

I once thought that these were created as jokes--that "Attiyah" was just the psuedonym of a humorist. Dozens of emails (and several years) later, I'm convinced that Attiyah is real and in earnest. In fact, I've discovered that he visits his theories upon astronomy message boards with some regularity, where he has revealed complete ignorance about how the scientific process works by demanding that others attempt to disprove his theories (the burden of evidence is on the newcoming theory) and by flatly ignoring the evidence that was provided against them. Two years ago, I myself was duped into providing a detailed refutation of Attiyah's Sun Theory, only to have my response disappear into the abyss of cyberspace.

But I'm not bitter. In some ways, Attiyah's doing a lot for science education by getting amateur scientists to review how we know what we know--reminding everyone that the reason we have such a widely adopted set of theories is that they are testably confirmed and mutually consistent. If only intelligent design, creationism, and young-earth hypotheses met with half the ridicule that Attiyah's theories meet on the message boards. Attiyah's only mistake, really, was failing to incorporate a little theology into the mix.

Playing, and Why The Fast Track Wasn't the Best Track

A NY Times article on playing today got me thinking about the indirect path I took to being an astronomer. I went to school at Harvard with a lot of very bright people (and even managed to marry one of them). The undergraduate academic experience at Harvard was a little hard on me, though it took me several years after I graduated to fully understand why. The first reason is pretty common to undergraduates at Harvard--intelligent and accomplished people who are used to being the best at what they do are suddenly brought into contact with quite a few people who are better than they are. For driven students, this blow to the ego can undercut some of the self-assurance necessary to work productively.

What was harder on me than turning in my "big fish" status as I moved to a larger pond was the cultural mismatch that existed between myself and the faculty with whom I came into contact. Harvard physics (or at least physics instruction) has a strongly theoretical bent to it, and while some modicum of application is maintained through the 2 lab courses we were required to take, I was always given the impression that applied fields were a cop-out for theoreticians who couldn't make the cut. The culture of disdain for experimentalists kept me on a theoretical track throughout college, long past the point at which I was "having fun". I can tell when I'm having fun, because I play. Playing, as defined in the article above, is "apparently purposeless activity." For me, that means trying to answer questions that aren't on the homework, just out of curiosity. It means starting projects, building things, and enjoying it. The farther I went down the theory track, I less I played with what I was learning. The undergraduate curriculum left little time for doing anything that wasn't strictly required, which was one problem, but the larger problem was that the path I was taking wasn't supporting the kind of playing I like to do.

I got lucky when I enrolled in an introductory electronics class with Paul Horowitz. I found myself, outside of class, trying to teach the computer I'd built to shoot a dart at a mechanical dinosaur. I modified a remote sensing, squawking penguin to spit water at passers-by. I didn't realize it at the time, but I was playing. After I graduated, somewhat at a loss for what to do, and burnt out with physics, I asked Paul if he knew anyone I could work for. He introduced me to Dan Werthimer at Berkeley, where I started designing and building electronics for SETI. Out of school, I suddenly had a lot more time for diversions, and I began learning Python and using it to write evolving programs that mutated their own source code. I tried writing speech recognition (I'll post someday about language acquisition, one of my favorite diversions). I made a guitar website to learn cgi programming. Most telling, I (mostly) gave up video games for computer programming, which indicates the degree to which this really was playing for me.

Eventually I stumbled into radio astronomy, where the physics that I learned (and really did love), met with the electronics and programming that I loved playing with. An incredible number of skills that I currently use were developed during my diversions, including Python programming, soldering, web programming, and signal processing. I never took classes in any of these things, I just learned them from my projects. What I didn't understand as an undergraduate was that working can really be "playing" if you find the right job, and that if you don't play with what you're doing, you might be barking up the wrong tree. Moreover, I was able to learn and accomplish much more when I was in a laboratory environment, playing with what I was learning, than in a classroom listening to lectures. Of course, you can't learn everything from playing--you need people to take you beyond what you have immediately at hand--but for me at least, I would rather this be the exception to the rule. Playing shouldn't just be for kids.

Rain Water

I consider myself an environmentalist. I try to do the basic things around the house that help reduce my carbon footprint (you can calculate yours here) like installing compact florescent lights, installing power strips to prevent appliances like TVs from pulling current while "off", driving the minimum possible, eating vegetarian, and drinking soy milk instead of the dairy variety (cows are surprisingly bad for the environment). And I'm always on the look-out for something crazy and interesting to try for the environment. An idea I've been tossing around in my head (especially now that I live in rainy Puerto Rico) is harvesting power from rainwater. Part of my recent interest in rainwater, I must admit, has to do with the fact that we average about one water-outage per month here; I want backup water. However, my original interest, a year or two ago, was actually in generating electricity from the rain falling on my roof.

Here are some order-of-magnitude calculations for what I might expect to get from this. Quick research indicated that average annual rainfall here is 62 inches (about 150 cm). This means that each square meter of rooftop collects about:

150 (cm) x 100 (cm) x 100 (cm) x 1 (g/cm^3) / 1000 (g/kg) = 1500 kg

of water per year. If a story of a building is 5 m high, then the potential energy in that water is:

Force x Distance = Mass x Gravity x Distance = 1500 (kg) x 10 (m/s^2) x 5 (m) = 75,000 (J) = .02 (kWhr)

So PR generates .02 (kWhr/m^2/story) annually. My apartment building is about 10 (m) x 20 (m), and is 4 stories high, bring the energy of rainfall to 16 kWhr annually. If electricity costs $0.10 per kWhr, I could save a whopping $1.60 off my power bill annually. That probably won't ever offset what it would take to build the generator. Sigh.

Of course, dams do exactly the above, but with an enormous collecting area (not just a rooftop--an entire drainage basin). It looks like rain on my roof isn't going to power my computer, though. But I still might try to use the water to flush my toilet when the water goes off next weekend.

Hidden Variables

I've been having a few conversations about hidden variables lately, so I thought I would post about it. First a little background:

Quantum mechanics (QM), as we know it, is weird. It is a classic example of science as a selection process, as I talked about in the previous post. It set about to solve the problem of predicting the locations, energies, and other microscopic attributes of fundamental particles. When the dust settled, we had one of the most accurate theories ever made (it predicts the mass of the electron to 14 decimal places), but the model used to make these predictions countermanded a lot of things we'd thought were true but never actually got around to testing--intuitive things like "a particle can only be one place at a time", well-established things like "no particle may carry information faster than the speed of light" (which is true, but can be violated over short distances), and most importantly for this post "if you had enough information, you could determine the outcome of any experiment." Nature's rejection of this last idea comes dangerously close to undermining the pillar of scientific philosophy that the universe is predictable insofar as it can be modeled and tested, and it offended a lot of scientists (including Einstein).

What QM says is that there are certain pieces of information that are not simultaneously knowable. If you know a particle's position perfectly accurately, then it is impossible to know its momentum. If you know a particle's orientation along one axis, you can't know it along any other axis. For a long time, many people (including Einstein) thought that this was a shortcoming of QM--that these particles have "actual, hidden values" that QM just didn't know how to predict, and eventually there would be a better theory that could tell us what these values are. Those hopes were shattered in 1964 when John S. Bell proved that there can exist no hidden variables in a way that is compatible with QM. His predictions have been validated by experiment, showing that the reason we don't know the state of a particle is because the universe hasn't made up its mind (excuse the anthropomorphification) until you measure the particle. Crazy.

I like to think of QM like a black curtain at a magic show that allows the universe (the magician) to perform all sorts of sleight-of-hand shielded from the eyes of the audience. On our side of the curtain, there are rules you have to follow--conservation of energy, speed of light, definity of location and state, etc. On the other side of the curtain, anything can happen, so long as when you pull it back out (when we make a measurement), the rules have been obeyed. The "curtain" idea isn't so far-fetched; it's an analogy to Feynman's well-tested theory that the outcome of an interaction is the sum over all possible interaction pathways. The universe takes advantage of this curtain to do things that we think should be impossible, like transmitting information about a measurement instantaneously between two particles that share a quantum state. Furthermore, the universe relies on the fact that we can never see behind the curtain (this is an interpretation of Bell's theorem) because if we could, we could use that machinery to transmit our own information faster than the speed of light, and that violates causality. Causality, by the way, is another principle that we cling to because it seems self-evident, but may in fact be wrong. It will take a unification of the theories of QM and general relativity to sort that out.

Shifting gears into philosophy, I was talking with my uncle, who came to visit last week, about how people look for meaning in their lives. His argument was that back in human history, when the universe seemed a jumble of arbitrary events, a physical, all-powerful deity with direct control over all that happened was a powerful metaphor for finding meaning in the events of ones life. However, as scientific knowledge has gradually encroached on the idea that a god can take direct physical action in the universe, religion has had to respond to the sense that science is pushing away meaning in life. My uncle's thoughts were that a physical deity is becoming an outdated way of looking for meaning in life, and that we need to think about a more spiritual, humanistic God. I agree with this philosophy, but I wonder if the rejection of the hidden variable hypothesis (the magician's curtain) provides a home for religions that require a physical deity.