Here are some streaming comments as I'm listening to results of the 2010 Decadal Survey. A shortlink to the report is here.
11:10 am EDT: Excellent placement of "what were the first luminous objects, and when did they form" second from the top of "Major questions to address this decade".
11:20 am EDT: Ooh, even better. "Cosmic Dawn" is the first of the 3 over-arching fields. Good fielding for EoR as a top science priority in the next decade!
11:23 am EDT: Now we're starting into descriptions of the panels. I was involved in several of the RMS (Radio, Millimeter, and Submillimeter) submissions, and am looking for HERA (the Hydrogen Epoch of Reionization Array). I'm also rooting for the Allen Telescope Array's Radio Sky Surveys Project. Finally, I'm hoping for some mention in the TEC (Technology development) program of prioritizing the development of shared solutions to digital signal processing hardware and libraries, which was the recommendation of a white paper I drafted with the help of the CASPER community.
11:45 am EDT: Mention of SKA as a priority for Radio Astronomy, unsurprisingly.
11:47 am EDT: Looks like Roger's wrapping up here. Turning to Q&A.
Meanwhile, I'm reading through the report. I see radio instrumentation is listed as a funding priority on ES-4, linking to 7-39. A good sign.
A painful line on 1-18: "U.S. participation in projects such as the Square Kilometer Array is possible only if there is either a significant increase in NSF-AST funding or continuing closure of additional unique and highly productive facilities." Ouch.
But on 2-12, my own sky map (well, with "permissions pending" for now). Now that's something!
12:01 pm EDT: An interesting question. "Why such a priority on habitable planets in the decadal review?" Sounds like the answer is that it's particularly primed to make big breakthroughs. I think I agree with that. I wasn't surprised that it was on there. There's some rumors going around that Kepler has found earth-like planets in earth-like orbits.
12:02 pm EDT: What about the surplus of post-docs relative to faculty positions? Answer: there are a wide range of careers and positions available to astronomers, so it's not unreasonable to have a larger number of post-doc positions where budding astronomers get training. I'm not sure that answer fully appreciates the scale of the problem.
Continuing with reading, on page 3-13, "The HERA program, a project that was highly ranked by the RMS-PPP and included by the committee in its list of compelling cases for a competed mid-scale program at NSF, provides a development pathway for the SKA-low facility. Progress on development of the SKA-mid pathfinder instruments, the Allen Telescope Array in the U.S., the MeerKAT in South Africa and the ASKAP in Australia, and in new instruments and new observing modes on the existing facilities ... will provide crucial insight into the optimal path towards a full SKA-mid." That's good to see mentioned. Sounds like it lays the groundwork for a strong future proposal to get funded. It's not a promise of funding, though. Not that such a promise was expected.
12:11 pm EDT: Second use of "tripwires" for projects. A very colorful phrase.
Interesting plot on 4-15: papers in all astronomy fields are increasing. Instrumentation papers seem to be low in number, but holding their own against other fields (same percentage contribution to total paper number).
On 5-14 for Data Reduction and Analysis Software: "Flexibility, openness, and platform independence, modularity, and public dissemination are essential to this effort. Focused investment in a series of small-scale initiatives for common tool development ... may be the most cost-effective approach, although there are undoubtedly synergies with the pipeline development needed for the large-scale projects." Sounds helpful for some of my projects like AIPY, SPEAD, and CASPER.
12:18 pm EDT: Comments on the SKA? Answer: SKA is the future, but the US can't pay for the construction on the proposed time scale (but slower might be ok). Technology development should be prioritized though. Low-frequency SKA, though, targets EoR, and we're interested in projects targeting that. Yes!
On 5-21 for Technology Development: "The committee received community input in the form of white papers on the funding needs for technology development in areas such as ... high speed, large N correlators. In these areas and others, researchers ... had come together to plan a coherent strategy for the decade. The OIR and RMS panels made a convincing case that the current level of ATI funding needs to be augmented in order to successfully pursue these highly-ranked technology development programs and roadmaps." Looks like my white paper fell on receptive ears.
12:28 pm EDT: Neil Tyson is closing down Q&A. He is one cool dude. I'm glad he was on the panel.
On 7-7 for Science Objectives for the Decade: "Find and explore the epoch of reionization using hydrogen line observations starting with the HERA telescopes that are already under construction." Wham. And in Table 7.1 on 7-32, Priority 2, Projects thought compelling: HERA.
On B-2 for Program Priorization, in Table B.1, I see both ATA and HERA. I'd say ATA didn't necessarily win big in the review, but at least they're there.
And finally, in appendix D-1: "Hydrogen Epoch of Reionization Array ... is a multi- stage project in radio astronomy to understand how hydrogen is ionized after the first stars start to shine. The first phase (HERA I) is under way and will demonstrate the feasibility of the technical approach. The second phase (HERA II) would serve as a pathfinder for an eventual world-wide effort in the following decade to construct a facility with a total collecting area of a square kilometer and the power to make detailed maps of this critical epoch in the history of the universe. Proceeding with HERA II should be subject to HERA I meeting stringent performance requirements in its ability to achieve system calibration and the removal of cosmic foreground emission." We've got our work cut out for us!
Showing posts with label software. Show all posts
Showing posts with label software. Show all posts
Friday, August 13, 2010
Wednesday, February 10, 2010
AstroBaki on MediaWiki
I just started up a new wiki called AstroBaki. The main reason I did this was that my MoinMoin AIPY wiki was clunky to use and was getting spammed lots. I switched to the MediaWiki engine, which has better automated control over these kinds of things. As an added bonus, MediaWiki has support for latex math. This got me thinking...
When I started grad school, I had a hard time transitioning from feeling like I was producing and contributing (I was working as a development engineer for SETI) to just absorbing knowledge. To make myself feel better and more invested in learning, I started doing something for which I became moderately famous around the department: latexing lecture notes on-the-fly. For full disclosure, I should mention that I copycatted the idea of latexing on-the-fly from my friend Phil.
The key to success is to use lots of "defs", and to recognize when you need to def a sequence of commands. When the same sequence of symbols started popping up, I would pretend that I had already def'd the command and start using it, and when there was a pause in the derivation, I would remember to scribble down what that command should mean. In my later years, I also started drawing figures in paint for inclusion in latex.
Anyway, I now have about 4 or 5 latex'd class notes that I have put on my website. From what I hear, they are still regularly used in UCB classes, and I occasionally get happy emails from grad students thanking me for the effort. Meanwhile, I've been reading a book about Nicolas Bourbaki, a famous pseudonym for a group of (mostly French) mathematicians who collaboratively re-wrote mathematics from 1935 to the 70s. Nicolas Bourbaki was a wiki, ahead of its time.
"Now wouldn't it be cool," I thought to myself, "if students using these lecture notes could fix them when they are wrong (after all, they were written on-the-fly), and re-organize them to make more sense?" Could these notes become a sort of open-source textbook for astronomy? So AstroBaki was born.
The difficulty, I am finding, is in translating latex (especially latex heavy in defs) into mediawiki. The best tool I've found so far has been pandoc, which didn't do the defs, but did everything else pretty well. I'm loath to do things by hand, so I'll see what can be automated, and I'll keep you posted.
When I started grad school, I had a hard time transitioning from feeling like I was producing and contributing (I was working as a development engineer for SETI) to just absorbing knowledge. To make myself feel better and more invested in learning, I started doing something for which I became moderately famous around the department: latexing lecture notes on-the-fly. For full disclosure, I should mention that I copycatted the idea of latexing on-the-fly from my friend Phil.
The key to success is to use lots of "defs", and to recognize when you need to def a sequence of commands. When the same sequence of symbols started popping up, I would pretend that I had already def'd the command and start using it, and when there was a pause in the derivation, I would remember to scribble down what that command should mean. In my later years, I also started drawing figures in paint for inclusion in latex.
Anyway, I now have about 4 or 5 latex'd class notes that I have put on my website. From what I hear, they are still regularly used in UCB classes, and I occasionally get happy emails from grad students thanking me for the effort. Meanwhile, I've been reading a book about Nicolas Bourbaki, a famous pseudonym for a group of (mostly French) mathematicians who collaboratively re-wrote mathematics from 1935 to the 70s. Nicolas Bourbaki was a wiki, ahead of its time.
"Now wouldn't it be cool," I thought to myself, "if students using these lecture notes could fix them when they are wrong (after all, they were written on-the-fly), and re-organize them to make more sense?" Could these notes become a sort of open-source textbook for astronomy? So AstroBaki was born.
The difficulty, I am finding, is in translating latex (especially latex heavy in defs) into mediawiki. The best tool I've found so far has been pandoc, which didn't do the defs, but did everything else pretty well. I'm loath to do things by hand, so I'll see what can be automated, and I'll keep you posted.
Friday, January 22, 2010
Where is GCC for FPGAs?
A lot of the digital signal processing that gets done in radio astronomy these days is done on Field Programmable Gate Arrays (FPGAs), and one of the projects I've been working on from the beginning in my research is developing open-source libraries for programming these chips. My part in this has generally been on the algorithmic/mathematical side: writing FFTs, filters, cross-correlation engines, etc. Another key aspect of this work, though, is a toolflow that allows people to design systems at a high level with parameterized algorithmic cores, and to turn that design into the wiring instructions that tell the FPGA how to implement the system.
We currently use a design entry system based on Simulink running on Matlab, and while it is an extremely powerful environment, we've also found it to be limiting, frustrating, and hard to maintain designs in. In October, I volunteered at an international workshop on astronomy signal processing to explore alternatives to this environment. My current favorite is MyHDL, which uses Python to generate lower-level code in Verilog or VHDL, and I may start looking more deeply into porting a design to use MyHDL.
Something that is bothering me, though, is that however much we work on porting our toolflow open-source equivalents, there is currently no open-source compiler for FPGAs. The state of affairs in FPGA-land is something like PCs in the '70s, when every personal computer had its own specialized compiler. For PCs, the problem was solved by GCC (the Gnu Compiler Collection), which became the default open-source solution for compiling most languages to target the many CPU architectures that exist in the world today.
I'm keeping my eye on gEDA, and notably Icarus, which seems to be a free synthesis tool (synthesis, mapping, and routing are the 3 main stages of compiling for an FPGA). Perhaps mapping and routing can never be open-source, since they tend to be very chip-specific. But here's hoping...
We currently use a design entry system based on Simulink running on Matlab, and while it is an extremely powerful environment, we've also found it to be limiting, frustrating, and hard to maintain designs in. In October, I volunteered at an international workshop on astronomy signal processing to explore alternatives to this environment. My current favorite is MyHDL, which uses Python to generate lower-level code in Verilog or VHDL, and I may start looking more deeply into porting a design to use MyHDL.
Something that is bothering me, though, is that however much we work on porting our toolflow open-source equivalents, there is currently no open-source compiler for FPGAs. The state of affairs in FPGA-land is something like PCs in the '70s, when every personal computer had its own specialized compiler. For PCs, the problem was solved by GCC (the Gnu Compiler Collection), which became the default open-source solution for compiling most languages to target the many CPU architectures that exist in the world today.
I'm keeping my eye on gEDA, and notably Icarus, which seems to be a free synthesis tool (synthesis, mapping, and routing are the 3 main stages of compiling for an FPGA). Perhaps mapping and routing can never be open-source, since they tend to be very chip-specific. But here's hoping...
Friday, January 8, 2010
Hands-On Cosmology Education
Yesterday I spent the morning giving a gosh-wow talk about cosmology to a physics class at Athenian High School taught by my housemate Dave Otten. It was a lot of fun, and the students were all very enthusiastic. It was almost entirely driven by their questions, and they loved being pitched curveballs (time is reference-frame dependent, the universe is expanding, spiral arms are standing waves, etc). The hour-and-a-half lecture was over before we knew it.
Afterward, Dave mentioned that it would be really cool if there were a way to talk about galactic-scale astronomy and cosmology that was in keeping with the philosophy of their school, which emphasizes lab-based, hands-on learning. He mentioned that PhET is a free resource he uses for providing interactive simulations that make hands-on labs out of subjects that otherwise would be too slow, small, big, fast, or dangerous to perform live in a classroom. He also lamented that there aren't any galactic- or cosmological-scale simulators there that could help to understand how systems on this scale behave, and that could perhaps illustrate exactly where the problems of dark matter and dark energy are encountered. Has anyone seen something like this?
Afterward, Dave mentioned that it would be really cool if there were a way to talk about galactic-scale astronomy and cosmology that was in keeping with the philosophy of their school, which emphasizes lab-based, hands-on learning. He mentioned that PhET is a free resource he uses for providing interactive simulations that make hands-on labs out of subjects that otherwise would be too slow, small, big, fast, or dangerous to perform live in a classroom. He also lamented that there aren't any galactic- or cosmological-scale simulators there that could help to understand how systems on this scale behave, and that could perhaps illustrate exactly where the problems of dark matter and dark energy are encountered. Has anyone seen something like this?
Monday, April 20, 2009
Interferometry File Formats
When astronomers talk about software done right, they often hold up FITS as the gold standard. I'll admit, FITS has done more to live up to its namesake (Flexible Image Transport System) than many believed possible. But unfortunately, there can be too much of a good thing. Sometimes too much emphasis is put on defining an end-all-be-all file format, when all we really need are good tools for converting between file formats.
Data formats are a problem in radio astronomy software. Currently, there are at least three major formats (MIRIAD, UVFITS, and MeasurementSets), each linked to a major software package (MIRIAD, AIPS, and CASA), with rudimentary/non-existant tools for converting between them. Many have taken this current state of affairs as a sign that multiple file formats are bad and that the community should decide on a single format. Since each format is intimately tied to major software package, this battle over file formats has escalated to a war between software packages.
The mistake made here was blaming the file formats. File formats are not the problem. The problem is that the software for reading them has not been circulated in easily accessible modules. I am encountering this problem as I'm trying to get AIPY to be agnostic about file formats by wrapping them all into Python. Here's where I am:
The MIRIAD file format was actually easily wrapped up, owing to MIRIAD having a developed programmer's API.
MeasurementSets (with CASA) are giving me a lot more trouble. It seems that CASA, with all of it's C++ objects that are passed between functions, is something of an "all or nothing" deal. If I want to read a MeasurementSet, I apparently need to wrap up the entirety of CASA. The failing here is code modularity.
UVFITS is giving me the opposite problem.
UVFITS was cooked up as a FITS-conforming file format to handle raw interferometric data. Unfortunately, interferometric data isn't in picture form yet, so an extension of the FITS format (the binary table) was cooked up to accommodate that (Cotton et al. 1995). The result was a file format that is so general that it does not tell the programmer what the data actually means.
File formats exist to support the needs of different applications. They've been created out of need, and should not be dismissed as unnecessary. I recommend to the radio astronomy software community that we embrace these file formats and work on modular code so that they are accessible from any software package.
Data formats are a problem in radio astronomy software. Currently, there are at least three major formats (MIRIAD, UVFITS, and MeasurementSets), each linked to a major software package (MIRIAD, AIPS, and CASA), with rudimentary/non-existant tools for converting between them. Many have taken this current state of affairs as a sign that multiple file formats are bad and that the community should decide on a single format. Since each format is intimately tied to major software package, this battle over file formats has escalated to a war between software packages.
The mistake made here was blaming the file formats. File formats are not the problem. The problem is that the software for reading them has not been circulated in easily accessible modules. I am encountering this problem as I'm trying to get AIPY to be agnostic about file formats by wrapping them all into Python. Here's where I am:
The MIRIAD file format was actually easily wrapped up, owing to MIRIAD having a developed programmer's API.
MeasurementSets (with CASA) are giving me a lot more trouble. It seems that CASA, with all of it's C++ objects that are passed between functions, is something of an "all or nothing" deal. If I want to read a MeasurementSet, I apparently need to wrap up the entirety of CASA. The failing here is code modularity.
UVFITS is giving me the opposite problem.
UVFITS was cooked up as a FITS-conforming file format to handle raw interferometric data. Unfortunately, interferometric data isn't in picture form yet, so an extension of the FITS format (the binary table) was cooked up to accommodate that (Cotton et al. 1995). The result was a file format that is so general that it does not tell the programmer what the data actually means.
File formats exist to support the needs of different applications. They've been created out of need, and should not be dismissed as unnecessary. I recommend to the radio astronomy software community that we embrace these file formats and work on modular code so that they are accessible from any software package.
Subscribe to:
Posts (Atom)