Learning Rust: Some Thoughts

Back in September I wrote a post about some utilities written in the Rust language, and mentioned I was toying with the idea of trying to learn a bit of it. With two weeks of vacation around Christmas, I decided to take the plunge and have been reading and working through the examples in the Rust Book off and on as I get a chance.

I was a bit tentative going in, but I find I'm really enjoying it. Prior to this, for reference, I taught myself Python a decade ago while working as an undergraduate research assistant and have dabbled a tiny bit in JavaScript and Lua, but that's the extent of my programming language coverage: I'm still essentially monolingual. I managed to get through my undergraduate time without ever taking any sort of computer science class, and it's left me a little self-conscious of my self-taught status when it comes to Python. Rust is a very different language to Python, and I was worried that perhaps I wouldn't be able to pick up a new language as easily as I could ten years ago if I've unconsciously generalized Python-specific quirks to programming languages as a whole.

However, contrary to my worries I've been finding it an interesting and fun experience. I suspect my experience in studying various far-flung human languages may be helping, as it may be helping me to generalize better between different programming languages. And Python and Rust are opposites in some rather key ways: Python is an interpreted language, which means that programs are (and need to be) compiled and interpreted by an interpreter program at run-time. Rust on the other hand is a compiled language, which means it needs to be compiled before running (but can be run afterwards without needing any external program). Python is a dynamically-typed language where variables can be created with ease as needed and converted to different types without oversight by the language. Rust is statically-typed, with a very strict enforcement of variable types all working at compile time. These are not minor differences, but almost diametrically opposed paradigms, like the difference between a case-based language and one which relies on word order.

One of the major unique features of Rust is its concept of ownership. In essence, this is a requirement (checked and enforced at compile time) that only one “part” of a program can change a variable's value at a given time. A value can be “borrowed” any number of times for use as long as it is not changed, and the system responsible for enforcing this is known as the borrow checker. Colloquially people joke about spending much of their time learning Rust “fighting the borrow checker,” as the concept of ownership is a novel one and wrapping one's head around it takes some time.

Thankfully, Rust has some really good error messages. I've managed to write a few small programs on my own so far, and the error message output usually contains both an explanation of what I've done wrong, and a suggestion for how to fix it. In fact it's generally gone so well so far that I'm a bit suspicious; when is the other shoe going to drop? I've had errors, sure, but I've been able to figure them out quickly and get what I want to happen (within my still-limited understanding of the language as a whole). Granted, I'm not exactly writing complicated programs, just simple ones to find prime numbers or convert temperatures between scales, but still; it's been a pretty pleasant learning experience so far. I've most certainly got a lot to learn left, but it's fun to be seriously learning a new programming language again with no stress about the outcome.

I don't know if this will ever be useful in a job down the line (although a number of large companies are starting to use Rust for its ability to remove even the possibility of whole classes of costly memory errors found in languages like C or C++ due to ownership), but even if not I'm sure the experience of learning a new—and very different—language will have benefits for my Python knowledge, in the same way learning other languages has helped me reason about English*. And you never know, it just might come in handy down the line somehow. A hui hou!

*Interestingly, the upcoming Python 3.10 is getting a “match” system for comparing multiple cases which is extremely similar to the one found in Rust, so it might prove to be more useful than I'd initially thought!

Celebrating Ten…Er, Eleven Years of Daniel's Musings!

Today marks the eleventh anniversary of this blog. I know, I too am amazed—that I didn't think to make this post a year ago at the ten-year anniversay. To tell the truth, I actually wrote most of this post thinking it was the tenth anniversary. (And they're letting me do a PhD in astrophysics…) Anyway, a decade just over a decade ago when I started this blog, I was one semester into getting a bachelor's degree at UH Hilo, and now, I'm (hopefully!) just a few months away from finishing a PhD at Swinburne.

I originally started this blog as a way to replace email for letting a few people back home know how I was doing. Over the years it's morphed into something like a public journal, where I can show off things I've made or done, or talk about things that interest me for later reference. Some facts about things since this blog was started:

1. I graduated with dual bachelor's degrees in physics (with a minor in mathematics) and astronomy.
2. I've worked five different jobs (while my job description and day-to-day work with the JCMT remained exactly the same during the handover between owners, it was still technically two different jobs with different employers), including ones at high altitude on the two tallest volcanoes in the world.
3. I started a PhD in astronomy, and am now just a few scant months away from completing it.
4. I learned Python, my first programming language, and have been using it for…let's say about seven or eight years in total, to various degrees. (And last year I started seriously trying to learn Rust, which I might have a post on sooner or later.)
5. I picked up a few hobbies:
• Astrophotrography
• Video editing/videography
• Knitting (and re-started crochet after learning it a few years prior to beginning this blog)
• Painting
• Digital music engraving
6. I created what is still, to date, the only video Let's Play of Dodge That Anvil!.
7. And I've written a total of 668 posts for this blog, for an average of one every 5.47 days (though this is obviously skewed by the earlier years, as we shall see).

In some ways I'm impressed that this blog is still going. My interests are often somewhat…mercurial, shall we say, and tend to shift on the scale of a few years. I can never tell if some intense new interest will turn out to be a passing fancy of a few months or years' duration, or an enduring passion still going strong a decade later. As other interests of mine have fallen by the wayside over the past eleven years, this blog is still going, even if my output waxes and wanes over time. Originally I tried to write at least once a week; a few years ago, under the time pressures of full-time employment, I settled on just trying to have at least two posts a month (and even then I've fallen short a month or two!). I do still hold to the rule I originally established of “no more than one post per day,” though that hasn't been too difficult as I can either merge things together into one post, or have material for multiple posts. I put that in place to remind myself that this isn't social media (though it's perhaps the closest thing I have to it), and that I want posts that are worth reading rather than throw-away thoughts. (I've put up a few short posts that might've been mostly filler over the years, but not too many.)

Anyway, back in 2016 I wrote a little Python script to make a plot of my posts-per-month over time. (Amusingly, I've just discovered that that post, written on the blog's sixth anniversary, was intended for the fifth anniversary, so I guess I've started a tradition now.) I still had it lying around on my hard drive, so I've modernized it a bit, applied a few things I've learned in the intervening years, and changed it to plot each year as its own color:

The numbers after each month are the total number of posts in that month across all eleven years.

Here we can clearly see me start off the first three years with a pretty consistently high output, where even the slowest months have only been rarely been equaled since. In 2010 and 2011 I was still a student, and though I did take some pretty brutal workloads I obviously still had plenty of time on my hands for writing. (I'm sure the novelty factor helped a bit, too.) For almost the entirety of 2012 I was working at the Visitor Information Station on Mauna Kea, where I had full days at work and then several days off during the week, so likewise still had plenty of time (though being un- or under-employed for much of 2016 didn't really bring it back up). Looking at the line for 2013, you can almost see where I enter full time employment at the JCMT in January: it starts off pretty high at seven posts, but steadily nose-dives for the rest of the year, establishing the pattern that will generally be followed afterwards.

After the first three years, it looks like February is consistently a slow month, while May can be somewhat variable (probably because I often try to make at least one post around my birthday). Both June and July I've slipped up and only had one post in, while September through November are remarkably uniform at between two and four posts for the past eight years. And finally, December can be either pretty quiet, or one of the busiest months of the year; usually when I'm visiting family for Christmas I don't have much opportunity to write (the two years with six posts, 2017 and 2020, were ones I spent in Australia), but it's not a hard and fast rule as I didn't write any more posts than normal in 2018 when I didn't go anywhere.

Anyway, that's pretty much it. The colors used in the plot were taken from an excellent Python package for scientific plots, CMasher, created just in the past few years by a fellow PhD student here at Swinburne. I took the opportunity while making this plot to learn how to scrape HTML with the Beautiful Soup library, so instead of manually entering post counts (which is so 2016) my script now automatically scrapes them from the blog itself, which should make doing the inevitable sixteenth anniversary plot a bit easier. That's all for now, though I've definitely got a few ideas for posts for the coming weeks (and months. I should try to bump February's count up this year…) A hui hou! (Which I was also gratified to discover I've been using since the very first post ten eleven years ago.)

Fun with Python Decorators on February 29

Happy leap year, everyone. Despite having been writing this blog for a decade now, it turns out I didn't think to write a post on February 29 either of the previous two leap years it's been around for. So I'm rectifying that oversight this year! Of course, there's a purpose to this post beyond merely taking the opportunity of having one up on February 29 (though that was a motivating factor). This week I wrote some rather interesting Python code, and thought I'd share it.

I was working on some code to generate synthetic data sets to test various line-fitting code, and wanted a way to add some random noise to the output, but potentially using different functions to generate the noise, which might take their own variable number of parameters. I turned to the concept of decorators in Python, which are, essentially, functions which operate on other functions. For a mathematical analogy, consider the following situation: \[f(x)=g(h(x)).\] Here we have a function, g, which operates on another function \(h(x)\), and we define this resulting function as \(f(x)\). In Python, the function g would be a decorator, as it takes another function and returns a modified form of it.

Of course, we can extend this idea further; what if, in addition to a function, g also takes additional parameters? Perhaps something like \[f(x)=g(h(x), a, b, c).\] It turns out we can do this in Python as well, though it's somewhat abstract and I don't fully understand it. I read some tutorials on the subject, I guessed at how to extend what they said into code which works, but I'd be lying if I said I truly understood it at this point. Though I'll still take a stab at explaining it. The process involves a triply-nested function to handle passing arbitrary functions and arguments to the decorator. I've embedded the entire decorator function below:

 def add_noise(noise_func, *noise_args, **noise_kwargs):  
   """Add noise from a given function to the output of the decorated function.  
   Parameters  
   ----------  
   noise_func : callable  
       A function to add noise to a data set. Should take as input a 1-D array  
       (and optionally additional parameters) and return the same.  
   args, kwargs  
       Additional arguments passed to this decorator will be passed on through  
       to `noise_func`.  
   Returns  
   -------  
   callable  
       A decorated function which adds noise from the given `noise_func` to  
       its own output.  
   """  
   def decorator_noise(func):  
       @functools.wraps(func)  
       def wrapper_noise(*args, **kwargs):  
           # Create the values from the function wrapped:  
           y = func(*args, **kwargs)  
           # Now generate noise to add to those values from the noise function  
           # provided to the decorator:  
           noise = noise_func(y, *noise_args, **noise_kwargs)  
           return y + noise  
       return wrapper_noise  
   return decorator_noise  

The main action happens in the third function where the decorated function is used to create a set of data points, y, the given noise function is used to create a variable noise, and then y and noise are added together to give the output result. You can then use this decorator on a function at definition time like so (assuming you already have a function called gaussian which takes a single parameter sigma and does the appropriate calculations:

 @add_noise(gaussian, sigma=10)  
 def generate_line_1d(x, m, b):  
   ... code ...  

This would mean any time you called the generate_line_1d function its output would be modified by the addition of Gaussian noise drawn from a distribution with a standard deviation of 10. If you instead wanted to define multiple instances of generate_line_1d with, say, different values of the sigma parameter, you could do the following:

 gaussian_10 = add_noise(gaussian, sigma=10)(generate_line_1d)  
 gaussian_20 = add_noise(gaussian, sigma=20)(generate_line_1d)  
 ...  

And so on and so forth. These various returned functions would be analogous to the \(f(x)\) defined above. You could also switch out the gaussian function for another function, which could itself take an arbitrary number of arguments.

Looking at it now, it feels less useful in the specific context I'm using it in than it seemed when I was writing it, but I'm still proud of it—it's basically more flexible and abstract than I really need, but it's still a pretty neat trick of abstraction. Come the middle of this year I'll have been using Python for a decade now, and I'm still learning new tricks and features. And hey, I might not necessarily need it now, but you never know when it might come in handy down the road! A hui hou!

Hawaiian Names in Astronomy, or, Fun with Regular Expressions!

I'm currently working on a post for Astrobites about the pronunciation of Hawaiian names due to the fact that there are a few Hawaiian names already used in astronomy and will likely be more in the future, so I figured it would be good for astronomers to be able to pronounce them correctly. I already knew of a few Hawaiian names used in astronomy, but it got me wondering if there were any I didn't know about. For reference, the ones I knew about beforehand were:

1. Haumea, a likely dwarf planet on the outskirts of the Solar System (and its two moons Hiʻiaka and Nāmaka),
2. Laniakea, the name of the galaxy supercluster to which our own Milky Way belongs,
3. and ʻOumuamua, the first known interstellar interloper.

I wondered, while writing, if there were additional Hawaiian names among the named minor bodies in the Solar System (besides Haumea), so I decided to search for them using the power of Python and regular expressions.

Regular expressions, in computing terms, are a sort of meta-language used for matching specific patterns in text. There's no official standard for them, but there are a few informal standard “dialects” which many different languages (including Python) adhere to. At their simplest, a regular expression can be a literal search—for instance, in the sentence:

“The quick brown fox jumps over the lazy dog,”

I could make a regular expression to match the single, literal word “fox.” You'll be familiar with this if you've ever tried to search for something in a text file. However, the power of regular expressions comes from their ability to search for more abstract combinations of letters (or numerals, or punctuation). For instance, I could also make a more complicated regular expression that matches “any group of three letters, surrounded by whitespace or punctuation, where the middle letter is a vowel and the outer letters are consonants,” which would match fox and dog, but not the.

The specific syntax is fairly complicated, so I won't go into it here. Instead, I'll walk through the conceptual process of how I used regular expressions to find Hawaiian names in the IAU list of named minor Solar System bodies.

1. I first visited this IAU webpage, which has a conveniently-alphabetized list of all 21,922 named minor bodies, saved the list to a text file, and read its contents using Python.
2. I then took advantage of the fact that Hawaiian orthography is quite regular (no pun intended). All words in Hawaiian are made of one or more syllables, which are composed of exactly one vowel or diphthong (two vowels), and which may optionally have exactly one consonant at the start. We can be smart about it by only looking for consonants which actually appear in Hawaiian (h, k, l, m, n, p, w, ʻ), and we can also exclude any words which have the same vowel repeated twice in a row, since those don't show up in Hawaiian words. (They do show up in Hawaiian words when the ʻokina isn't written, but I figured any Hawaiian names in the list would be recent enough that the person naming them would have taken care to use the correct letters, so I decided not to worry about potentially missing some this way.)
3. This returned 288 matches, and is also where I ran out of clever tricks. Unfortunately, there are a lot of names in the list which could be Hawaiian names, but aren't (as far as I know) For instance, the first result, by numerical order on the list, is asteroid 32 Pomona. This could be a perfectly fine Hawaiian name, but it's not; it's the name of a Roman goddess of fruit trees (at least, I know it was named for the Roman goddess, I can't actually say that Pomona doesn't occur as a name in Hawaiian somewhere). At this point I sifted through the remaining names in the list and checked each one on the IAU website, which, conveniently enough, includes an explanation of where most of the names come from or who they're named for.

Anyway, to make a long story short, I found 15 names of definite Hawaiian origin, in addition to Haumea. (I discarded one, 197708 Kalipona, which didn't have an explanation and which I wasn't sure about.) Here they are, in numerical order:

• 2202 Pele: named after the Hawaiian goddess of volcanoes. Name given in 1972, which is the earliest I could definitively find.
• 7613 ʻAkikiki: a critically-endangered honeycreeper native to Kauaʻi.
• 14764 Kīlauea: named for Kīlauea volcano on Hawaiʻi island. (I notice that although the names I found correctly use the ʻokina, they don't seem to use the kahakō that indicate long vowels, so I'll add them as appropriate when I'm aware of them.)
• 88297 Huikilolani: the name of the Hawaiian Astronomical Society, “Hui Kilolani,” which translates to “club of sky watchers.”
• 123290 Mānoa: a valley and residential district on Oʻahu. (Where the University of Hawaii at Manoa is located, I presume.)
• 136108 Haumea, which we've already seen, but it serves as a good consistency check!
• 171183 Haleakalā: the largest and tallest volcano on Maui, where a number of observatories reside.
• 284891 Kona: named for the region on the west side of Hawaiʻi.
• 342431 Hilo: my favorite place to live!
• 115801 Punahou: a school in Honolulu.
• 361267 ʻIʻiwi: a species of brilliant scarlet honeycreeper found in Hawaii.
• 374710 ʻŌʻō: an extinct genus of Hawaiian birds.
• 378002 ʻAkialoa: an extinct genus of Hawaiian honeycreepers. (Noticing a pattern yet?)
• 388282 ʻAkepa: a type of crossbill bird endemic to the Hawaiian islands, though likely extinct on all but Hawaiʻi. (ʻAkepa means “agile” in Hawaiian.)
• 469219 Kamoʻoalewa: an unusual asteroid which is currently the most stable quasi-satellite of Earth; it orbits the Sun with nearly the same orbit, and never gets too far away. One of two asteroids named by the new A Hui He Inoa program, an initiative for helping get more Hawaiian names into astronomy.
• 514107 Kaʻepaokaʻāwela: an unusual asteroid which orbits in a 1:1 resonance with Jupiter, but in a retrograde orbit. The other asteroid named by A Hui He Inoa.

So there we have it, a list of all the Hawaiian names I can find currently used in astronomy. If you know of any more, I'd be interested to hear of them—it's possible I could've missed some. It's certainly a lot more than I knew of before I started! A hui hou!

A Slug and a Shell

In news which will be of no surprise to those following my painting progress, I've gone back and reworked a painting of mine due to dissatisfaction with how it turned out. The painting in question is my blue glaucus painting, which I already reworked to add a shadow. Turns out I was never satisfied with the way the shadow looked, as it came out very gray and solid, so I decided to try redoing it with a thin black glaze today. I used some ivory black, which is the only black I have listed as anything less than “opaque” (it's “semi-opaque”). I think it worked out fairly well, you can judge for yourself below:

The old version…

…and the newly-blackened shadow.

This way the shadow looks a lot more distinct from the slug casting it, which is also quite silvery-gray. It'd probably have been good to make the shadow more out-of-focus from the start, but at this point I think I'm happy with it.

And what's the shell mentioned in the title, you may be asking? In computer terms, a shell is (to massively simplify, since I'm not even sure I understand it completely) essentially a program that you can use to send commands to the computer by typing them into a terminal window. These commands are pretty low-level, and are immensely powerful, to the point where you can easily save yourself hundreds or thousands of actions if you know what you're doing.

Shells are comparatively ancient in computer history terms, so there are a few common ones and several less commons variants floating around out there. The generally most common one is called Bash, which came out the year I was born in 1989 and stands for Bourne Again SHell (as it was written to be a free software replacement for the then-popular Bourne shell, which came out a decade earlier). This is by far the most common one encountered, as it comes as the default shell in most Linux distributions and macOS, and it's the one I've got the most experience with.

A few days ago I came across another shell called “xonsh” (pronounced “konsh” in a play on “conch shell;” perhaps the ‘x’ is meant to represent a Greek χ?). It had the intriguing premise of being written in Python, the language I've been using at work for quite a few years now, and being able to execute Python code directly in the shell while still retaining access to familiar Bash routines. In Bash you need to open the Python interpreter directly before being able to type and evaluate Python code, and you lose access to Bash's lower-level functions like being able to change directories or list their contents while doing so (technically Bash simply calls little executable programs for that, but you can't do so directly in the Python interpreter). Xonsh offers an alluring alternative where Python code and lower-level functions can be called and mixed freely, so I installed it on my desktop at home this weekend and intend to give it a try. I actually do run into situations in my PhD where I'd love to be able to execute some Python code at the terminal while using Bash, so it'll be interesting to see if this goes anywhere.

The caveat of any non-Bash shell, of course, is adoption; any other shell is not as popular, and therefore won't have as many resources about it online or people familiar with it to ask. I doubt I'll be ditching Bash anytime soon, but maybe I'll be surprised and find I can really replace it entirely. We'll see! A hui hou!

A Birthday, Collaboration, and the Open Source Process

Yesterday I turned thirty, and this past month I got my first and second “real” pull requests accepted, into the Astroquery module of Astropy.

If you don't understand what I just said, I'm going to need to do some explaining. Let's start with the concept of “open source” mentioned in the title: open source, as used in computing, refers to computer programs where the source code for the program is available somehow for inspection. An open-source program is one where anyone can come along and look at the underlying code, and usually (though it depends on the license) take it, modify it, and use it themselves. Typically it also involves an idea of open collaboration, where anyone can suggest improvements to the code for the benefit of all users.

A “pull request” is one such way to suggest an improvement, using the popular version control software Git (originally written by Linus Torvalds, also the creator of the original Linux kernel). The website GitHub.com hosts vast numbers of Git repositories (the name for a collection of all the source code for a project) and makes it easy to coordinate collaboration from many people around the world. A pull request is a request to the maintainer of a repository to merge (or “pull in”) some changes from another source.

Around a month and a half go ago I started using the Astroquery module of the Astropy project (which is a collection of Python code for use in astronomy). The Astroquery module allows you to query various astronomical databases that don't have official APIs; I use it for searching for information about atomic transitions from the National Institute of Standards and Technology (NIST) Atomic Spectra Database (ASD). Anyway, I discovered that there was some information being returned that wasn't being parsed into the returned results, so I made a one-line addition to my local copy of the code (after a little experimentation) which made it work. I figured it might be of interest to other people, so I made a pull request to the maintainers of the package, and after going through the review process it got accepted!

This was more of a feature addition than anything, but a week or so later I discovered an actual bug in the handling of certain Unicode characters present in the database. (The dagger character [†] was being written as an HTML multi-character code which broke the fixed-width formatting that was being performed on the query results.) This required a little more detective work to figure out, and some back-and-forth with the package maintainers on what a good fix would look like, but I found a simple, effective fix and submitted a pull request for that as well. This time the process was slightly more involved, as I wrote an automated test to cover the situation and a change log entry for the issue I'd raised regarding the bug, but after another week or so this one got accepted as well.

I've long admired the idea of open source, of people around the world giving of their time and creativity to improve software freely available to everyone, and it's a great feeling to finally be part of it myself. A person's contributions to open source projects can look good on a résumé as well (it shows you can code and work as part of a team), so it has practical benefits as well. I don't know what form future contributions might take, but I'd definitely like to continue contributing in the future as my knowledge and skill allow. A hui hou!

Problem Solving with Python

Last year I took over the task of finding short messages for the sign out in front of my church building. It's not a particularly frequent or regular task; I usually try to find something new to put up approximately every 2–4 weeks.

Now:

1. The church building is only about 3/4 mile from my house.
2. In an effort to get more exercise I prefer to walk there to change the sign (usually in the cool of the evening) rather than driving.
3. This makes carrying the box containing all the letters impractical, as it's just a bit too large and unwieldy to carry that far by hand.
4. However, carrying only the letters required (usually a few dozen) is pretty easy.

This all leads to me needing to figure out how many of each letter I'm going to need to take with me every few weeks. This week I finally got tired of counting manually and dashed off a quick script I can use in the future. It's pretty simple, so I figured I'd post here for anyone interested. I've tried to comment it pretty thoroughly (a lot more thoroughly than I normally do, if I'm honest).

#!/usr/bin/python3

# Script to print the number of each letter in a given string.
from string import ascii_uppercase, ascii_letters
import argparse

# Use the argparse module to handle command line arguments.
parser = argparse.ArgumentParser(description='Count letters in a string.')

# Add an argument for the string to be read and counted.
parser.add_argument('count_string', type=str, action='store',
                    help='The string to count the letters in.')
args = parser.parse_args()

# Take the command line argument and convert it to an uppercase
# string.
input_string = args.count_string.upper()

# Create a variable to track the number of letters:
total = 0
for character in input_string:
    # Check if each character is a letter, and update the total if so.
    if character in ascii_letters:
        total += 1

# Check against each uppercase character:
for char1 in ascii_uppercase:
    # Create a variable to store the number of this particular character.
    num = 0
    for char2 in input_string:
        # Check each character in the input against the given character in the
        # list. If they match, increment the variable by 1.
        if char1 == char2:
            num += 1

    if num > 0: # Don't bother reporting zero occurrences.
        # Print the results for the character.
        print("# of {letter}'s: {number}".format(number=num, letter=char1))

print('------')
print('Total number of letters: {0}'.format(total))

exit(0)

A typical output looks like this:

$ char_count 'A simple string to test the new script!'
# of A's: 1
# of C's: 1
# of E's: 4
# of G's: 1
# of H's: 1
# of I's: 3
# of L's: 1
# of M's: 1
# of N's: 2
# of O's: 1
# of P's: 2
# of R's: 2
# of S's: 4
# of T's: 6
# of W's: 1
------
Total number of letters: 31

For the script syntax highlighting I'm trying out this Javascript libary I found called SyntaxHighlighter. It unfortunately doesn't show up while previewing the post, but seems to work well enough in the finished posted product.

Python Practice Results

Well, the Python training course last week was a bit tiring (three eight-hour days in a row at a fast-and-furious pace), but also quite interesting. I definitely learned a lot of new stuff from it.

Some of the exercises we did had us creating graphs or plots, and while they don't mean much in and of themselves, I thought I'd show one or two of them off.

This picture is hard to see at the size it shows up, but it represents a noisy data set to which various lines have been fitted. This was quite inspiring for me because I've actually tried a few times in the past to fit lines using NumPy and hadn't been able to do so on my own.

(If you can't read the legend, the gray dots are points from the original curve with some random noise added, the blue, green, and red lines are various order polynomial fits [linear, quadratic, and cubic, respectively], the dashed line is a least-squares fit, and the gray line at the bottom is the residuals between the least-squares fit and the original line.)

This second picture below is interesting because it represents some real-world data. The spiky black line is readings from the water vapor meter (WVM) on the James Clerk Maxwell Telescope for a period of one night a few weeks ago, taken every 1.2 seconds. It measures the amount of precipitable water vapor in the atmosphere in the direction it (and the telescope) is pointing. (Lower is better, as water is extremely efficient at intercepting the sub-millimeter wavelength light that the JCMT collects.) The extremely thin red, blue, and green lines are again polynomial fits of various orders (and possibly a least-squares fit, I didn't get time to add a legend before we had to move on since this was actual data and not an exercise).

Anyway, just thought I'd share some of what we did. I'm really looking forward to being able to put some of this new knowledge to work in the future, and I'm especially excited to be able to do my own fits to data.

Python Programming Programme

The first three days of this week we're having a Python training course at my work, which I've been attending. Having no formal training in Python (or any computer language for that matter), I've been looking forward to this. While the first day was mainly an introduction to Python itself (and thus covered a lot of what I've learned myself), I still learned several new things, including some that have been mysteries to me for years.

Today we dove into efficient numerical processing with the NumPy package, something I don't really have much experience in (although much of the morning was spent working with the matplotlib graphing package, which I have been working [and struggling] with extensively for the last three weeks). I was quite excited at the end of the day when I managed to read a FITS image and display it, with a crude image stretch, entirely with Python code. It was quite exhilarating. It was cool to discover that our instructor was also an amateur astronomer, and used code he'd written to help process his images.

Tomorrow we're supposed to jump into the SciPy suite of scientific tools. I've at least used NumPy a little bit from time to time, but I haven't ever really found a need to use SciPy before so this should be very informative.

Anonymous Functions

There's a certain joy that comes of figuring out something yourself that you don't really get when someone else teaches you (although the best teachers teach in such a way that you're constantly figuring things out yourself).

I've never taken a computer science class in programming, so most of what I've learned I've figured out for myself. There are many little milestones along the way, but there are some big ones that stand out as large leaps forward in understanding. The day I suddenly understood classes in Python, for instance. I'd read the documentation on them several times before, but had never been able to make heads or tails of it, until the day when I really needed to use one...at which point it all suddenly clicked, and I reread the documentation and understood everything. It was an exhilarating, empowering feeling. My abilities as a programmer grew a bit that day, and I've used classes often since to write better code than I otherwise could have.

Last Friday I had a similar experience, suddenly understanding lambda functions in Python (known more generally as anonymous functions). (I also understood two specific functions last week, but that's not quite as interesting or exciting as understanding a whole new class of object.)

Lambda (or anonymous) functions are functions that you create when you don't need or want to go to the bother of defining a full-fledged function. Sometimes all you need is a simple function for one particular part of your code that you don't intend to use anywhere else, for instance if you need to square each item in a list of numbers. It's such a trivial thing that it isn't worth the trouble of writing a separate function for, especially if that's the only place you're going to need it.

Instead, you can write a simple lambda function to do the work for you. For example, given a list of numbers:

someList = [1, 2, 3, 4]
for number in map(lambda x: x * x, someList):
    print(number)

This code would then print out 1, 4, 9, 16. (Incidentally, the map function is one of the other functions I understood last week, as it works closely in conjunction with lambda functions.)

Basically, the map function takes all the items in an list (such as someList) and applies another function to them. The "lambda" there signifies that I'm defining a function, and the "x" means that it's only got one variable; the part after the colon, "x * x" means to take the input of the function and multiply it by itself (squaring it). So, "take each item in someList, and multiply it by itself". The print(number) part simply tells the computer to display the results of the operation on screen; it has nothing to do with printing on paper.

Anyway, it's really wonderful to be exercising the creative aspects of my brain in my job, and learning new things as I go along. A hui hou!

Solving The Raindrop Problem

Have you ever noticed that when you're driving during a rainstorm, the number of raindrops hitting your windshield seems to increase along with your speed? I first noticed it soon after I started driving, and ever since then I've intended to sit down and work out a formula to explain it. (Living in Hilo and having it rain frequently while driving up and down from Mauna Kea has tended to keep “the Raindrop Problem” as I've come to call it fresh in my mind.)

I've toyed with it on occasion, but never definitively solved it, so I finally decided to sit down and work it out rigorously. So, without further ado I shall put my years of mathematical training to the test and attempt to figure out just how the amount of rain hitting your windshield changes as a function of your speed, while simultaneously trying to explain it in terms you can follow. Ideally, I'd like to get a graph out of it.

We start out in the grand tradition of physicist everywhere by considering a very simple, idealized case. Let us assume that for our purposes, the density of raindrops is uniform everywhere that we are considering. On a small enough spatio-temporal scale this is not a bad assumption. Furthermore, assume that the raindrops are falling straight down, with no gusts of wind or other forces acting on them other than gravity and air resistance. Again, a fairly plausible scenario, especially for a lot of the rain we get in Hilo, which often comes without any accompanying wind. A steady or gusty wind such that the raindrops had a set non-zero horizontal velocity or acceleration could also be taken into account, but is more complicated than I'd like to get into right now.

Let's begin by assuming the simplest possible case. Imagine a sheet of glass of width w and height h laying horizontally under a steady, uniform rain with evenly distributed raindrops falling at uniform velocity r (for “raindrop”. I'm saving v for later). Now, we want to know how many raindrops will hit the glass in a time interval \(\Delta t\) (pronounced "delta-t", if you don't know).

If we know that the raindrops are falling at speed r, then we can multiply by the time interval \(\Delta t\) to figure out how far they fall during that interval. Thus, any raindrop within a distance  \(d=||r||\cdot\Delta t\) above the glass will hit it within time interval \(\Delta t\). (The double vertical bars around the ‘r’ serve to remind us that it is technically a vector quantity and indicate that we want the length [or magnitude] of the vector in this equation.)

Intuitively, this gives us a rectangular box of volume \(V=d\times h\times w\) over the sheet of glass within which raindrops will be able to hit the glass in time interval \(\Delta t\). Less intuitively but more rigorously this can be achieved by a double integration of the raindrop fall distance   \(d=||r||\cdot\Delta t\) over the sheet of glass:
\[V=\int_{0}^{h}\int_{0}^{w}||\overrightarrow{r}||\cdot\Delta t\ dx\,dy\]
You may refer to the image below to help keep all these symbols and concepts straight:

At this point we've nearly solved the problem of how many raindrops will hit the sheet of glass in time interval \(\Delta t\), which for our purposes will be 1 second. We just need to know the numerical density N of raindrops per unit volume times the volume where raindrops will be able to hit the glass. Putting everything we have so far into a formula, we have
\[\begin{align}n&=N\cdot V\\
&=N\cdot h\cdot w\cdot ||r||\cdot\Delta t\end{align}\]
This is all well and good, but there are two additional factors we must take into account to better approximate a car's windshield. Those factors are the angle of the windshield, and the fact that we are interested in a moving windshield.

We will now consider each effect independently, before adding them together to get a full picture of the situation.

Let's start by introducing a non-zero angle of repose to the glass sheet. Refer to the picture below to see what I mean (I've added a coordinate system for future reference):

Now, the basic problem remains the same: figuring out the volume marked by the blue parallelograms and the glass sheet. This figure is known in geometry as a parallelepiped (PARR-uh-lel-EH-pi-ped), and has the following formula for its volume (from vector calculus)
\[V=|\overrightarrow{a}\cdot(\overrightarrow{b}\times \overrightarrow{c})|\]
where a, b, and c are the vectors that make up three of the sides that meet at a vertex and the \(\times\) sign and dot have special meanings because these are vectors. (I'm not being super consistent about notating all my vectors all the time due to the constraints of working in a blog post, but I'll try to keep it clear when the distinction is important.)

“But wait a minute,” you may be thinking to yourself at this point. “Wouldn't it be easier in this case, in order to find the volume, to simply multiply the height h by a factor of \(\cos(\theta)\) to account for the diminished surface area as seen from above (where \(\theta\) runs between \(0^\circ\) for a flat sheet and \(90^\circ\) for a vertical one), and then multiply by d and w?”

Indeed it would, astute reader. In this case, such a formula would be simpler. In fact, the formula for the volume would be simply \(V=w\cdot\cos(\theta)\cdot h\cdot||r||\cdot\Delta t\).

However, the second effect we will be considering is the velocity of a moving car and attached windshield, and since I foresee vector addition on the horizon I think it would be prudent to begin incorporating vectors into the picture now.

That brings us to considering the velocity of the car (and by extension windshield) intself. Let's assume that the car is moving with a constant horizontal velocity in the positive x-direction at velocity v, as per the picture below.

The nice thing about using vectors to find the volume of the parallelepiped is that it's very easy to find the length of one of the blue lines (what we were calling d before) in the above picture: it's simply the sum of the vectors r and v. Let's call it g (for no particular reason), and we can define it as
\[\overrightarrow{g}=\overrightarrow{v}+\overrightarrow{r}\]
We can break vectors into their component parts along each axis, and in this case we have \(g_x=||\overrightarrow{v}||\) (since the x-component of g is coming from the velocity of the car), \(g_z=||\overrightarrow{r}||\) (since the z-component is coming from the velocity of the rain), and \(g_y=0\) (since we are assuming the rain is falling straight down and the car is traveling only in the x-direction).

Using the volume formula for a parallelepiped from before, the volume can be found by
\[V=|\overrightarrow{g}\cdot(\overrightarrow{h}\times\overrightarrow{w})|\]
as long as we can find a vector for g, h, and w. Once we have, the volume is given by the absolute value of the determinant of a \(3\times3\) matrix like so:
\[\begin{align}V&=\begin{Vmatrix}g_x &g_y &g_z\\ h_x &h_y &h_z\\ w_x &w_y &w_z\end{Vmatrix}\\
 &=|g_x\begin{vmatrix}h_y &h_z\\w_y &w_z\end{vmatrix}-g_y\begin{vmatrix}h_x &h_z\\w_x &w_z\end{vmatrix}+g_z\begin{vmatrix}h_x &h_y\\w_x &w_y\end{vmatrix}|\\
&=|g_x(h_yw_z-h_zw_y)-g_y(h_xw_z-h_zw_x)+g_z(h_xw_y-h_yw_x)|\end{align}\]
At this point it's an easy matter of plugging in the appropriate numbers to get the volume, after which we can use \(n=N\cdot V\) to get the number of raindrops. And to make it even better, a little reflection shows that we can simplify this symbolic equation even further.

We already know that \(g_y=0\) from above, and we can remove several other quantities upon analysis. Remember, the vector w runs only in the y-direction, so \(w_x=w_z=0\). Similarly, g and h run only the x- and z-directions, so \(g_y=h_y=0\). Upon dropping all the zero terms, the equation simplifies nicely into
\[V=|g_zh_xw_y-g_xh_zw_y|\]
We can now replace the vector components with the original quantities they stand for to remind ourselves what everything is.
\[\begin{align}g_z&=||\overrightarrow{r}||\\
h_x&=-\cos(\theta)||\overrightarrow{h}||\\
w_y&=||\overrightarrow{w}||\\
g_x&=||\overrightarrow{v}||\\
h_z&=\sin(\theta)||\overrightarrow{h}||\end{align}\]
(\(h_x\) is equal to negative \(\cos(\theta)\) because I'm taking all vectors to start at the origin, and the windshield extends slightly backwards into the negative x-axis.)

Thus, for the final volume equation we have
\[V=|\big(||\overrightarrow{r}||\cdot -\cos(\theta)||\overrightarrow{h}||\cdot ||\overrightarrow{w}||\big)-\big(||\overrightarrow{v}||\cdot \sin(\theta)||\overrightarrow{h}||\cdot ||\overrightarrow{w}||\big)|\]
At this point we are nearly ready to begin making graphs. Since we are assuming that h, w, \(\theta\), and r are all constant, we are left with an equation in v, which lends itself well to plotting.

We just need to put numbers to all our variables. Let's assume that our hypothetical windshield has a width of \(w=2\) meters and a height of \(h=0.5\) meters, giving it a surface area of \(0.5\times2=1\) m\(^2\). A little searching on the Internet finds that a “typical” raindrop has a terminal velocity of \(r=9\) meters per second. One source I found suggested a value of about \(780\) cubic millimeters of water per cubic meter of atmosphere for the density of rain. Raindrops may have diameters anywhere between 0.5 and 5 millimeters (any larger and it breaks up on the way down, any smaller and it's technically not rain, but drizzle) so the volume of an “average” raindrop of diameter 3 millimeters is
\[V=\frac{2}{3}\cdot\tau\cdot(1.5\,\text{mm})^3=14.14\,\text{mm}^3\]
Thus, on average, a cubic meter of atmosphere contains \(780\,\text{mm}^3\div14.14\,\text{mm}^3\approx55\) raindrops. So \(N=55\) for our example here.

Plugging all of this into a Python script I wrote, I was able to use the matplotlib graphing package to generate the graph below with multiple plots for different windshield angles.

The results are, perhaps, not too surprising, but still interesting (and make a rather pretty graph). The number of raindrops appears to increase linearly with speed, with steeper windshield angles (like those found in trucks or large vans) having a higher rate of increase than lower angles (like those found more in cars). The limiting cases of \(0^\circ\) and \(90^\circ\) are illustrative; a flat surface (like on the roof of vehicles) would have no change whatsoever with changing speed, while a vertical surface would start out with no raindrops hitting it (as expected), but would eventually have the most raindrops hitting it if you could go fast enough (45 meters per second is about 100 mph, so it would have to be pretty fast, but you can see that even by about 75 mph it has surpassed nearly all other angle inclinations but the \(75^\circ\) one.) I'd estimate that most cars have windshield angles around \(45^\circ\pm15^\circ\), which is best represented by the cyan line on the graph.

So, now you know (or at least have a good idea) why you need to run the windshield wipers at faster speeds when going faster. And it only took three semesters of calculus to do! (Granted, I could have done it algebraically for this simple case, but why pass up the chance to do some exciting vector calculus?) It might be interesting for a follow-up post to consider time of travel, and how many raindrops you would actually encounter at different speeds for a given trip length. Anyway, a hui hou!

Victory! Vindication! Version 0.4.0!

Success! I'm very excited this evening because the newest version of FluxClassify, my spectra-categorizing program, is now functional. The three weeks I spent learning new code and racking my brain over how to do things was not in vain. It's already a huge step up from the previous version, and much, much, easier to use. It's not quite ready to release to my eager group of spectra-classifiers due to some details I intend to implement to make it even easier, but I hope to have it out with a bare minimum of extra features by this weekend. I'm even more excited about this, my first program written with Pygame, than I am with its predecessor, written in wxPython (and my first GUI), because Pygame is much lower-level than wxPython so I had to learn how to do a lot of things the hard way or re-invent the wheel in various places. The flip side is that I have extremely fine control over everything, so if I can figure out how to do something, I can do it exactly the way I want it. Ah, the sweet taste of victory.

See this? This is a shot of FluxClassify's main menu. It may not look like much, but that list of observations to choose from is auto-generated. That's right, it automatically takes stock of the data you give it to dynamically create a clickable menu on start-up. Also? Everything you see there is relatively placed based on the size of the screen, so it should work just fine on differently-sized screens.

Although, it is a bit sparse right now...I may add a better background in the future. I do plan to add some "Options" buttons later this week, such as the ability to mute the sound.

...Oh, I didn't mention it has sound effects? That's another nice thing about Pygame, it allows easy integration and playback of sounds. It's amazing how much a simple "click" sound when you mouse over a menu option  adds to the experience. In essence, I'm designing a program that I would enjoy using, to make sure that other people enjoy using it too. And I won't stop improving it till it's nice and polished. It doesn't hurt that I have fairly high standards in program usability and quality.

The entire reason I spent three weeks learning and writing new code was for what you see before you here. This here is a shot of FluxClassify in action. As I mentioned before, one observation comes in two parts, one taken through a blue filter, one taken through a red one. Up until today, people have had to classify those two parts separately, by manually flipping through pictures and switching between them and FluxClassify to record their results. It's time consuming, inefficient, somewhat boring, and obscures the big picture of the observation. This program is designed to do away with all that, by displaying both the blue spectrum (on the left) and the red spectrum (on the right) together in one place, something I couldn't figure out how to do in wxPython (I couldn't get it show any pictures, period). I chose this picture, incidentally, because of the strong flux seen in both spectra - a good hydrogen-beta emission line on the left, and an even nicer hydrogen-alpha line on the right.

If you're wondering how exactly to use this program, well, that's why I said it's not quite ready to release yet. Currently you can classify spectra with a few key presses (and I plan to keep that), but I also plan to include mouse clicking, and buttons to show people their options and what they picked. Once I have that stuff down, and a few other odds and ends I've thought of, it'll be ready to release for beta testing! A hui hou!

Pygame to the rescue!

I don't think I've mentioned this before, but in the project Dr. Takamiya and I have been working on we came to the conclusion (after many attempts to stave it off) that the analysis of spectra could not be done reliably by computer (too many false positives), but would need to be done by a human. And when I mention we have a total of 33,525 spectra, you'll appreciate that it's going to take some time.

Because of that, I decided to implement Project Spectra Zoo, modeled after the much more famous and polished Galaxy Zoo. Basically, it involves volunteers from the large number of incoming freshmen this semester doing the analysis for me (in return for getting their names in the paper and some valuable experience). Now, in the spectra to be analyzed, there are a lot that are not entirely clear as to how they should be classified even for someone like myself, so the idea is to have every spectrum analyzed by at least two people and then compare the results, paying attention to those cases where people disagreed. Since having some sort of standardized results would facilitate that happening, I decided to write a program to help people with the analysis.

(Now, I'd just like to say that writing this program shows just how much I have learned and grown over the past year. I used a Python package called wxPython for writing graphical user interfaces [GUI's], and it would have been completely impossible for me to have written something in it at the end of last summer. I know, because I tried for a few days, and got nowhere. Only after learning a bunch more about Python, including the concept of classes, was I able to understand how to use wxPython.)

wxPython is good at what it does, which is created programs with a native feel to them for whatever OS you're using. Using it I was able to create FluxClassify, seen below in version 0.3.0.

FluxClassify.py, V0.3.0

Briefly, each numbered checkbox represents a spectrum in the 15-by-15 array format we're working with. Users can check boxes to represent the presence of a spectral line in an image (or even activate a third state for an "I don't know" answer), save their classifications for both the red and the blue filters (each picture comes in two parts of 225 spectra each), and output a file containing all the classifications they made in an easy computer-readable format.

It's perfectly usable as it is, but it requires users to manually flip between the spectra and the program which gets distracting and slows the process down considerably. I dreamed of a new program, one more similar to Galaxy Zoo: one that would display a single spectrum, wait for the user to classify it, then move on to the next one.  With this grand vision in mind, I set about learning how to place graphics in wxPython (the spectra are all saved as JPEG files).

This turned out to be a surprisingly difficult and daunting task. It turns out that the wxPython documentation is awful. I've always found the (plain) Python documentation to be an amazingly helpful and useful resource, and never realized just how above-par it was. After a two days of bumbling around with confusing and complicated wxPython tutorials, I was ready to—well, not give up, but to look around for other options. It was looking for other options after my first tangle with wxPython that led me to the much-more-useful-for-my-purposes Python Imaging Library last summer, after all.

So a few days ago, I was considering the problem of getting graphics—images—into a program with a GUI. And then the idea came to me: what programs, out of all computer programs, tend to make the heaviest use of graphics? The answer is games. Games, more than any other program, use the most and most complicated graphics. This line of thought came to me because I had recently downloaded the Pygame package when I discovered it had a version for Python 3.2, intending to check it out later. Now, I've spent several hours over the last two days checking out Pygame's documentation, and it turns out to be much simpler getting graphics on-screen. I haven't begun writing FluxClassify's successor yet, but I'm sure it will be a lot easier. It may not look quite as polished, but it'll be more efficient and easier to use, and that's what counts.

And who knows, considering I'll have used a package intended for making games to write it, maybe I'll throw a gaming element or two in there as motivation for my volunteers to classify spectra. It doesn't hurt to make scientific research fun, after all!

Plenty o’ Python Packages

I feel like I've been talking about Python a lot recently, but I just can't help it. When you find such a useful tool it's only natural to want to tell other people about it. And yesterday I discovered a wonderful resource for anyone interested in Python: the Unofficial Windows Binaries for Python Extension Packages.

There are a few reasons I found this collection of dozens of Python packages significant. First, currently most ‘official’ Python packages tend to be for 32-bit versions of Windows. I have a 64-bit machine, however, and it's bugged me in the past that there weren't versions capable of taking full advantage of the improvements offered by the extra power of 2. The author of this site basically created 64-bit packages of his own for nigh-on a hundred different Python packages, usually for several different versions. Which brings me to my second point.

The Python 3.x line came out a few years ago, and for various reasons having to do with certain inefficient and old aspects of the language was made incompatible with previous versions. What this meant is that new versions of old Python packages had to be reworked, which can often take a long time. There are a few key Python packages that I've wanted to get for quite some time but which were not available for the 3.x line yet. I didn't want to be shackled into using an older version (which would, eventually, die out), so I held off on getting them. Anyway, the point is, many of the packages on this site have versions for the 3.x line that I didn't previously know about, despite checking the official sites every few months. I went on a bit of an installation frenzy Monday night as a result, and I'm itching to try out some of the new features of the packages I have now. Since several of the packages I got are bindings for various graphic-related tasks, you may get to see some of my experiments sometime in the future.

FluxClassify.py

This weekend I wrote my very first program with a graphical user interface (GUI, often pronounced "goo-ey"). It's a program to help me classify the thousands of spectra we have for our research project. After several months of effort, Dr. Takamiya finally decided that we couldn't rely on the computer to make determinations about the presence or absence of a particular spectral line because it produced too many false positives. It's  one of those jobs that are just better handled by a human eye and brain.

Anyway, since we have around 30,000 spectra to classify, I decided that I really needed a program that would allow me to work in more than one dimension at once, so I got to work learning wxPython and produced FluxClassify over the weekend. The version you see below is version 0.2, the first version with all the features needed to produce workable results.

FluxClassify version 0.2.

As you can see it's a fairly simple program, despite being over 1,200 lines long. Most of those lines are extremely repetitive ones dealing with the hundreds of checkboxes on this thing, so I actually wrote some other code to write that part for me. I think I only actually wrote maybe 100-150 lines of the code by hand. Each of the numbered checkboxes represents one of the 225 spectra in a cube file. The ones along the top check or uncheck an entire column, while the one labeled "Check All" does exactly what it says.

WxPython is basically a program that works with an existing Python installation (only versions 2.4-2.8 as of this writing, sadly) to provide wrappers for the wxWidgets library that provides a native look and feel for programs written in it no matter what platform they're implemented on. It's pretty cool, since it automatically includes such Windows 7 features as automatically expanding to fill half the screen when I drag the program to one side of my desktop.

...And looking at my screenshot here I just realized that I have both columns and rows labeled as columns. Time for version 0.2.01! A hui hou!

Daniel's Software Musings (Part 1)

As you can see, it's been a few days since I said I would do some reviewing. When I wrote that, I conveniently managed to forget that I tend to put off doing reviews because the philosophical concentration required to achieve an Aristotelian level of understanding of the review subject's essence is somewhat taxing. But I didn't forget, and it's been nagging me for the past two days, so here's the first part of what will most likely be a multi-part series over time of various programs.

To start off, I'm going to review some programs that I use fairly frequently. I'll start of by mentioning that all of the programs I'll be reviewing are free; most free in the sense of “freedom to use for your own purposes” (which usually means open source), all free in the sense of “no cost”. Also, because of that, most of these programs will run on pretty much any operating system.

Creativity

Blender

Blender is a 3-dimensional modeling program that I've been using since the autumn of 2008. I've posted pictures made using it before, most recently here. It is a very powerful program, and as such has a learning curve that may charitably be described as “precipitous”. However, for the persevering, there are a lot of resources online detailing how to use it, ranging from tutorials for those who have never opened it before to discussions of many of the highly advanced features that are hidden within. Blender may not be quite on par with similar commercial products, but it is still an incredibly versatile and powerful program, and is a couple of thousand dollars cheaper to boot. If you're willing to put the time into learning it, you can produce some amazing stuff. In addition to still renders, it can also do animation.

The GIMP

The GIMP, or GNU Image Manipulation Program as the full unimaginative name goes, is an incredibly powerful application on par with the much costlier Photoshop in many ways. And the ways it isn't are ones that typically only the most advanced of users will notice. I use this program for all my picture editing needs (such as the many panoramas I make) because it can handle anything from the simplest of tasks to projects of extreme difficulty and complexity. For instance, the picture in this post was made entirely with GIMP. I used it to make a few modifications to the Blender logo picture seen above, in fact. Like any new program it will take you a while to get used to, but it definitely has an easier learning curve than Blender by virtue of the fact that you're usually working with pre-existing pictures instead of making them from scratch, and you can get your feet wet doing small modifications before jumping into the deep end of photo editing.

Productivity

Python

If you need a powerful, multi-purpose programming language that is also simple to learn, you need Python. This elegant language is easy to pick up (I've learned everything I know about it by consulting its thorough and well-organized help documents) and yet has a wide variety of standard commands that provide most of what you need right away (“batteries included” is a phrase often used by fans to describe it). And if that function you need doesn't happen to come with the download, chances are, unless it's an extremely uncommon or unusual function, that someone, somewhere, has already coded it up and made it available. Ultimately, Python provides a nice “first language” for beginners that can be explored and extended to match the user's growing skill. It's well summed up in the unofficial motto: “easy things should be easy, and hard things should be possible.”

Recreation

Quod Libet

Quod Libet is a very interesting little music organization and playback program that plays a variety of common formats. It comes with Ex Falsa, a bundle of code that organizes your music based on tags. Or, rather, it lets you organize your music the way you want to using tags. It had a definite learning curve to it, but I find it easy and enjoyable to use now. It lets you search your music based on any tags it has: name, composer, album, artist, etc. Like other such programs, you can organize your music into playlists, which it will play very well. And it has one real stand-out feature I haven't seen before: by right-clicking on the Play button while a song is playing you can tell Quod Libet to stop after that song, which turns out to be a surprisingly useful feature when you only want to listen to one song in a playlist. It's one of those didn't-know-you-needed-it-but-now-you-can't-live-without-it functions. (If you're wondering about the name, “quod libet” is a Latin phrase meaning reoughly “whatever you like” or “what you will.” It comes from a phrase in logic “ex falsa, quod libet”, which means “from a falsehood, whatever you want” referring to the fact that if you begin with a false premise you can logically prove any (incorrect) thing you want. The developers liked the idea of “whatever you want” to describe the organization capabilities of Quod Libet, hence the name.)

Keep watching in the future for more informative and hopefully interesting reviews! A hui hou!

Now you're thinking with Python...

Today I created something new and wonderful, a master database containing all the information on all the data Dr. Takamiya and I have collected for our research. This is a big database. We have 67,050 individual spectra which come in pairs (one red, one blue), for a total of 33,525 individual spaxels. Currently there are 46 different pieces of information for each spaxel, for a combined total of 1,542,150 pieces of information contained therein. The information for each spaxel was spread out over 5 different files prior being collected and organized (that's 5 files for each spaxel, there are actually a total of 596 files for the individual spaxels, plus one more that contains certain information for each group of 225 spaxels that makes up one-half of an observation). This is a single text file that's 11 megabytes in size, bigger than the pictures my camera takes at maximum resolution (though not so large as to be unmanageable).

I'm pretty excited about this as you can imagine. This is the culmination of over a year of work on my part. I've really enjoyed working with my mentor Dr. Takamiya for the entire time. She wonderfully handled the balance between letting me try and fail and learn on my own and being there to help when I really couldn't figure something out. For the first several months I was too ashamed to admit that I didn't fully comprehend the big picture of what we were doing, but she was always willing to explain it to me again and able to show me the next little step to take to get there (and yes, I finally understand it now). When I say the database is the culmination of work on my part, I really mean it; while she showed me the way to go, she left nearly all the actual design and implementation decisions up to me, and the process indelibly bears my mark in many ways (for good or bad). I've grown in so many ways as a Python programmer during this time I can hardly begin to list them all, though I think the biggest is my recent sudden understanding of Python classes, something that would never have happened if I hadn't spent so much time thinking and struggling and working with Python over the last year.

Of course, this reminiscing isn't to say that we're done. Far from it! In some ways the real work is just getting started. Now we can begin rigorously plotting the data to really tease out the relationships that have been hinted at previously. That should keep us occupied for some time...

Classes!

There are two things exciting me today, so I decided I'd write about both of them. First of all, to resolve yesterday's cliffhanger, yes, I was able to see the supernova. Not by eye, but I was able to capture an imager of it using the imaging telescope at the Vis (another volunteer there told me he'd spent the night before trying to see it visually with the largest telescope at the Vis and had no luck, so I didn't bother looking for it visually).

I haven't had time to reduce the image yet due to the other thing exciting me, but hopefully I can show you all pretty soon what it's like to see a single star putting out more energy than an entire galaxy 25 million light years away.

The other thing exciting me is that I suddenly learned how to use classes in Python, my favorite scripting language. Classes are basically user-defined objects, and a definite step up in the abstraction department. The reason I'm so excited about it is because of the way it happened.

Now, before Tuesday, I'd never used a class in Python, nor did I understand them. I'd read the documentation introducing them before two or three times, but couldn't really make heads or tails of it. Then, Tuesday, as Dr. Takamiya and I were trying to pin down an unrelated bug in my code, I became painfully aware just how convoluted my data was getting in my script. Lists were stacked in other lists, which were stacked in still more lists, and it was becoming a nightmare remembering what was what while executing the script.

Finally I took a little break to think, and the thought popped into my mind “I think I need to use a class for this”. Remember, I had zero prior experience with classes up to that point. However, since we weren't really getting anywhere the way we were going, I went home to do some quiet thinking and see if this idea would pan out.

And did it ever! I found and read the documentation on classes again, and it's as if everything had clicked into place in my mind. I understood everything perfectly clearly. I had to run off to go up to Mauna Kea soon afterwards, but I had the same experience today when I took it up again. It still feels somewhat surreal, as if someone just flipped a switch in my mind. I see it as another case of the fact that sometimes, at least for me, I need to figure out something on my own before I can effectively learn it. I essentially came up with the idea, “I need to be able to do this in Python”, and lo and behold, there exists a method to do this. I'd never appreciated it before because I hadn't really needed like I did recently, but when I finally did, it came practically like second nature. (If you're interested, classes are basically what I said before, user-defined objects. You can use them as containers to store arbitrary data, which is mostly what I was using them for, but they're much more versatile than that. Hard to explain in a few sentences. Let's just leave it that they're really useful and handy.)

On the Beautiful Smooth Hawaiian Language.

The more I study the Hawaiian language, the more entranced by it I grow. It is both an elegant and a beautiful tongue. It uses an incredible economy of sounds -- there may exist languages with fewer sounds in the world, but none that I know of. It is sufficiently different from other languages I've studied to inflame my curiosity, yet simultaneously I recognize many tantalizing common linguistic threads running through it. Songs or chants in it often have a haunting beauty, and it lends even simple greetings a peculiar charm.

Hawaiian is interesting to me because, unlike other languages I've studied like Latin and Greek, its verbs do not decline or change at all. Mood, tense, and other things are handled by additional words in the sentence, somewhat similarly to English, but taken much further. Nouns, too, do not change to indicate their number. Like English, words can be used in various ways in different contexts. Some words can be used as nouns, verbs, and adjectives!

Edit: last night while writing this I couldn't think of an English example, but this morning I thought of one: "set". If you think of more, post them in the comments!

Like Arabic, Hawaiian also has no "to be" verb. It takes a little getting used to, but once you do it seems the most natural thing in the world to omit. It simplifies and shortens phrases, and makes the language much more concise than English in most cases.

I was a bit bored tonight, so I wrote a quick function in Python to randomly generate some short, simple Hawaiian phrases for me to practice translating. It generated some pretty funny phrases, such "this student of my chair", and "the uncle of my island". I had some ideas about writing a program to generate random phrases and eventually sentences for practice, so this was sort of a proof of concept. It's like really high-tech, random flashcards.

Well, I need to get some sleep now, so a hui hou!

Image and imaging.

This Sunday I was fortunate to spend my 4th of July at the Vis on Mauna Kea under a clear dark sky. This time of year the Milky Way is simply gorgeous, so bright in the dark sky that people often mistake it for clouds. Sunday I finally fulfilled a wish of mine, and took a number of shots which I stitched together into a panorama of the Milky Way.

Summer Milky Way from Mauna Kea.

Again, you really need to click on this one to see. I should also mention that this is not exactly what the Milky Way looks like on Mauna Kea. Stitching the pictures together to make the panorama introduces some horrible stretching and distortions into the image, which due to the way I composed the shot is most visible just left of center, but is present everywhere to a small degree. It's very similar to the atrocities perpetrated upon geography by mapping a spherical globe onto a flat map. The picture is also not quite representative because it was made using 30-second exposures, so not everything in it is visible to the naked eye in such detail (although is is possible to get a glimpse, at least, of pretty much everything in this picture).

This picture is facing towards Hilo, the lights of which you can see on the horizon as an orange glow in the center of the picture. Although difficult to make out in the picture, you can just spot the glow of Kīlauea on the limb of Mauna Kea to its right. The center of our galaxy is located in the direction of the brightest region, in the upper-right part of the picture. The red-lit foreground comes from the low light level conditions and red lights used at the Vis.

In other unrelated news, I had major breakthrough with my job vocation yesterday, where I finished a project I'd been working fruitlessly on for almost a week in less than a day. I was tasked with creating images from the lists of data that I've been working with -- doable in principle, since the data came from (complicated) pictures in the first place, but not immediately obvious in practice.

I spent several days looking in the wrong places, before coming across a great resource, the Python Imaging Library. The night before I'd had trouble setting it up on my computer so I wasn't sure what to expect, but it went flawlessly: I was able to create my first test picture in less than 20 minutes after installing it, and had my first actual image within a few hours (most of which was spent figuring out how to map a one-dimensional list of numbers back into a two-dimensional image).

I then ran the script on all the data files, and the feeling I got while watching image files pop into existence in the folder I'd created for them was pure exhilaration. I'm sure the creators of the first digital cameras must have felt similar when they got their first images. The pictures aren't much to look at -- just extremely grainy, gray-scale images -- but knowing the exact process of their creation makes them much more interesting to me.

Well. I guess it's appropriate that this post about a digital image turned into a discussion about digital imaging. Perhaps at a later date I can share some of the more interesting pictures created by my script with you all.