How Not to Do a Survey

Here is the biggest problem with my life these days, personal problems aside. I am working on a spectroscopic survey - ~20,000 galaxies, taken over 4 years with 4 different instruments and during 15 observing runs. Wow. We have observed 28 different fields, half of them were observed with more than one instrument and/or during more than one observing run. Putting all these data together in one pile is a challenge because of all sorts of systematic effects that array of observations may have. But we did next to nothing to take care of that problem during our actual observations and now that we are trying to figure out what the errors are drawing straws. We are banking on two methods - one is measuring the velocities (i.e. offset from zero) of the sky lines (which I have already talked about), and the second is matching redshifts of objects observed twice. The second is the one causing my sleepless nights these days, the biggest problem being that the number of objects observed more than once is miniscule and that the very reason we observed them twice is because we didn't get them the first time. Errors turn out to be big, bigger than my advisor likes and I feel like crying.

My computer got upgraded today, a happy news except for the fact that the OS got upgraded too and now nothing seems to works quite as I would like it. Aghhhrrrrr!

Every other Tuesday our department holds Galaxy Group meetings for people working in extragalactic astronomy. The format is the usual "Bring-your-lunch-and-listen-to-two-speakers" thing. One of the talks today was particularly striking - what to do with a Spitzer warm mission when all that's left will be the IRAC 3.6 and 4.5 um chanels. It was ... wow!

More later... getting too sleepy...

Wow! Haven't written since Wednesday! Bad girl! I should be going to bed now (it's 2:15 am) so here's a brief summary of what I've been up to the last few days. I analyzed the good files and I am currently finishing up the reduction of the last bad one. The last 2 bad ones need to be analyzed which is a task for tomorrow. I am running some more sky extractions. I actually wrote the nth nearest neighbor script. Yey! Sounds like not much work but it took 3 days!

For tomorrow: Match up objects observed during different runs and see what the dispersion in the redshift differences are after the correction from the skylines. Make a tar-ball with all Hectospec files for my advisor. Play with the nth-nearest neighbor algorithm and talk to a new postdoc for ideas. Finish up with the analysis of the last 2 files!

A little musing on grad student life: it is Sunday night. I've spend most of the day in my office working. I wasn't the only person in the building, there were other grads and postdocs there. I skipped dinner because there's nothing dinner-able in the building and had a cupcake instead. What the hell!?!? Why am I doing this to myself?

Stop postponing!

I did almost all I had planned for today. I finished the analysis of the good data. I am rerunning the 2 files of bad data. And my computer is crunching the numbers on the extraction of ~1000 sky spectra. I am almost done with the extraction of 50% of the sky spectra.

The nth nearest neighbor code seems to be the most postponed piece of work I have ever had. It's been on my "To Do" list for about a month. It really shouldn't be that hard to do. In fact I wrote out the algorithm and seems really easy. I just haven't been inspired to do any coding lately. IDL would probably be the easiest to write the script in. Once I get around to coding I need to do 2 more things - rewrite Ann's velocity dispersion script (very important) and finish the damn coffee page (less important).

Getting up early and hiking was great! If I had gotten some more sleep the night before it would have been even better. It was so beautiful and lush and full of butterflies! Wow! Took a picture of a petroglyph of a "sunbathing man/lizard". Hope I get some more sleep tonight. Too stressed to sleep ...

Today was the first Journal Club for the semester. It was pretty packed. Journal club is an hour long weekly meeting where two people present recent papers in 30 min each. The presentation today were vastly different but both interesting (which is not always the case). One was on a recent discovery of a planet (possibly 2) around an M star and the statistics of planets - apparently more massive stars have planets more frequently. The other was on a yet another weird cluster (exhibit #2) where the galaxies, the DM and the intracluster gas are all in different positions. Is there really dark matter or is it all just a modification of Newtonian gravity? I don't know enough about the latter to pick a stand (but nobody knows anything about the former and that doesn't stop them).

Tasks for tomorrow: Nth nearest neighbor algorithm first thing in the morning. The last 2 files with crappy data. Analyze all the sky spectra I've extracted. Analyze the 2 files of data I re-extracted today.

Frustration is setting in.

I didn't do all I had planned for today. I am hanging my head down in remorse. I did about half. I *almost* analyzed 2 of the 4 big files. I found that the other 2 need some re-reduction - the edges of the slits are too bright for some reason and that screws up the following extraction which thinks the light is in the artificially bright edge. I need the re-run the the extraction with a lower "upper" parameter to get rid of the edge.

My computer keeps extracting sky spectra.

A friend of mine is visiting town this week and he is staying on our couch tonight. We had a BBQ in his honor so that he can meet some people and so that I can have people over at our house which I love. Lots of people came. We had a good time.

Tomorrow: get up early and have a short hike. Re-extract the 2 files which I found to be screwed. Re-run the cross correlation for them. Keep extracting sky spectra. Write the n-th nearest neighbor algorithm.

Another Late Night

I have been working pretty late nights lately (when I am not out drinking late nights that is). And then I come home after midnight and I start doing chores and cooking amd making all sorts of noises that probably keep my house-mate up. But then he gets up at 5am and Skype-s with his g-friend who's on the other side of the world and wakes me up. So I guess we are even.

Work is coming along. But so is frustration. I am THAT close to finishing my main data analysis which involves looking at every single one of my 20,000 spectra. I have less than 1,000 left. If I finish 4 of the 6 files I will be VERY happy. It is mind numbing but I want it to be over that bad, I will spend the whole day tomorrow doing it. I have promised myself I will get a belly-button piercing when I am done with my data reduction and analysis. I have spend a year working on it! I kind-a need a physical memento of the experience. Other than the hunch I mean. I am afraid that will have to wait for a little longer though. I have this teeny dataset I also have to get re-reduced and that may take a while to figure out. I need to flux it, format it, cross-correlate it and look at every single one of the ~900 spectra. I really shouldn't be complaining, though I was hoping I wouldn't have to do that. Patience, my dear, patience.

My computer is crunching numbers at an unprecedented rate, extracting sky spectra like you wouldn't believe it. I even managed to kill my task bar tonight trying to speed it up. Well, I did speed it up, but the gam_server does actually do something! My Adviser agreed I can get a small upgrade. Yay!

Tasks for tomorrow: Finish the analysis for the 4 big and easy files. Write the nth nearest neighbor algorithm. Do more sky-subtractions. Get a ticket for the burlesque. Have a fun BBQ.

I really need to get my act together with this blog. I have been chased down with guilt over the last week or so and haven't written much because of that. Just the usual - at the end of the day I don't feel like I have done much and I don't feel like writing. This is a really bad condition and I need to work on it because it propagates to the day after and then the day after and so on. This state is in large part due to personal problems that I have been having a difficult time dealing with - people whose mentoring I have greatly enjoyed leaving the department as well as trying to sever a relationship with a colleague leaving me less than motivated and concentrated. Both will probably get their separate posts in the future when the issues are not so emotionally charged in my mind. Now back to business. Last week I wrote a draft of the data reduction and analysis sections of the paper I am working on. The goal is to have the paper in a solid draft format by the end of the month. Goals are good and motivating but I haven't managed to get my adviser on board of this one and it's a little disconcerting about her lack of plan and sense of time.

I am still working on figuring out the systematics in the data for my thesis (I should probably describe my project one day too...). After the data reduction and analysis this is certainly the next biggie. And because we will be looking for low velocity dispersion systems we want to hammer the systematics of our redshift measurements down to minimum. So half of the data looks fabulous. But the rest is just sketchy, with varying errors. I am thorn between trying to figure out what is the ultimate source of these errors and just hardwiring a correction. For now I think we are going with the latter, but it is still a huge effort and requires many, many CPU hours. I need to upgrade my computer!

Finally, I am trying (desperately) to finish the analysis of my data. There are still a few bits and pieces here and there and it is driving me crazy! I did a huge data analysis effort earlier in the summer but then I ran out of steam and got brain-dead and stopped. That was bad, I should have just knuckled up and finished it. It's getting close...

All righty, until tomorrow!

Slow Friday

There are some of those days when the world is just not cooperating! And today was one of them. The reduction code I have run 1000 times was giving me trouble all day long (and still is as of this moment). I am hoping that the particular dataset I was working with is to blame and I am trying with a different dataset. I learned a few fun little tricks in IDL - how to construct a histogram and how to fit a Gaussian, very useful. I am using the sky spectra to zero-point my spectroscopy. At least for the instrument I am looking at, the offsets of all ~300 sky spectra per mask are fairly small (~10km/s) and their distribution is tight and very well fit by a Gaussian. Both the mean and the median as well as the center of the Gaussian seem to be good measures of the center of the distribution - they are within 1 km/s of each other. Sky spectra are messy!

Dear Blog....

Today is a random Thursday afternoon in a random month (August) of a random year (2007) and I am sitting in a house on a street in a town on the planet Earth in the Solar System of the Milky Way which is flying in a semi-random direction in our (supposedly) Lambda-CDM Universe. I am a graduate student in Astronomy in this random town and I am graduating a year from now which points to an extremely ill defined graduation date within the next 12 months. I will try to write about the work I do every day for the next 12 month. By then I would have (hopefully) graduated and (hopefully) found a job. So if you want to see what the life of a last-year grad student in Astronomy is, you've come to the right place. Let the journey begin!