Using Python for Telescope/CCD Camera Automation and Data Processing

After having gone on my first real astronomy data run at the Pinto Valley Observatory, I saw quite a few things that could be done to improve the workflow for next time around. The two things that I am focusing on at the moment are:

  • Telescope and CCD camera integration for data acquisition
  • Data processing

The goal is to eventually be able to feed a list to a script, and go to sleep while it completes the data acquisition for the night.

Telescope and Camera Control

For the telescope and camera integration, I would like to write some code that can take a list of targets, integration times, and perhaps some other settings, and is able to figure out what order to aim at the targets and take data based on when it is closest to the zenith for the least amount of atmospheric noise. It was very cumbersome do take notes on the data set and target on one laptop, control the telescope from a second laptop, and take the data and control the camera from a third laptop. All of this could be simplified with some scripting that uses the provided lists of targets, moves the telescope (with ASCOM, INDI, SiTech, etc.), corrects for tracking errors and centers the target, and then starts the data acquisition and stores it away somewhere.

The main problem with all of this seems to be the complete lack of standardization in the telescope and camera world. Most every one operates with it’s own set of standards, which defeats the purpose. The camera we were using on PVO run was an Andor Luca-S which requires the proprietary SDK in order to control it. SBIG and a few others are able to be controlled using free and available libraries and software. On the telescope side, ASCOM was a well intentioned idea, but it’s limited to Windows based machines, which I do not have. I use Ubuntu linux, so I’m looking for alternatives that can work on any platform. INDI can do this, but it has limited device support. Fortunately, it works with the Celestron NexStar 6 that I have to play with.

So, in short, there seem to be three pieces to getting somewhere with this:

  • Figuring out how to handle the target list. Do I need to have a catalog, or am I going to rely on the user to provide RA/Dec for the targets?
  • Telescope control and tracking adjustment. Dan Gray wrote a simple spiral search script for the SiTech controller at PVO which worked well. It required manual intervention to hit “stop” once the target was in the FOV, but that could be automated easily if the camera is integrated by looking at the overall delta brightness of the images and once it changes by a certain threshold, you know that the target is in the FOV. Then you could center using this mechanism by moving in the x direction slowly until the delta changes, backup to get it in the FOV again, then move in the negative x direction until the delta drops again. Divide that by two and that target is centered for the x direction, then rinse and repeat to center in the y.
  • The camera automation seems easy enough with the Andor SDK, but, it’s proprietary which means that if I wanted to work with a different kind of camera, it would take more code. There probably isn’t a way around this other than writing functions to deal with detecting supported camera types and having code that can deal with it.

Data Processing

I’ve already started writing a bit of Python code for the data processing side of things. For the PVO run, we took data and recorded it in the proprietary Andor format called SIF. It is an undocumented format and there is nothing in the SDK for messing with these files, so you are stuck using their SOLIS software to convert it to different formats. I’m never taking data like that again. :) The Andor camera supports the FITS file format, which is used by NASA and lots of astronomy folks out there, and has quite a bit of support, including with the python module called PyFITS. I’ve been messing with that a bit today. Since all of our data from PVO is in SIF, we have to manually convert it to FITS using SOLIS before we can manipulate it by any other means. This is a tedious process which can be avoided in the future by recording data directly into the FITS format.

FITS files can be multidimentional “cubes” of data, so that our entire 2000 image set of a single target is in a single, convenient file. This is very easy to manipulate and run analysis on, and requires no extra conversion. This is what we will use for future runs.

Dr. Genet and I also use an wonderful piece of software called REDUC which is written by Florent Losse to compile our data and do various things like lucky imaging or speckle interferometry. The only issue is that REDUC only supports single dimensional FITS files, so we have to break up the FITS file from SOLIS into individual FITS files for a set of data. I’ve already been talking via email with Florent to get support in his software for data cubes, since this is widely used and accepted as a standard format for astronomical data. In the meantime, I’ve written a script that can slice up a multidimensional FITS into individual FITS files, and it also writes out a log file (in CVS format for Excel or LibreOffice, at the moment) with contains the header information for each data set, such as the exposure time, EM gain, number of photos, temperature of the camera, etc. If Florent can get the data cube support build in, I won’t need to slice up the FITS files anymore and no data conversions will need to take place which is ideal. I will still want to be able to write out a log file, since this comes in really handy once you’ve got a few nights worth of data.

The other piece which I just started playing with is actually analyzing the data. Python has some neat tools, such as the very powerful matplotlib, which might help in doing this and I’ve only begun to scratch the surface, doing some simple histograms, heat maps, and contours for some of the data that I have. This is a tantalizing possibility and I’m going to be very curious to see what I can come up with.

So there are plenty of pieces to these puzzles, and I’m going to attempt to figure something out. The data processing seems like the easiest part to tackle for the moment, and I will brainstorm on the other pieces as I go along. I’ll be sure to post something once I figure out anything substantial, since there seems to be bits and pieces all over the web, but no one with solid solutions. It looks like some of the big observatories have it figured out, but they have a lot of resources, and completely custom written software for their specific equipment. I would like to come up with something that helps out us amateurs so that we can make our own contributions to science without all of the headache.



The Roweville Run: Three Days at the Pinto Valley Observatory

Sunset at the Pinto Valley Observatory.

We made it to Roweville, deep in the heart of the Mojave Desert, just as the sun was setting. Dr. Russ Genet and I had made the 8 hour journey and were greeted by our host, Dave Rowe, and two others, Dan Gray from Sidereal Technology and Jonah Hare from PlaneWave Instruments. We quickly got settled in and ate dinner, and by the time we were done, it was already very dark since we were completely isolated from any light pollution. It was time to get to work.

The Pinto Valley Observatory, or as we call it, “Roweville”, consists of a 20 inch PlaneWave Corrected Dall-Kirkham (CDK) telescope, a marvelous instrument with superb optics that was designed by Dave, and it was controlled with equipment and software that SiTech (Dan’s company) had produced. The day before Russ and I had arrived, the crew had outfitted the scope with Renishaw encoders which allowed us very high precision tracking and finding sky objects. This was very important since we were using an 4x and a 2x Barlow (for a total of 8x) to do our research, with an Andor Luca-S EMCCD (electron multiplying CCD) camera to gather our data. Our field of view was about 49×32 arc-seconds, so having the ability to precisely slew to objects was crucial, since we didn’t have a lot of wiggle room for inaccuracy.

The 20″ PlaneWave CDK Telescope

Our first night of observations was a success. We used only the 4x Barlow on this night with no filter. We struggled with a few minor problems, like camera drivers issues in Windows, getting the camera to focus since the Barlow was shifting the focal plane very far back (we realized that we had made a mistake and needed to put the spacers in front of the Barlow, not between the camera and the Barlow). After we sorted these minor issues out, I was given a brief lesson on the software by Dan and Jonah, and they let me loose on the controls for the telescope and the camera/data collection. I was very new to all of this equipment, but after we hit a couple of targets, I settled into my work flow and managed to be highly productive at gathering our data. We did a few meridian flips with the scope and the tracking seemed to get off at this point. It was determined that it was not the fault of the scope at all, but the minor shift in the camera instrumentation hanging off of the end of the scope. Our instrumentation extended about 1.5 feet from the back of the scope, creating a perfect lever arm that was at the mercy of gravity to make minor adjustments, even the slightest of which would throw off our FOV at such high magnification. We still were able to get a lot of usable data this first night, with 33 sets and 49,600 images collected. We focused mostly on calibration and drift targets, mostly to get a feel for the equipment and to make sure that the data reflected accurate results. We also collected numerous targets at varying altitudes in the sky in order to compile data on the variance of atmospheric scintillation.

Looking for rocks! From left to right: Jonah, Russ, Dave, Dan

We went for a nice hike the next morning to go rock hunting. Dave knew where to find a nice accumulation of agate, which he explained, had formed from a nearby volcanic explosion that rained a thick layer of white ash down on the area, and as rain water percolated down through the soil, it had deposited silica in successive layers, forming the agate. There were so many beautiful colors in the rocks, and a wide variety of shapes and sized. It provided a good hike and we saw many interesting features of the surrounding area such as Wild Horse Mesa, the Stone House, and many other neat geological features. All of the roads were unpaved, and some required a high clearance vehicle to get across. There were also a couple of campgrounds hidden away in the hills.

A mountain of agate.

The second night, we were able to use our experience from the first night to be much more successful. Dan had made a couple of changes to the tracking software that Russ and I had suggested after using it a bit, and Dan had also written us a simple spiral search script, which allowed us to quickly locate a target that didn’t show up in the FOV right away. This was a huge time saver. We also operated the equipment remotely from the warmth of the main cabin.

We definitely had our system down for this second night, recording 16 sets of data per target, with four different exposure times on four different filters (none, R-band, V-band, and I-band). This allowed us to compare the results and see what integration times and filters worked best for our speckle interferometry. We were able to split some sub arc-second separation double stars using these techniques, which was a huge success for us. We also collected data for a few calibration and drift stars. We got a total of 66 sets and 128,570 images this second night.

Wild Horse Mesa

We woke up and had a nice breakfast, and decided to go for a hike. Dan brought his quadcopter along and we went up into the New York Mountains, into Caruthers Canyon. When we got to the top, we found some abandoned copper mines which Jonah and I ventured into a bit. They seemed pretty steady, and we found old reminants of the mining days in the 1920s, such as big chutes to load up the mine cars, and parts of the track that they ran on. Dan flew his quadcopter up high enough that we almost lost sight of it, then returned it safely to the ground.

Our third night was the most productive, from a scientific standpoint. Dan refined the spiral search script for us, and we knew it was out last night to get data, so we started early, around 7pm. Our focus for this last night was almost entirely production stars, in other words, measuring doubles of various separations, most of which were below 2 arc-secoonds. About two hours into the night, the battery voltage for the ranch was down to 24.0v, so we needed to shut everything down for the night or risk damaging the batteries (Roweville runs off of solar power). We made the decision to unplug all of our laptops and turn off everything on the ranch but the telescope, which got us back up to 24.3v and allowed us to continue the data collection.

Russ looking for the next targets.

Now that Russ and I were under the gun, we became highly productive. We had to move out to the cold “warm room” of the observatory since we couldn’t connect remotely to the observatory computers anymore. Russ and I quickly got situated, and since time was running out, he started listing off targets to me, usually around five at a time. I would then slew the telescope to each of these and record the data in quick succession, and be would have a new list of the next few targets by the time I was done. Unfortunately, power ran out around 11pm and we had to shut everything down, but we had managed to collect 58 sets of data and 99,040 images this last night.

Relaxing and enjoying the view.

I can’t describe how much I learned during this three night stay at Roweville. Dave was a complete gentleman and excellent host. Everyone there during those few days contributed their piece to our research, and provided good company. Even though I was completely unexperienced when I showed up, everyone was very open to showing me how to do things and sharing their vast knowledge and experience with me. As a result, Russ and I will have probably five or six scientific papers coming from the data of this run, although it will take us a few months to compile the 275,000 images, over 25GB of data. It’s a beautiful spot to enjoy dark skies and some great hiking. This was truly a unique experience for me, and made me want to continue on my path towards uncovering the mysteries of the universe even more.

Roweville or Bust!

I’m headed out to the Mojave Desert with Dr. Russ Genet to use the 20-inch PlaneWave CDK telescope at Pinto Valley Observatory at Roweville, owned and run by David Rowe, and I believe Dan Gray (from Sidereal Technology) is also joining us. We have a list of many different targets for gathering data, with our primary goal being speckle interferometry. We are also going to attempt to record data from a lunar occultation of a double star. I can’t wait to leave, this will be an exciting trip! Hopefully the weather cooperates…