## Thursday, February 28, 2013

### Setting up your Windows 7 machine for Python scientific work

A few days ago, I wrote a blog about setting up Python and (many science related modules) on a Mac. Well, today, I had to do the same on a windows 7 machine that I have to use for work. I never thought I'd say this but it turns out it was easy on windows (way easier than on my Mac ... I feel pain just writing this, and no, it won't make me like Windows). Here is how I did it.

## Installing Python

I started the installation by following instructions from this link, which basically told me to install Python from the official Python page (apparently, it was recommended to install the 32 bit version ... even if my computer is 64 bit). I installed Python 2.7.3. It downloads an installer, which you simply open and follow the instructions.

The installer will not automatically set-up your system path to use python, so we have to do it manually.

• Right-click Computer and select Properties.
• In the dialog box, select Advanced System Settings.
• In the next dialog, select Environment Variables.
• In the User Variables section, edit the PATH statement (or create a PATH variable if it does not exist) to include this:

C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;

Once the PATH is set up, you should be able to run python from the command line (open Start Menu > All Programs > Accessories > Command Prompt and type python).

## Installing Python modules

• setuptools: installs easy_install. Download the setuptools installer here and simply run it.
• pip: from the command prompt => easy_install pip
• ipython: last but not least, download ipython here (you'll need admin rights if you want the installer to put a item in the start menu)
• pyfits: pip install pyfits
• astropysics: pip install astropysics
• pyephem: easy_install pyephem
• lmfit: pip install lmfit

## Tuesday, February 26, 2013

### Setting up your mac for Python scientific work

Alright, I've just received my new iMac at work. So, I've spent some decent amount of time setting Python and modules I commonly work with. Since, the process can be rather tedious and painful, I thought I'd put some details here. This post is about how to set up Python and some scientific python modules on your Mac. Here is a list of what we will install: python (2.7.3), numpy, scipy, matplotlib, pil, pyfits, lmfit, astropysics, pyephem, ipython, qtconsole and notebook.
When I had to do it myself, I found this link very useful (this post is basically a summary of that one, with some small variations). Also, here's a newer version of the same link.

### homebrew

So, you got your brand new shiny Mac (running Mountain Lion). The first thing you want to do is install homebrew, which simply is the least painful package manager that currently exists for Mac OS. You'll install it by opening up a terminal window and typing:

 ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"  You need this for all steps in this post, so if it doesn't work, you're screwed. Well, not entirely, you can look here for some help if it doesn't install. If it does install, you want to call the Doctor and check that everything is working before you install any package:  brew doctor  This will basically check that everything you need to install new packages is working fine. Do whatever the Doctor says. He or She will tell you things like "this program is outdated, install a newer version or make sure that this is on your PATH, ...". It doesn't always tell you how to solve the problem but gives you good clues. Then, you should make sure the directory where homebrew installs program (/usr/local/bin/ if you followed standard homebrew installation) is on your path. Type  echo$PATH in the terminal. If you don't see /usr/local/bin/, add a line in your ~/.profile or ~/.bash-profile (create the file if it does not exist). That line should be:

 export PATH="/usr/local/bin":$PATH  If /usr/local/bin/ was on your path but after /usr/bin/, you should edit the files that set up the path in the first place. You can rearrange the lines of the /etc/paths. If that doesn't work, check this post. We will be installing various programs that are currently not in the default brew distribution (tap). So, "tap" the following distributions:  brew tap home-brew/science brew tap samueljohn/python  ### python Alright, you have brew working and your path is set, now let's start installing things. Lots of the software we will install work best you re-install python using homebrew. Yes, I know. You already have python, why would you install it again? To avoid pain (check here for actual reasons and details). To install Python (2.7.3 at the time I wrote this post), type in a new terminal window (to refresh the path):  brew install python --framework --universal  Again, make sure /usr/local/share/python is in your path and add it if not (export PATH=/usr/local/share/python:$PATH). Reload Terminal after any change. After installing Python, you should tell your system that the newly installed version is the one that you want to use by default. To do that, we will create a symbolic link to the desired version of Python:

 cd /System/Library/Frameworks/Python.framework/Versions sudo rm Current ln -s /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/Current 

If it works, typing which python should return /usr/local/bin/python. You might have to restart your terminal. Furthermore, if you type python, you should see "Python 2.7.3".

### Python modules

Install these. One at a time. Note that I had a few problems while installing numpy and PIL because of conflicting older files. brew told me what to do though. I had to link numpy and PIL manually (brew link numpy --overwrite), which overwrote the conflicting files.

 brew install numpy # Installs numpy brew install gfortran # Fortran compiler for scipy brew install scipy # Installs scipy brew install matplotlib # Installs matplotlib brew install pil # Installs PIL pip install pyfits # Installs pyfits pip install astropysics # Installs astropysics pip install pyephem # Installs pyephem pip install lmfit # Installs lmfit 

### IPython, QTConsole and notebooks

 pip install ipython # Installs IPython brew install pyqt # Installs pyqt brew install zmq pip install pyzmq pip install pygments 

You should now be able to launch the qtconsole by executing:

 ipython qtconsole --pylab=inline 

For installing notebook, see this link.

### ssh configuration file

Here is an excellent post on how to configure your machine, so you won't have to remember host names, logins, and various preferences for your favorite ssh servers.

## Thursday, February 21, 2013

### IPython configuration files

The IPython config files are located in your IPython profile (in this example, the default profile) directory (depending on your installation, it can be at ~/.ipython/profile_default/ or ~/.config/ipython/profile_default/, ...). I'm talking about these files:

ipython_config.py
ipython_qtconsole_config.py (if you use the qtconsole)
ipython_notebook_config.py (if you use notebooks)

If you don't see them in that directory, you can create them by recreating the profile (still the default profile):

 ipython profile create 

You should now see and edit the config files.

It's very easy to print your Python path within Python but since I can never remember it, I thought I'd write it down:

 import sys print sys.path 

On the subject of imports and PYTHONPATH, here is a nicely-written article. Also, since I was having a few complications trying to make my PYTHONPATH working, I thought I'd write my set up (on my Mountain Lion MAC using bash). In my /private/etc/bashrc file or whichever bash startup file you prefer, I wrote:

 PYTHONPATH="/Users/johnsmith/python":$PYTHONPATH PYTHONPATH="/Users/johnsmith/python/myutils":$PYTHONPATH export PYTHONPATH 

Note that you should use the full path (no ~ or relative path), since python will not transform the path for you. That's why it did not work for me in the first place. In doubt, go print the sys.path in python and check whether your path is in there and properly written.

## Monday, February 18, 2013

### Redirecting Bash output to a file

Say you have a bash shell script and you want to have the output (stdout) and the error message (stderr) of a command written to a file. Here is how you should do it. I also found some useful information here.

 $mycommand 1> outputfile.txt # Writes stdout to outputfile.txt $ mycommand > outputfile.txt # Same as above
$mycommand >> outputfile.txt # Writes stdout to outputanderrorfile.txt but appends instead of overwriting $ mycommand 2> errorfile.txt # Writes stderr to errorfile.txt
$mycommand 2>> errorfile.txt # Writes stdout to errorfile.txt but appends instead of overwriting $ mycommand > outputanderrorfile.txt 2>&1 # Writes stdout and stderr to outputanderrorfile.txt
$mycommand &> outputanderrorfile.txt # Writes stdout and stderr to outputanderrorfile.txt $ mycommand >> outputanderrorfile.txt 2>&1 # Writes stdout and stderr to outputanderrorfile.txt but appends instead of overwriting

## Saturday, February 16, 2013

### Reading IDL save file in Python

See IDLSave module.

IDLSave has been included in Scipy since Scipy 0.9.0. Here is how to use it:

 from scipy.io.idl import readsav s = readsav('myfile.sav') 

## Wednesday, February 13, 2013

### Strings with zero-padded values and other string formatting examples in Python

Assume you want to convert a number to a string but you want to pad the smaller numbers with zeros so that all strings have the same length (for example, '0009', '0069' and '0132'). There are many ways to do this but in Python, there is a very elegant way using string formatting:

 anumber = 9 thestr = "%04d" % (anumber) print thestr 0009 anumber = 69 thestr = "%04d" % (anumber) print thestr 0069 anumber = 132 thestr = "%04d" % (anumber) print thestr 0132 anumber = 12541 thestr = "%04d" % (anumber) print thestr 12541 

Since the format() function is preferred (since Python 2.6 I think), you might want to use it instead (see here):

 anumber = 69 thestr = "{0:0>4}".format(anumber) print thestr 0069 thestr = "{0:0<4}".format(anumber) print thestr 6900 thestr = "{0:0<8}".format(anumber) print thestr 69000000 thestr = "{0:0^8}".format(anumber) print thestr 00069000 thestr = "{0:-^8}".format(anumber) print thestr ---69--- 

### Signal-to-noise calculations in photometry

In short,

• N(star) is the number of electrons from the star which fall within the aperture
• N(backpp) is the number of electrons Per Pixel
• N(thermpp) is the number of electrons Per Pixel
• R is the readout noise per pixel, in electrons
• npix is the number of pixels in the aperture

## Tuesday, February 12, 2013

### Notes on python modules and python scripts

It is very easy to make a python module: just write a bunch of functions and variables in a .py file, e.g. mymodule.py. You can then import it as a module within python with import mymodule.

As mentioned in the python documentation on modules: "Within a module, the module’s name (as a string) is available as the value of the global variable __name__".

When you run a python module as a script (python mymodule.py <arguments>), the code in the module will be executed, just as if you imported it, but with the __name__ set to "__main__". That means that by adding this code at the end of your module:

 if __name__ == "__main__":    my_command_line_1    my_command_line_2 

you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is executed as the “main” file. When you import the module, those lines are not executed.

Also, here is a nice post on python modules and PYTHONPATH.

Now that we can run our python module from the command line, let's make it nicer by adding the possibility to add arguments. You can find some nice posts about it. For example, this one. I'll just summarize.

You first need to define your options and keywords:

 letters = 'm:cv:G' # : means an argument needs to be passed after the letter keywords = ['month=','create','variable=','Group'] # = means an argument needs to be passed after the keyword 

Now use the getopt module to parse the arguments:

 import getopt import sys opts, extraparams = getopt.getopt(sys.argv[1:],letters,keywords) # keywords are optional # starts at the second element of argv since the first one is the script name # extraparms are extra arguments passed after all option/keywords are assigned # opts is a list containing the pair "option"/"value" 

 var='default' month=3 for o,val in opts:   if o in ['-v','--variable']:     var = val   elif o in ['-m','--model']:     month = int(val)   elif o in ['-c','--create']:     doSomething()   elif o in ['-G','--Group']:     doSomethingElse() 

So, in summary, here is how most python scripts should look like:

 if __name__ == "__main__":      letters = 'm:cv:G'   keywords = ['month=','create','variable=','Group']   import getopt   import sys   opts, extraparams = getopt.getopt(sys.argv[1:],letters,keywords)   var='default'   month=3   for o,val in opts:     if o in ['-v','--variable']:       var = val     elif o in ['-m','--model']:       month = int(val)     elif o in ['-c','--create']:       doSomething()     elif o in ['-G','--Group']:       doSomethingElse() 

Finally, to make your python script executable. Start the file with

#! /usr/local/bin/python

or whatever your python path is. You'll also need to change the permissions on the file (chmod +x mypythonscript.py).

## Monday, February 11, 2013

### Reading NIKON Raw files in Python

I wanted to read a Nikon raw file using Python. On the off chance that someone else wants to do it, I thought i'd write it down. Actually, it turns out that other people have done that. I found this awesome nef_decoder. Once you have downloaded all the files in that directory (I used SiteSucker to do that). You have to install the python module (see this post for details). If you're lucky, a simple  python setup.py install  will work. Because of cython compatibility issues, it didn't quite work. I got the following error message:

pixelutils.pyx:207:13: 'bool' is not a type identifier

Luckily for me, somebody had this problem before and I could easily solve it by adding a line at the top of pixelutils.pyc:

from cpython cimport bool

I added the line and restarted the install and it finished installing without any problem. I then opened IPython and entered the following code:

 import nef_decoder file = "myfile.NEF" md, mn, im = nef_decoder.decode_file(file)

That should have worked but it didn't. I got a KeyError on line
--> 295 info['img_orientation'] = ifd[img_orientation_tag_id][-1]
Since I was only interested in the image and not the EXIF tag about the image orientation. I have simply bypassed the problem (not pretty, I know) with Try and Except:

 try:    info['img_orientation'] = ifd[img_orientation_tag_id][-1] except:    info['img_orientation'] = 0 .

I tried again (reloaded the module and restarted decode_file) and it worked. I had a nice RGB image to work with. Thanks all for solving my problems today.

## Sunday, February 10, 2013

### Redirecting a website using .htaccess

A few weeks ago, I started working for a new institution. I needed to move my website from the old institution's web server to the new one, that is from http://old.institution.edu/~jspronck/webs/ to http://new.institution.edu/~spronck/.

For simple cases, you can get away with Redirect 301.

If you need something a little more sophisticated, you can use Apache's URL rewriting engine, which can do some fancy stuff. This link was also useful.

In my case, I ended up adding the following three lines in the .htaccess file (the file should be located in the directory of the site that needs redirection; if there is none, just create one with a text editor. Remember that files starting with . will automatically be hidden).

 RewriteEngine on RewriteRule .*Photography/(.*) http://new.institution.edu/~spronck/Photography/$1 [R=301,L] RewriteRule .* http://home.new.institution.edu/~spronck/Welcome.html [R=301,L]  The second line redirects all pages of the Photography directory into the same page of the photography directory of the new institution. The third line redirects any page of the directory where the .htaccess is saved to the Welcome.html page at the new institution. R=301 means that it is a permanent change and L is there to say that the URL will not be rewritten by future RewriteRule if the URL matches the rule. ### Positioning <div> tags with CSS Whenever I need to resfresh my memory on how to position <div> tags on a webpage, I come here. ## Saturday, February 9, 2013 ### Regular expression back-references in Python It's simple (assuming you know about regular expressions). Here is how. Say you have a string s containing a date of the format MM/DD/YYYY that you want to change into YYYY-MM-DD. You first compile your regular expression using re.compile, define your string s and use re.sub to replace the content of the string.  >>> import re >>> pat=re.compile(r’^(0?[1-9]|1[0-2])/(0?[1-9]|[12][0-9]|3[01])/(\d\d\d?\d?)$’) >>> s = ‘12/31/2021’ >>> re.sub(pat, r’\3-\1-\2′, s) ‘2021-12-31’ 

And that's it. Hope this is useful.

A small tool to manipulate jpeg header (e.g. autorotating or getting the image size), it is called jhead.

### String manipulation and parameter expansion in bash

Some examples with filenames:
 filename=$(basename "$fullfile") extension="${filename##*.}" filenamenoextension="${filename%.*}" 

### Starting a bash script

At the beginning of the file, write #!/bin/bash.

### Mastering .htaccess files

Here is some link on .htaccess. On Apache webservers, .htaccess files can be very handy (especially if you don't have admin rights on your webserver). Adding simple instructions in a file named .htaccess allows you to do many things. Among others:
• Redirecting a web page to another
• Handling errors and using custom-made error pages (for example, a 404 page like these)
• Password-protect a page or directory
• Prevent your directory to be indexed (and show its content to the world)

### Making beautiful plots in IDL

You know how IDL plots can look pretty bad. One day I came across this link. From that day on, I've made consistently nice looking eps figures with IDL. Thanks much, Alfred. I slightly modified the commands given in this link and wrapped it in a proplot.pro file that I call before calling my regular plotting commands.

 pro proplot, ps, font_size=font_size, xsize=xsize, ysize=ysize set_plot, 'ps' ;colors loadct, 39, /silent !p.color=0 !p.background=255 ;font !p.font = 0 !p.charsize=3. if ~keyword_set(font_size) then font_size=5.5 ;thickness !p.thick = 4 !x.thick = 4 !y.thick = 4 !z.thick = 4 ;figure sizes if ~keyword_set(xsize) then xsize = 8.8 margin = 0.15 wall = 0.07 a = 1. - (margin + wall) if ~keyword_set(ysize) then begin b = a * 2d / (1 + sqrt(5)) ysize = (margin + b + wall)*xsize end else b = ysize/8.8 - margin - wall ;plot position x1 = margin*8.8/xsize x2 = x1 + a*8.8/xsize y1 = margin*8.8/ysize y2 = y1 + b*8.8/ysize !p.position=[x1,y1,x2,y2] ;device device, filename=ps, font_size=font_size, /schoolbook, /color, xsize=2.5*xsize, ysize=2.5*ysize, /encapsulated end 

After finishing the plot, you need to close the file and possibly reverse all things you have changed (otherwise your non-PostScript plots will look weird). I wrote a wrapper for that:

 pro mypsclose if (!d.name ne 'PS') then begin message, 'DEVICE is not set to PS!', /INFO return endif ; CLOSE THE POSTSCRIPT DEVICE... device, /CLOSE_FILE ; SET THE GRAPHICS OUTPUT DEVICE TO X WINDOWS... set_plot, 'X' if keyword_set(color) then setcolors,/sys,/sil !p.font=1 device, retain=2, decomposed=0, /tt_font,set_character_size=[20, 24] defsysv,'!digits',strtrim(sindgen(10),2) loadct,39,/silent !p.background=255 !p.color=0 !p.position=[0.15,0.2,0.92,0.92] !p.thick = 0. !x.thick = 0. !y.thick = 0. !z.thick = 0. !p.charsize=0. end 

In IDL,

 IDL> proplot, 'testfile.eps' IDL> plot, ..., xtitle='!C The X Title', ... IDL> mypsclose 

Note that the "!C" in the xtitle is to add a carriage return at the beginning of the line. Otherwise, the title may be too close to the plot (or even on the plot).

### Adding TrueType or Postscript fonts to IDL

IDL plots can look quite ugly at times. One way to improve their look is by using other fonts than the default fonts. Here is how. I'll first tell you how to add a true type font and then a postscript font.
1. Find the .ttf file of your favorite true type font (ex: ilovethisfont.ttf)
2. Go to your idl directory (on a Mac, /Applications/itt/idlxx/) and then to resource/fonts/tt/
3. Open the file ttfont.map in a text editor
4. Add a line to it with the following information:
• the name that you want to give to your font (how you want to call it within IDL) between quotes (Ex: "This_is_my_font")
• the .ttf file name (ilovethisfont.ttf)
• two numbers that can be set to 1.0 (that have to do with the scaling of the font)
• the line should look like:
"This_is_my_font" ilovethisfont.ttf 1.0 1.0
5. Restart IDL
6. You can now use your font by
• setting your device to true type fonts: IDL> !p.font = 1
• changing the device font:
IDL> device, /tt_font, set_font='This_is_my_font'
However, for quality publications, postscript fonts are recommended. Personally, I use

 IDL> !p.font = 0 ;; Device fonts IDL> device, /schoolbook ;; New Century Schoolbook
This command is equivalent to
IDL> device, set_font='NewCenturySchlbk-Roman'

You can also add a Postscript font in a similar way by adding a line in the file /Applications/itt/idlxx/resource/fonts/ps/font.map (see existing lines)

## Friday, February 8, 2013

### zip, tar.gz, ...compressing and uncompressing

Here is a link with the syntaxes when archiving and unarchiving zip or tar.gz files.

### Installing python module from source

1. Download the module. It usually comes as a tar.gz or zip file (for example, markdown).
2. Uncompress the file (using The Unarchiver, winzip, the Linux command tar -zxvf archive_name.tar.gz, ...).
3. cd to the newly extracted directory. In this directory, there should be a file called setup.py

### On using IPython and Notebooks

Also some useful link for installing IPython and notebook (which can be non-trivial). It tells you that you need to install ipython, pyzmq, tornado and mathjax to have notebook working.

Since I always want pylab and inline plots when using notebooks, I use

### iWeb greyed out

I haven't been using iWeb much simply because I prefer a good old text editor for making web pages. Nevertheless, I wanted to check something out in iWeb. So, I opened it. Nothing happened and all menus were greyed out. The website I had worked on before was not visible. I didn't find the cause for this but here is one solution:

1) Close iWeb.

2) Go to this folder in your user Library folder (/Users/Your_User_Name/Library/Application Support/iWeb/). Now, in Mountain Lion, the user Library is hidden. Three possible solutions to this problem:
• From the Finder, menu "Go" > "Go To Folder...", type /Users/Your_User_Name/Library/Application Support/iWeb/ and hit "Go".
• From the Terminal, type
$open /Users/Your_User_Name/Library/Application\ Support/iWeb/ (don't type the dollar sign). • Unhide the user Library folder from the Terminal, $ chflags nohidden /Users/Your_User_Name/Library.

3) Now that you are in the /Users/Your_User_Name/Library/Application Support/iWeb/ folder, you might see a "Domain" file. Move it somewhere safe (don't delete it as it contains all the information about your existing site) like on the Desktop for example.

4) After moving the file, open iWeb.

5) It should now ask you what type of template and site do you want? Choose anything you want.

6) If you want to access your old site, close iWeb. Go back in the /Users/Your_User_Name/Library/Application Support/iWeb/ folder. Delete the newly created Domain file and then move back the old Domain file into this folder.

7) Open iWeb. It should work again.