Thursday, February 28, 2013

Setting up your Windows 7 machine for Python scientific work

A few days ago, I wrote a blog about setting up Python and (many science related modules) on a Mac. Well, today, I had to do the same on a windows 7 machine that I have to use for work. I never thought I'd say this but it turns out it was easy on windows (way easier than on my Mac ... I feel pain just writing this, and no, it won't make me like Windows). Here is how I did it.

Installing Python

I started the installation by following instructions from this link, which basically told me to install Python from the official Python page (apparently, it was recommended to install the 32 bit version ... even if my computer is 64 bit). I installed Python 2.7.3. It downloads an installer, which you simply open and follow the instructions.

The installer will not automatically set-up your system path to use python, so we have to do it manually.

  • Right-click Computer and select Properties.
  • In the dialog box, select Advanced System Settings.
  • In the next dialog, select Environment Variables.
  • In the User Variables section, edit the PATH statement (or create a PATH variable if it does not exist) to include this:

C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;

Once the PATH is set up, you should be able to run python from the command line (open Start Menu > All Programs > Accessories > Command Prompt and type python).

Installing Python modules

  • setuptools: installs easy_install. Download the setuptools installer here and simply run it.
  • pip: from the command prompt => easy_install pip
  • numpy: Download the installer here and run it.
  • ipython: last but not least, download ipython here (you'll need admin rights if you want the installer to put a item in the start menu)
  • pyfits: pip install pyfits
  • astropysics: pip install astropysics
  • pyephem: easy_install pyephem
  • lmfit: pip install lmfit

Tuesday, February 26, 2013

Setting up your mac for Python scientific work

Alright, I've just received my new iMac at work. So, I've spent some decent amount of time setting Python and modules I commonly work with. Since, the process can be rather tedious and painful, I thought I'd put some details here. This post is about how to set up Python and some scientific python modules on your Mac. Here is a list of what we will install: python (2.7.3), numpy, scipy, matplotlib, pil, pyfits, lmfit, astropysics, pyephem, ipython, qtconsole and notebook.
When I had to do it myself, I found this link very useful (this post is basically a summary of that one, with some small variations). Also, here's a newer version of the same link.

homebrew

So, you got your brand new shiny Mac (running Mountain Lion). The first thing you want to do is install homebrew, which simply is the least painful package manager that currently exists for Mac OS. You'll install it by opening up a terminal window and typing:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

You need this for all steps in this post, so if it doesn't work, you're screwed. Well, not entirely, you can look here for some help if it doesn't install. If it does install, you want to call the Doctor and check that everything is working before you install any package:

brew doctor

This will basically check that everything you need to install new packages is working fine. Do whatever the Doctor says. He or She will tell you things like "this program is outdated, install a newer version or make sure that this is on your PATH, ...". It doesn't always tell you how to solve the problem but gives you good clues.
Then, you should make sure the directory where homebrew installs program (/usr/local/bin/ if you followed standard homebrew installation) is on your path. Type echo $PATH in the terminal. If you don't see /usr/local/bin/, add a line in your ~/.profile or ~/.bash-profile (create the file if it does not exist). That line should be:

export PATH="/usr/local/bin":$PATH

If /usr/local/bin/ was on your path but after /usr/bin/, you should edit the files that set up the path in the first place. You can rearrange the lines of the /etc/paths. If that doesn't work, check this post.
We will be installing various programs that are currently not in the default brew distribution (tap). So, "tap" the following distributions:

brew tap home-brew/science
brew tap samueljohn/python



python

Alright, you have brew working and your path is set, now let's start installing things. Lots of the software we will install work best you re-install python using homebrew. Yes, I know. You already have python, why would you install it again? To avoid pain (check here for actual reasons and details). To install Python (2.7.3 at the time I wrote this post), type in a new terminal window (to refresh the path):

brew install python --framework --universal

Again, make sure /usr/local/share/python is in your path and add it if not (export PATH=/usr/local/share/python:$PATH). Reload Terminal after any change. After installing Python, you should tell your system that the newly installed version is the one that you want to use by default. To do that, we will create a symbolic link to the desired version of Python:

cd /System/Library/Frameworks/Python.framework/Versions
sudo rm Current
ln -s /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/Current


If it works, typing which python should return /usr/local/bin/python. You might have to restart your terminal. Furthermore, if you type python, you should see "Python 2.7.3".

Python modules

Install these. One at a time. Note that I had a few problems while installing numpy and PIL because of conflicting older files. brew told me what to do though. I had to link numpy and PIL manually (brew link numpy --overwrite), which overwrote the conflicting files.

brew install numpy # Installs numpy
brew install gfortran # Fortran compiler for scipy
brew install scipy # Installs scipy
brew install matplotlib # Installs matplotlib
brew install pil # Installs PIL
pip install pyfits # Installs pyfits
pip install astropysics # Installs astropysics
pip install pyephem # Installs pyephem
pip install lmfit # Installs lmfit


IPython, QTConsole and notebooks

pip install ipython # Installs IPython
brew install pyqt # Installs pyqt
brew install zmq
pip install pyzmq
pip install pygments


You should now be able to launch the qtconsole by executing:

ipython qtconsole --pylab=inline

For installing notebook, see this link.

ssh configuration file

Here is an excellent post on how to configure your machine, so you won't have to remember host names, logins, and various preferences for your favorite ssh servers.

Thursday, February 21, 2013

IPython configuration files

The IPython config files are located in your IPython profile (in this example, the default profile) directory (depending on your installation, it can be at ~/.ipython/profile_default/ or ~/.config/ipython/profile_default/, ...). I'm talking about these files:

ipython_config.py
ipython_qtconsole_config.py (if you use the qtconsole)
ipython_notebook_config.py (if you use notebooks)

If you don't see them in that directory, you can create them by recreating the profile (still the default profile):

ipython profile create

You should now see and edit the config files.

Printing your Python path

It's very easy to print your Python path within Python but since I can never remember it, I thought I'd write it down:

import sys
print sys.path


On the subject of imports and PYTHONPATH, here is a nicely-written article. Also, since I was having a few complications trying to make my PYTHONPATH working, I thought I'd write my set up (on my Mountain Lion MAC using bash). In my /private/etc/bashrc file or whichever bash startup file you prefer, I wrote:

PYTHONPATH="/Users/johnsmith/python":$PYTHONPATH
PYTHONPATH="/Users/johnsmith/python/myutils":$PYTHONPATH
export PYTHONPATH


Note that you should use the full path (no ~ or relative path), since python will not transform the path for you. That's why it did not work for me in the first place. In doubt, go print the sys.path in python and check whether your path is in there and properly written.

Monday, February 18, 2013

Redirecting Bash output to a file

Say you have a bash shell script and you want to have the output (stdout) and the error message (stderr) of a command written to a file. Here is how you should do it. I also found some useful information here.

$ mycommand 1> outputfile.txt # Writes stdout to outputfile.txt
$ mycommand > outputfile.txt # Same as above
$ mycommand >> outputfile.txt # Writes stdout to outputanderrorfile.txt but appends instead of overwriting
$ mycommand 2> errorfile.txt # Writes stderr to errorfile.txt
$ mycommand 2>> errorfile.txt # Writes stdout to errorfile.txt but appends instead of overwriting
$ mycommand > outputanderrorfile.txt 2>&1 # Writes stdout and stderr to outputanderrorfile.txt
$ mycommand &> outputanderrorfile.txt # Writes stdout and stderr to outputanderrorfile.txt
$ mycommand >> outputanderrorfile.txt 2>&1 # Writes stdout and stderr to outputanderrorfile.txt but appends instead of overwriting

Saturday, February 16, 2013

Reading IDL save file in Python

See IDLSave module.

IDLSave has been included in Scipy since Scipy 0.9.0. Here is how to use it:

from scipy.io.idl import readsav
s = readsav('myfile.sav')

Wednesday, February 13, 2013

Strings with zero-padded values and other string formatting examples in Python

Assume you want to convert a number to a string but you want to pad the smaller numbers with zeros so that all strings have the same length (for example, '0009', '0069' and '0132'). There are many ways to do this but in Python, there is a very elegant way using string formatting:

anumber = 9
thestr = "%04d" % (anumber)
print thestr
0009

anumber = 69
thestr = "%04d" % (anumber)
print thestr
0069

anumber = 132
thestr = "%04d" % (anumber)
print thestr
0132

anumber = 12541
thestr = "%04d" % (anumber)
print thestr
12541


Since the format() function is preferred (since Python 2.6 I think), you might want to use it instead (see here):

anumber = 69
thestr = "{0:0>4}".format(anumber)
print thestr
0069

thestr = "{0:0<4}".format(anumber)
print thestr
6900

thestr = "{0:0<8}".format(anumber)
print thestr
69000000

thestr = "{0:0^8}".format(anumber)
print thestr
00069000

thestr = "{0:-^8}".format(anumber)
print thestr
---69---

Signal-to-noise calculations in photometry

See this link.

In short,



  • N(star) is the number of electrons from the star which fall within the aperture
  • N(backpp) is the number of electrons Per Pixel
  • N(thermpp) is the number of electrons Per Pixel
  • R is the readout noise per pixel, in electrons
  • npix is the number of pixels in the aperture

Tuesday, February 12, 2013

Notes on python modules and python scripts

It is very easy to make a python module: just write a bunch of functions and variables in a .py file, e.g. mymodule.py. You can then import it as a module within python with import mymodule.

As mentioned in the python documentation on modules: "Within a module, the module’s name (as a string) is available as the value of the global variable __name__".

When you run a python module as a script (python mymodule.py <arguments>), the code in the module will be executed, just as if you imported it, but with the __name__ set to "__main__". That means that by adding this code at the end of your module:

if __name__ == "__main__":
   my_command_line_1
   my_command_line_2


you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is executed as the “main” file. When you import the module, those lines are not executed.

Also, here is a nice post on python modules and PYTHONPATH.

Now that we can run our python module from the command line, let's make it nicer by adding the possibility to add arguments. You can find some nice posts about it. For example, this one. I'll just summarize.

You first need to define your options and keywords:

letters = 'm:cv:G' # : means an argument needs to be passed after the letter
keywords = ['month=','create','variable=','Group'] # = means an argument needs to be passed after the keyword


Now use the getopt module to parse the arguments:

import getopt
import sys
opts, extraparams = getopt.getopt(sys.argv[1:],letters,keywords) # keywords are optional
# starts at the second element of argv since the first one is the script name
# extraparms are extra arguments passed after all option/keywords are assigned
# opts is a list containing the pair "option"/"value"


Then loop through your options:

var='default'
month=3
for o,val in opts:
  if o in ['-v','--variable']:
    var = val
  elif o in ['-m','--model']:
    month = int(val)
  elif o in ['-c','--create']:
    doSomething()
  elif o in ['-G','--Group']:
    doSomethingElse()


So, in summary, here is how most python scripts should look like:

if __name__ == "__main__":
  
  letters = 'm:cv:G'
  keywords = ['month=','create','variable=','Group']

  import getopt
  import sys
  opts, extraparams = getopt.getopt(sys.argv[1:],letters,keywords)

  var='default'
  month=3
  for o,val in opts:
    if o in ['-v','--variable']:
      var = val
    elif o in ['-m','--model']:
      month = int(val)
    elif o in ['-c','--create']:
      doSomething()
    elif o in ['-G','--Group']:
      doSomethingElse()


Finally, to make your python script executable. Start the file with

#! /usr/local/bin/python

or whatever your python path is. You'll also need to change the permissions on the file (chmod +x mypythonscript.py).

Monday, February 11, 2013

Reading NIKON Raw files in Python

I wanted to read a Nikon raw file using Python. On the off chance that someone else wants to do it, I thought i'd write it down. Actually, it turns out that other people have done that. I found this awesome nef_decoder. Once you have downloaded all the files in that directory (I used SiteSucker to do that). You have to install the python module (see this post for details). If you're lucky, a simple python setup.py install will work. Because of cython compatibility issues, it didn't quite work. I got the following error message:

pixelutils.pyx:207:13: 'bool' is not a type identifier

Luckily for me, somebody had this problem before and I could easily solve it by adding a line at the top of pixelutils.pyc:

from cpython cimport bool

I added the line and restarted the install and it finished installing without any problem. I then opened IPython and entered the following code:

import nef_decoder
file = "myfile.NEF"
md, mn, im = nef_decoder.decode_file(file)


That should have worked but it didn't. I got a KeyError on line
--> 295 info['img_orientation'] = ifd[img_orientation_tag_id][-1]
Since I was only interested in the image and not the EXIF tag about the image orientation. I have simply bypassed the problem (not pretty, I know) with Try and Except:

try:
   info['img_orientation'] = ifd[img_orientation_tag_id][-1]
except:
   info['img_orientation'] = 0
.

I tried again (reloaded the module and restarted decode_file) and it worked. I had a nice RGB image to work with. Thanks all for solving my problems today.

Sunday, February 10, 2013

Redirecting a website using .htaccess

A few weeks ago, I started working for a new institution. I needed to move my website from the old institution's web server to the new one, that is from http://old.institution.edu/~jspronck/webs/ to http://new.institution.edu/~spronck/.

For simple cases, you can get away with Redirect 301.

If you need something a little more sophisticated, you can use Apache's URL rewriting engine, which can do some fancy stuff. This link was also useful.

In my case, I ended up adding the following three lines in the .htaccess file (the file should be located in the directory of the site that needs redirection; if there is none, just create one with a text editor. Remember that files starting with . will automatically be hidden).

RewriteEngine on
RewriteRule .*Photography/(.*) http://new.institution.edu/~spronck/Photography/$1 [R=301,L]
RewriteRule .* http://home.new.institution.edu/~spronck/Welcome.html [R=301,L]


The second line redirects all pages of the Photography directory into the same page of the photography directory of the new institution. The third line redirects any page of the directory where the .htaccess is saved to the Welcome.html page at the new institution. R=301 means that it is a permanent change and L is there to say that the URL will not be rewritten by future RewriteRule if the URL matches the rule.

Positioning <div> tags with CSS

Whenever I need to resfresh my memory on how to position <div> tags on a webpage, I come here.

Saturday, February 9, 2013

Regular expression back-references in Python

It's simple (assuming you know about regular expressions). Here is how. Say you have a string s containing a date of the format MM/DD/YYYY that you want to change into YYYY-MM-DD. You first compile your regular expression using re.compile, define your string s and use re.sub to replace the content of the string.

>>> import re
>>> pat=re.compile(r’^(0?[1-9]|1[0-2])/(0?[1-9]|[12][0-9]|3[01])/(\d\d\d?\d?)$’)
>>> s = ‘12/31/2021’
>>> re.sub(pat, r’\3-\1-\2′, s)
‘2021-12-31’


And that's it. Hope this is useful.

Exif Jpeg header manipulation tool: jhead

A small tool to manipulate jpeg header (e.g. autorotating or getting the image size), it is called jhead.

String manipulation and parameter expansion in bash

Here is the link.

Some examples with filenames:
filename=$(basename "$fullfile")
extension="${filename##*.}"
filenamenoextension="${filename%.*}"

Starting a bash script

At the beginning of the file, write #!/bin/bash.

Mastering .htaccess files

Here is some link on .htaccess. On Apache webservers, .htaccess files can be very handy (especially if you don't have admin rights on your webserver). Adding simple instructions in a file named .htaccess allows you to do many things. Among others:
  • Redirecting a web page to another
  • Handling errors and using custom-made error pages (for example, a 404 page like these)
  • Password-protect a page or directory
  • Prevent your directory to be indexed (and show its content to the world)

Making beautiful plots in IDL

You know how IDL plots can look pretty bad. One day I came across this link. From that day on, I've made consistently nice looking eps figures with IDL. Thanks much, Alfred. I slightly modified the commands given in this link and wrapped it in a proplot.pro file that I call before calling my regular plotting commands.

pro proplot, ps, font_size=font_size, xsize=xsize, ysize=ysize

set_plot, 'ps'

;colors
loadct, 39, /silent
!p.color=0
!p.background=255

;font
!p.font = 0
!p.charsize=3.
if ~keyword_set(font_size) then font_size=5.5

;thickness
!p.thick = 4
!x.thick = 4
!y.thick = 4
!z.thick = 4

;figure sizes
if ~keyword_set(xsize) then xsize = 8.8
margin = 0.15
wall = 0.07
a = 1. - (margin + wall)

if ~keyword_set(ysize) then begin
b = a * 2d / (1 + sqrt(5))
ysize = (margin + b + wall)*xsize
end else b = ysize/8.8 - margin - wall

;plot position
x1 = margin*8.8/xsize
x2 = x1 + a*8.8/xsize
y1 = margin*8.8/ysize
y2 = y1 + b*8.8/ysize
!p.position=[x1,y1,x2,y2]

;device
device, filename=ps, font_size=font_size, /schoolbook, /color, xsize=2.5*xsize, ysize=2.5*ysize, /encapsulated

end


After finishing the plot, you need to close the file and possibly reverse all things you have changed (otherwise your non-PostScript plots will look weird). I wrote a wrapper for that:

pro mypsclose

if (!d.name ne 'PS') then begin
message, 'DEVICE is not set to PS!', /INFO
return
endif

; CLOSE THE POSTSCRIPT DEVICE...
device, /CLOSE_FILE

; SET THE GRAPHICS OUTPUT DEVICE TO X WINDOWS...
set_plot, 'X'
if keyword_set(color) then setcolors,/sys,/sil

!p.font=1

device, retain=2, decomposed=0, /tt_font,set_character_size=[20, 24]

defsysv,'!digits',strtrim(sindgen(10),2)

loadct,39,/silent
!p.background=255
!p.color=0

!p.position=[0.15,0.2,0.92,0.92]
!p.thick = 0.
!x.thick = 0.
!y.thick = 0.
!z.thick = 0.
!p.charsize=0.

end


In IDL,

IDL> proplot, 'testfile.eps'
IDL> plot, ..., xtitle='!C The X Title', ...
IDL> mypsclose


Note that the "!C" in the xtitle is to add a carriage return at the beginning of the line. Otherwise, the title may be too close to the plot (or even on the plot).

Adding TrueType or Postscript fonts to IDL

IDL plots can look quite ugly at times. One way to improve their look is by using other fonts than the default fonts. Here is how. I'll first tell you how to add a true type font and then a postscript font.
  1. Find the .ttf file of your favorite true type font (ex: ilovethisfont.ttf)
  2. Go to your idl directory (on a Mac, /Applications/itt/idlxx/) and then to resource/fonts/tt/
  3. Open the file ttfont.map in a text editor
  4. Add a line to it with the following information:
    • the name that you want to give to your font (how you want to call it within IDL) between quotes (Ex: "This_is_my_font")
    • the .ttf file name (ilovethisfont.ttf)
    • two numbers that can be set to 1.0 (that have to do with the scaling of the font)
    • the line should look like:
      "This_is_my_font" ilovethisfont.ttf 1.0 1.0
  5. Restart IDL
  6. You can now use your font by
    • setting your device to true type fonts: IDL> !p.font = 1
    • changing the device font:
      IDL> device, /tt_font, set_font='This_is_my_font'
However, for quality publications, postscript fonts are recommended. Personally, I use

IDL> !p.font = 0 ;; Device fonts
IDL> device, /schoolbook ;; New Century Schoolbook

This command is equivalent to
IDL> device, set_font='NewCenturySchlbk-Roman'

You can also add a Postscript font in a similar way by adding a line in the file /Applications/itt/idlxx/resource/fonts/ps/font.map (see existing lines)

Friday, February 8, 2013

zip, tar.gz, ...compressing and uncompressing

Here is a link with the syntaxes when archiving and unarchiving zip or tar.gz files.

Installing python module from source

  1. Download the module. It usually comes as a tar.gz or zip file (for example, markdown).
  2. Uncompress the file (using The Unarchiver, winzip, the Linux command tar -zxvf archive_name.tar.gz, ...).
  3. cd to the newly extracted directory. In this directory, there should be a file called setup.py
  4. Type $ python setup.py install.
  5. If you do not have admin rights, you might need to specify a folder where the module will be installed $ python setup.py install --home=path/to/new/INSTALL_DIRECTORY/. In addition, the specified INSTALL_DIRECTORY must be added to the python path. In cshell, this can be done for example by adding a line in the .cshrc file (setenv PYTHONPATH path/to/new/INSTALL_DIRECTORY:${PYTHONPATH}
  6. )

IPython startup script

You can have scripts run automatically when starting IPython by placing startup .py files in ~/.ipython/profile_default/startup/. All files in that directory will be read and executed by alphabetical order. On my Mac, the startup directory is at ~/.config/ipython/profile_default/startup/.

Exporting IPython Notebooks

A few things to know to export IPython notebooks.
  1. To export an existing notebook into a .py file, you can use the IPython magic command %notebook:
    In [1]: %notebook -f py your_notebook.ipynb
  2. To export a notebook into static html page, you can use nbconvert. To install nbconvert, you'll need a few things:
    After installing all these, I had a couple errors that I fixed/bypassed before I could make an html page out of my .ipynb file:
    • In ./converters/utils.py, I replaced the line "from IPython.nbformat.v3.nbjson import BytesEncoder" by "from IPython.nbformat.v2.nbjson import BytesEncoder". That's because IPython.nbformat.v3 did not exist in my older version of IPython. This is clearly not the best way to solve it but just a way around.
    • I was missing the fbm.css file in your_ipython_package_directory/frontend/html/notebook/static/css/
    • . I found the file online. If you do not have admin rights, put the file somewhere else and write the correct path to it in ../converters/html.py.

    The following command can then be used to convert a .ipynb file to html:
    $ ./nbconvert.py -f html your_notebook.ipynb

On using IPython and Notebooks

Click here for Notebook basics!

Also some useful link for installing IPython and notebook (which can be non-trivial). It tells you that you need to install ipython, pyzmq, tornado and mathjax to have notebook working.

Since I always want pylab and inline plots when using notebooks, I use

$ ipython notebook --pylab inline (for which I have set up an alias in my .cshrc file).

Unhide hidden folder

Some folders are visible, some are hidden. For example, in Mac OS X Mountain Lion the user Library folder is hidden. To unhide a hidden folder, type in the Terminal:

$ chflags nohidden Path/To/The/Folder

iWeb greyed out

I haven't been using iWeb much simply because I prefer a good old text editor for making web pages. Nevertheless, I wanted to check something out in iWeb. So, I opened it. Nothing happened and all menus were greyed out. The website I had worked on before was not visible. I didn't find the cause for this but here is one solution:

1) Close iWeb.

2) Go to this folder in your user Library folder (/Users/Your_User_Name/Library/Application Support/iWeb/). Now, in Mountain Lion, the user Library is hidden. Three possible solutions to this problem:
  • From the Finder, menu "Go" > "Go To Folder...", type /Users/Your_User_Name/Library/Application Support/iWeb/ and hit "Go".
  • From the Terminal, type
    $ open /Users/Your_User_Name/Library/Application\ Support/iWeb/
    (don't type the dollar sign).
  • Unhide the user Library folder from the Terminal,
    $ chflags nohidden /Users/Your_User_Name/Library.

3) Now that you are in the /Users/Your_User_Name/Library/Application Support/iWeb/ folder, you might see a "Domain" file. Move it somewhere safe (don't delete it as it contains all the information about your existing site) like on the Desktop for example.

4) After moving the file, open iWeb.

5) It should now ask you what type of template and site do you want? Choose anything you want.

6) If you want to access your old site, close iWeb. Go back in the /Users/Your_User_Name/Library/Application Support/iWeb/ folder. Delete the newly created Domain file and then move back the old Domain file into this folder.

7) Open iWeb. It should work again.