Code-itch » Python

Boto : Super easy Python library to interact with amazon web services

harijay — Tue, 23 Nov 2010 12:54:48 +0000

I had first heard of Mitch Granaats boto thanks to “Monster Muck Mashup” where he used boto to build a video transcoding service using amazons compute and storage cloud EC2 and S3. Being a relative python newbie then, I remember reading the code examples and not understanding what they were doing entirely.

After a long period of dormancy I decided to resurrect and cleanup my S3 buckets. Stupid me! , I had turned on logging for the code-itch bucket ever so long ago. Result!,tens of thousands of log entries. Every S3 browser I tried, struggled to display and scroll the massive number of files. I needed a script and quick.

Initially I considered jets3t a java based super-library for all things AWS. But after reading an answer to this question on stackoverflow I decided to try boto again. This time around , I found the syntax very easy to comprehend. Within minutes I had a script that worked. The script read like pseudocode. I will reproduce it here.

I will definitely be delving more into boto to explore amazons many offerings.

Mitch Granaat also writes an excellent blog on all things aws and boto at elastician

Boto library home on googlecode

#!/usr/bin/python
import boto
from boto import *
import re
# Regexp to match the log file names
log_key = re.compile("code-itch\.[\d]{4}\-[\d]{1,2}\-[\d]{1,2}\-[\d]{1,2}\-[\d]{1,2}\-[\d]{1,2}\-[a-fA-F\d]{16}")
f = open("listbuckets_code-itch_deleted.txt" , "w")
if __name__ == '__main__':
s3 = boto.connect_s3()
mybucket = s3.get_bucket("code-itch")
for key in mybucket:
if log_key.match(key.name):
f.write(key.name + "\n")
mybucket.delete_key(key.name)
print "Deleted",key.name
f.close()

Python scripting from within Pymol

harijay — Sun, 21 Nov 2010 00:52:41 +0000

I have been using pymol for a while , but only recently did I start using python scripts to automate my tasks within pymol.

The extreme power of pymols API allows you to call python scripts from within a Pymol session as I have done in the screencast.

This text will be replaced

Additional Links and information:Scripting in Pymol

The code segment I read into the pymol session is given here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# My list of objects in pymol session
objs = ["3org","1ots","mypdb"]
# List to keep track of pairs that have been rendered
images_done = []
for i in objs:
for j in objs:
if i == j :
continue
else:
if (j,i) in images_done or (i,j) in images_done:
continue
else:
cmd.disable("all")
cmd.enable(i)
cmd.enable(j)
cmd.png("%s_%s_superpose3.png" % (i,j),640,480,72,1,1)
cmd.disable("all")
images_done.append((i,j))
print "all images rendered"

Ahh ! for argparse

harijay — Wed, 01 Jul 2009 15:17:29 +0000

Its been a while since I blogged . Just a lot happening on the crystallography side of things to allow me the time to blog.

I have also been coding a lot lately and have gotten started with some GUI writing in wxpython.

This post is about trying to get back into the groove by telling you about argparse . Having talked about optparse and command line parsing , I heard about argparse thanks to a talk that I caught by clicking a link on the Pycon2009 master schedule . This link has all the Pycon 2009 slides and video links in one place – a great resource.

For those interested in argparse check out the “Plenary Evening session ” on Sunday Mar 29th 1.20 pm at PyCon2009 where Steven Bethard talks about argparse.

The most important difference between argparse and optparse is that argparse provides better handling for positional arguments in addition to optional arguments . Argparse also provides automatically generated useage information and takes care of handling cases like when the user forgets to provide any arguments and prints the help information by default.

The key differences are summarized in the excellent documentation at this URL , so I will not rehash it . But I have happily switched over to argparse from optparse and hope argparse becomes part of the python standard libraries soon.

For some code examples check out my github repository especially the script maskconvert.py

XML parsing – python

harijay — Thu, 12 Apr 2007 17:17:29 +0000

[youtube=http://www.youtube.com/watch?v=L-_6tiTR8v0]

XML is a great way to organize information. I first learnt of the power of XML to systematize information when I used it to output a whole bunch of search results from NCBI in the Tinyseq XML format. Once I had this XML document , I could read it into Excel and then very easily analyze the information since it was nicely laid out as an Excel sheet.

Backpackit a service I use to take notes detailing my experimental research results
outputs all of the account data in XML format. Before I can move this data elsewhere , it helps for me to understand the data structure. So the first task I set out to do was to parse the XML output.

I decided to use Python for this , because I felt using Java here would be like using an elephant to crush a fly ( or whatever the expression is ). Also a lot of the data is text , and I always used perl previously to handle text. So a general basis for my codeitch will be What I did in Perl before I wold like to do in Python now. Java will be used once for more heavyweight tasks.

What I needed my program to do was :

Read the XML output
Create objects for each element or node in the output

I can then imagine that once I have these objects I can ask questions like how many objects have embedded images , how many objects have outgoing links etc etc..

The “Dive into python “ book gave me a quick introduction into the xml.dom package. I then ran into some encoding or codec issues and learnt all about “utf8″ and “iso8859″ character encoding. Once I learnt how to handle the UnicodeEncodeError , I had a full fledged three line program that parsed my input file , created the document object and as proof of successful parsing and printed my XML file back out.

The screencast above documents my travails.

Codeitch- or the desire to codeify and systematize everything I do

harijay — Wed, 11 Apr 2007 08:52:18 +0000

This blog is roughly about my attempts to codify , algorithmize and systematize everything I do. In it I will hopefully detail my march to coding and getting proficient in a bunch of computer languages. After a long process of looking around , I have narrowed my focus to the following 3 languages in no particular order.
Java , Python and Javascript.

The reasons for these will hopefully emerge as I begin posting. But I will try and spell them out here

Java :I like java for two reasons , its one of the most widely used languages in the enterprise space and the second and very important reason are the Java IDEs. Both java IDEs I use namely Eclipse and Netbeans are Free and amazingly featured. Code prompting available in both IDEs make mastering an API a lot easier than learning the same functionality using other languages or platforms. Also, I love Javadoc !. It really makes picking up new APIs a little easier

Python : My first crack at automating anything came with Perl scripting. I will not lie if I say that If I have to do anything today I will first use Perl. But after several year of Perl use I found I was re-using very little code. I have to get more object oriented in the way I code, and since I never quite got a hang of Perl objects , and its namespace conventions!. Python which is at its heart a purely object oriented scripting language with libraries that easily rival Java was a natural scripting alternative.Learning Python I hope will teach me how to script smart objects that will beg to be reused.

Javascript : This is a surprising bedfellow to my codeitch. I want to learn javascript simply because it is becoming very fashionable. Google Maps and gmail have AJAX at their core and Javascript is the J in AJAX. Plus I have always fancied having a web frontend to everything I do and I am sure Javascript will beg to be used when that happens.

The above are the three languages I want to master.

Apart from these there are two platforms that I want to get comfortable with and they are

Excel : Everyone in the business world uses Excel. Spreadsheets were the PCs killer App and Excel is in my mind microsofts great product. I have seen the amazing things ou can do in Excel without writing a single line of code, and I want to learn to use its power.

Matlab: This platform from Mathworks is the bread and butter of engineering computation. I am anything but an engineer but have seen matlabs power when it comes to simulations. A lot of the very academic questions that I have in my research can really benefit from learning the Matlab platform and no I am not fully convinced on why exactly I need to use Matlab, maybe I will find a more concrete reason.

refs: My tumblr feed