I read the SpringPython Book about a month ago, and liked it, so about two weeks ago, started building a small web application with it in an attempt to learn more about it. I have never used Python to build anything larger than a script, so this was a first for me. One reason I haven't even tried to do a Python webapp before is that we are a Java shop, and maintainability becomes an issue. In this instance, however, I think I have a use case which just feels more natural to solve with Python than with Java.
The use case is exposing expensive computing resources to the users via a webapp. We have batch jobs that run for long periods and are fairly memory and processor intensive. Developing and running these jobs are restricted to a few people who are given access to large fast machines (or clusters in some cases). The downside (for these supposedly lucky folks) is that they often have to run jobs (often at odd hours) for people who dont have access. The downside for the others is that lot of good ideas die because of the limited scope for experimentation (having to ask other people to run stuff for you repeatedly can be a real initiative-killer).
The app that I built (and am about to describe) provides a simple web interface for people to submit jobs. A job is basically a Unix shell script (which would call a Java class), its input file(s), configuration parameters and output file. Jobs are queued up in a database. A server component calls the selected script and runs it on the user's behalf, reporting progress of the job via email.
ScriptRunner.py Contains the main method to launch the webapp - its name is derived from the application name given to coily.
app_context.py Contains the objects in the application context, annotated as SpringPython @Object.
view.py Contains the CherryPy "controller" objects annotated by @cherrypy.expose.
ScriptRunner.py Main method with option to switch between web client, server and cleaner mode.
app_context.py Contains a (2 level deep) hierarchy of application context objects. Each child application context is customized to the modes that ScriptRunner can run in (listed above).
server.py Contains the top level classes for running server and cleaner modes.
services.py Contains classes for various services. These services encapsulate functionality that are provided as service objects in the context and are injected into the top level objects.
view.py Contains the top level class for running the web mode. Similar to the view.py generated, except it has more @cherrypy.exposed methods, and I have moved out non HTML rendering code (as far as possible) into the service methods.
The generated coily app only provides code for a web application, but I needed code to also do the server portion (to consume and process the queue), and to do cleanup (delete old files and database entries, scheduled via cron) in addition to that. So I made some changes in the ScriptRunner.py file to use options to run in different modes. Different modes need to run as different processes, ie, the server is a completely different process from the web client and needs to be started separately. Here is the usage output.
Coily's generated webapp comes with a In-Memory User Details Service which is populated at startup from a dictionary. The model I wanted for my application was a self-service model - there is no concept of administrators or super-users. People sign up and login with their email/password. Login is required for identifying who to send the email notifications to. Because of this, I decided to populate the In-Memory User Details object with the contents of a database table, rather than go with the Database User Details object.
I wanted this app to be standalone, ie, with as few moving parts as possible. So I decided a embed a SQLite3 database inside the application. One problem with that, however, was that SQLite3 does not allow a database connection to be shared by multiple threads. Obviously, this doesn't work if a database connection is created by the SpringPython container and then used by a request thread. However, this check can be disabled by passing a flag during connection factory construction. As a temporary measure, I added this in to the SpringPython code and reported it, and SpringPython 1.2 will allow this.
The DDL for the user and job tables are shown below. Currently, I don't have code to autodetect the absence of the DDL file and generate these tables automatically, but I will probably put this in.
The generated coily app contained a single Config object. Since I was essentially building three related components, each with a overlapping but different set of objects in their Config objects, I ended up building 4 Config objects, like so. Each config object corresponds to one of the ScriptRunner modes.
The generated coily app contained an empty controller.py file. The view.py file seemed to me to be the closest (Spring-Java) analog to a controller. I guess since we use Python for the view layer, there is really no necessity for a separate controller layer. However, I noticed that the view.py became "less controller-like" if I refactored some of the code into a services layer, so I created a services.py file and deleted the controller.py file.
In order for ScriptRunner to figure out what information is needed to be passed into a job, either the scripts need to be self contained and expose an uniform interface, or (the approach I have chosen), describe their interface using a special annotation. For example, here is an example from one of my test scripts. The code parses parameter information from the lines with the "##~~" prefix (configurable via the conf file, see above).
Inline comments are provided where I thought it made sense. All methods that coily generated are marked as such. The ScriptRunnerCommonConfig contains all the references to ScriptRunner core classes.
"""This is the main object defined for the web application."""
its parent, an InMemoryUserDetails object.
""" Autogenerated by Coily """
""" added /signup.* because that needs to be open too """
""" main server object """
""" main cleaner object """
This is my attempt to refactor some of the view.py code into a services layer.
from an embedded SQLite3 database table.
valid user from the dictionary.
Convenience method to validate that a password is long enough.
We can get more fancy here if we need to.
""" Returns True if there are no jobs to process """
select id from jobs where status = "New"
""" Add a new element into the queue """
""" Get the next job from the queue and mark it Started """
""" returns a list of queue elements """
""" Remove job from queue when complete (mark it complete) """
""" Remove all completed jobs with completion dates older than cutoff """
readable ISO-8601 format for display.
and parse script and input files from the local file system.
##~~ output - the output file, etc.
""" delete files in a directory that are older than cutoff """
contents of email to STDOUT (if mock=True).
""" Sends start and completion messages """
Your Job (Job# %d) has started at %s.
Your Job (Job# %d) completed at %s.
print "---- Sending message ----"
This corresponds to the Controller and JSTL layer in the Spring Java world. Each method annotated with @cherrypy.expose corresponds to a page.
""" Standard header used for all pages """
""" Standard footer used for all pages """
"""Presentation layer of the web application."""
""" Controller for login and view for login/signup page """
errorMsg = "Bad email/password, please try again"
""" validate the inputs """
errorMsg = "<br/>Malformed Email Address"
errorMsg += "<br/>Password too short (< 8 chars)"
""" redirect to login page """
""" controller for index page (documentation) """
resource-intensive jobs on a large machine (or cluster of machines).
""" controller for job submission """
""" generate wizard pages based on pageNum """
""" save the job into the database """
""" controller for listing all submitted jobs """
""" controller to log out user """
"""Authenticate a new username/password pair using the authentication manager."""
""" Generate select widget to allow selection from collection """
""" Generate table widget to input script parameters """
""" Generate list of parameter (name,value) from request params """
""" Format list of params to different formats (HTML, CSV) """
Like the view.py contains the ScriptRunnerView class for the web client, this file contains the top level classes for the server and cleaner components.
This is the main class that is called from the command line. The original code has been changed to add and parse options, and to switch between three modes of operation. Also some code has been factored out to the services layer.
""" run in web client mode """
""" set up security """
""" run in server mode """
""" scheduled from cron to periodically delete old files and job entries """
print "Select one of --web, --server, or --cleanup"
Here are some screenshots of the web client, showing how someone would sign up or login, submit a job, then view the status of the job.
Login/Sign up page. New users sign in and are immediately logged in (ie without a separate login step). Existing users log in with their email address and password.
Logged in users are sent to an index page which provides a little documentation about the app, and what you can do with it. Now that we recognize the user, links to various pages are exposed in the footer. The header now contains a message showing that me@mycompany.com has logged in.
User clicks Submit Job. This is a 3 page wizard that will lead the user through the steps to submit the job. The first step is to scp your input file over to the box, then choose the script to run.
User provides parameters to the script. Any parameters whose names begin with the configured input prefix (currently "input") have the drop down containing all the files from the inbox - one or more of which the user would have scp-ed into the application's inbox.
Submission is successful! At this point the job is queued up for processing.
User clicks on the "check status" link in the last page, or the "View Jobs" link in the footer. The default view is to show all jobs, but you can also filter for your own jobs using the tab at the top of the page. In this case both are the same.
Your Job (Job# 1) has started at 2010-08-29 09:00:12.
...and one when it completes.
Your Job (Job# 1) completed at 2010-08-29 09:00:12.
After the start and end of each job, the timestamps and current job status are updated. Here is what the View Jobs page looks like after the job is completed. As mentioned above, currently me@mycompany.com is the only user configured, so the two tabs on this page show the same results.
Coily provided a working archetype - this helped immensely. Building a webapp involves many components, and having to only work on the application can be a huge time (and effort) saver.
Dependency Injection - Since I was familiar with Java Spring, using SpringPython's DatabaseTemplate and @transactional annotations, and even the concepts of ApplicationContext, etc, felt very familiar and intuitive to use.