Tuesday, January 13, 2009

Django on OpenBSD

This is an article about setting up the Python web framework Django in OpenBSD's chrooted apache server. For those who want to learn more about Django or OpenBSD, please visit their homepages.

I've been looking through the web for awhile now trying to find a way to hook up django in OpenBSD's version of Apache with little avail. I've seen methods that incorporated installing Apache 2.2 and mod_python or through other webservers (like lighttpd) but nothing for chrooted Apache. Since the version of mod_python needed for Django only runs on Apache 2.0 or later, I needed to find out how to run Django on FastCGI in order for it to run on Apache 1.3. So I toyed around with it with the help of a couple of sites to get me started and I got it running. If anyone is interested, here are the directions I took to get things going....

Before we begin, I used OpenBSD 4.4, Python 2.5, and django 0.96.2 during my experiments. Not to say that it won't work for future or past versions but I wanted to be clear to avoid problems due to file versions listed here...

The first thing to do is to install the necessary tools on the OpenBSD system of choice. This can be done by simply typing the following command:
pkg_add -i python py-flup py-django fcgi mod_fastcgi

Depending on which database adapter you want to use, just tack it on. I'm personally biased towards Postgresql but many people use mysql so type in either:
"pkg_add -i py-psycopg2" for postgresql or "pkg_add -i py-mysql" for mysql.

The next thing to do was setup Python in a chrooted environment. It may sound daunting but it's actually not that bad. You just need to make sure that /var/www has enough space. First, copy all of the python files and associated files to the chrooted directory. Don't forget to create the necessary directories in the chroot environment (ie: /bin/sh -> /var/www/bin/sh). This includes the following files in no particular order:
  • /bin/sh
  • /sbin/ldconfig
  • /usr/local/bin/python2.5
  • /usr/local/bin/pydoc2.5
  • /usr/local/bin/python2.5-config
  • /usr/libexec/ld.so
  • /usr/lib/libc.so.48.0
  • /usr/lib/libcom_err.so.16.0
  • /usr/lib/libcrypto.so.14.0
  • /usr/lib/libm.so.3.0
  • /usr/lib/libncurses.so.10.0
  • /usr/lib/libpthread.so.11.0
  • /usr/lib/libreadline.so.3.0
  • /usr/lib/libssl.so.11.0
  • /usr/lib/libstdc++.so.45.0
  • /usr/lib/libutil.so.11.0
  • /usr/lib/libz.so.4.1
  • /usr/local/lib/libfcgi++.so.1.0
  • /usr/local/lib/libfcgi.so.0.0
  • /usr/local/lib/libpython2.5.so.1.0
Side note: we are copying all of the python executables into chroot as is. Since django (and possibly others) may rely on the actual "python" name as the executable, you can either copy them over without the version number (ie: python2.5 -> python) or setup symbolic links. If you go with the latter route, make sure you link them without the chroot directory. Otherwise, you will probably run into problems.

I also copied the entire /usr/local/lib/python2.5 directory over to make sure that none of the modules were missed. Some of you may want to be more cautious and only move over the files you feel are necessary. Your call..

More files to copy. If you are using postgresql, you need the following:
  • /usr/local/lib/libecpg.so.7.0
  • /usr/local/lib/libecpg_compat.so.4.0
  • /usr/local/lib/libpgtypes.so.4.0
  • /usr/local/lib/libpg.so.5.1

For mysql, you'll need:
  • /usr/local/lib/libmysqlclient.so.19.0
  • /usr/local/lib/libmysqlclient_r.so.19.0
  • /usr/local/lib/mysql/libmysqlclient.so.19.0
  • /usr/local/lib/mysql/libmysqlclient_r.so.19.0
Edit: I missed this part last time so my apologies to anyone listening. Django requires some additional nodes to run correctly. Since we are chrooting in /var/www, Django will not be able to access them. They can be recreated in the necessary area by issuing the following commands...
mknod -m 666 /var/www/dev/null c 2 2
mknod -m 644 /var/www/dev/random c 45 0
mknod -m 644 /var/www/dev/urandom c 45 2
If you want to know more about the above commands, I suggest reviewing the OpenBSD manpages since they are very detailed and clear on the commands.

Once all of the files are copied, generate a chrooted ld.hints.so file by typing the command:
chroot /var/www /sbin/ldconfig /usr/local/lib
Once this is done, you should now have a chrooted python to play with. You can confirm this by typing:
chroot /var/www /usr/local/bin/python
If a python prompt comes up, it means that python successfully chrooted in /var/www. You should be able to play with the modules, including the database adapters to make sure that they are operating correctly. I would suggest that once you get in this area, you should start playing with some of the Django modules. In particular, try something like:
import django
django.VERSION
If python errors out or if one of the database adapters doesn't import correctly, it means that a file (usually an .so library) was missed somewhere. Use the ldd command to help find out which files are missing.

Now build a django project. More detailed directions on how to do this can be found at Django's website. After the project is created, create an empty __init__.py file in the directory one level up. You'll need this in a minute. Now, in the project directory, you'll need to create an fcgi file for apache to use. You can name it anything you wish, as long as you are consistant and the file is located in the project root path. I will use mysite.fcgi just for reference. In this file, it should look something like this:

#! /usr/local/bin/python

# This file is based off of the example posted on Django's website:
# http://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/#howto-deployment-fastcgi

import os,sys

# This should be the chrooted path one level up from your Django project
RootPath = "/django"

sys.path.insert(0, RootPath)

os.environ['DJANGO_SETTINGS_MODULE'] = "(project name).settings"

from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")
Once the file is created, apache needs to be setup. Usually, it can be setup as a virtual host. I like to import virtual host setups from external files for easier management. Mine is setup like so:
# Chrooted path to Django project root path
<Directory /django/OpenDjango>
Options ExecCGI FollowSymLinks
AllowOverride all
Allow from all
Order allow,deny
</Directory>

<VirtualHost *>
AddHandler fastcgi-script fcg fcgi

ServerName django.blahblahblah.doesntexist.org

# Complete (not chrooted) path to Django project root path
DocumentRoot /var/www/django/OpenDjango

Alias /media /usr/local/lib/python2.5/site-packages/django/contrib/admin/media

RewriteEngine On
RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ /mysite.fcgi/$1 [QSA,L]
</VirtualHost>
Restart apache and check the website. You should be greeted by generic Django page saying you need to get started. I hope this helps those who were running into similar issues that I was when trying to get this thing going.

A few notes to keep in mind:
  • Remember to remove the "nodev" option from the /var partition in fstab before proceeding (thanks for the reminder Thodoris)
  • You should be comfortable with python, OpenBSD, and apache before attempting this
  • Before I did this to my production server, I setup a test system via Qemu or VirtualBox to make sure what I was doing.
  • Follow the instructions that are given to you by pkg_add when installing the packages
  • FYI -When going this route, the python processes that are started run under the www user. I'm not sure of the security implications of this but it beats running external FastCGI servers under the root user.
Any questions or comments are welcome. I'm sure that this process can be improved upon...

Thanks...

14 comments:

  1. Hello, i have tried 2 days with this and i must inform the users who come in this page about an issue. First there is ready script for this procedure and it is located here: http://code.google.com/p/chrootpython/source/browse/trunk/chroot_python.py
    Secondly, by default (at least at my 4.8 installation) the /var is mounted with nodev option on fstab so you must remove this option from fstab in order to make /dev/urandom, random, null to work :) thanks Syn_Ack

    ReplyDelete
  2. D'oh, completely forgot about nodev on var in fstab. I'm so use to removing it myself that I spaced it when I wrote this. Sorry about that. Thanks Thodoris.

    Also, chroot_python.py was just updated. Not too much changed except for taking advantage of optparse and tested on OpenBSD 4.9

    ReplyDelete
  3. Doing in your office. It does not work ... In the log is written:
    [Fri Jul 1 12:26:13 2011] [error] [client 127.0.0.1] File does not exist: /djcode/test/mysite.fcgi/
    Catalogue with Jango - /var/www/djcode/test
    What could be wrong?

    ReplyDelete
  4. Hard to tell without seeing the setup.

    What files are listed in the /var/www/djcode/test directory? Is the "mysite.fcgi" file in that directory?

    ReplyDelete
  5. Yes, the file is located in the directory mysite.fcgi /var/www/djcode/test
    in /var/www/djcode/ I created an empty file __init__.py
    In httpd.conf add:


    Options ExecCGI FollowSymLinks
    AllowOverride all
    Allow from all
    Order allow,deny


    and:

    AddHandler fastcgi-script fcg fcgi
    ServerName mysite.mydomen.com


    DocumentRoot /var/www/djcode/test
    Alias /media /usr/local/lib/python2.6/site-packages/django/contrib/admin/media

    RewriteEngine On
    RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^/(.*)$ /mysite.fcgi/$1 [QSA,L]

    ReplyDelete
  6. Syn_Ack, thanks, I switched to lighttpd and Django works:)

    ReplyDelete
  7. I want to install django on my OpenBSD. thanks are guided on how to install.

    ReplyDelete
  8. The files you list for the chroot are very version-specific. A better approach is to copy all files from the package:

    pkg_info -Lq python%3.7 | pax -rw -v /var/www/

    Python also has depencies you will need as soon as you try to do anything with virtualenv and so on. Dependencies can be found with:

    pkg_info -f python%3.7 | grep '^@depend' | cut -f 3 -d :

    You can use the approach above again to copy all necessary files. Also you need to check for dependencies of dependencies (e.g. libiconv is a required dep of gettext) and likely pip (which has its separate packages now, pi-pip and py3-pip).

    Another note: if one is setting up a chroot from scratch, he will need to set the HOME variable at the very least, or things like pip will blow up.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete