Django 1.1 imported to OpenBSD Current

It is kinda old news now but it was nice of the powers that be to update the ports tree to the most current version of Django. Even though it wasn't committed in time for 4.6 (timing was off between the Django release and ports lock), it should be in for 4.7. I'm just happy that the 4.6 release will have something greater than the 0.96 version of Django.

Go ahead and preorder your copy of OpenBSD 4.6 today!!!

Hello Lighttpd, nice to meet you

Since I've been trying to wrap my brain around web server configs for the past couple of weeks, I thought that I would give Lighttpd a try since I've read quite a bit of positive reviews of it. I must say, I wish I went this route the first time. As a test, I thought I would setup django 1.1 on OpenBSD using Lighttpd instead of the native httpd that comes with OpenBSD out of the box. I was able to get it going in about 10-15 minutes after setting up the configure files using the how-to guides on Django's site. It was nice to see that good admin login screen after little tinkering. The only draw back that took me longer than I wanted was that Lighttpd was adding unwanted items to the url that prevented me from logging in completely. However, this was easily rectified by adding the following line to the settings.py file:

FORCE_SCRIPT_NAME=""

This may be the route to take with django on OpenBSD systems from now on because of the easy to use settings. Just remember to install flup...

mod_wsgi on OpenBSD - take 2

Before we get going, I would like to point out that this information is provided to you at your own risk. You should know how to modify configuration files and rebuild programs before you attempt to perform the steps above. I am not responsible for any damage, loss of data, or loss of use if you attempt to use any of this information on your own systems.

Sorry, I have to cover my a$$. :)

Let's get this thing rolling, shall we?

As I stated in my previous post, the only ways to get mod_wsgi working with the version of apache that comes stock with OpenBSD are:

1. Recompile apache with the pthread lib -or-
2. Use the LD_PRELOAD trick to preload the pthread library before running httpd

If you are squeemish on recompiling, go ahead and use the LD_PRELOAD trick. Please keep in mind though that you need the exact path and version number of pthread for it to work. This means that you also need to keep track of the version number of pthread whenever a new version of OpenBSD is released. Example:

LD_PRELOAD=/usr/lib/libpthread.so.11.0 apachectl start

To be honest, recompiling apache is a bit easier and less of a hand cramp. Per James Turner's post, you just need to add one line to the configure file: /usr/src/usr.sbin/httpd/src/Configure. At line 519 of the configure file, just add LIBS="$LIBS -pthread". So the section will go from this:

*-openbsd*)
OS='OpenBSD'
DBM_LIB=""
DB_LIB=""
DEF_WANTHSREGEX=no
;;

to this:

*-openbsd*)
OS='OpenBSD'
DBM_LIB=""
DB_LIB=""
DEF_WANTHSREGEX=no
LIBS="$LIBS -pthread" # Add pthread library
;;

Then rebuild httpd using the following commands. This is where I had problems last week. I wasn't rebuilding this correctly...

apachectl stop
cd /usr/src/usr.sbin/httpd
make -f Makefile.bsd-wrapper obj
make -f Makefile.bsd-wrapper cleandir
make -f Makefile.bsd-wrapper depend
make -f Makefile.bsd-wrapper
make -f Makefile.bsd-wrapper install

Then restart apache. Do this every time you install a new version of OpenBSD and you'll be able to use mod_wsgi.

Afer some more tweeking of the settings, I was able to get a sample django project to run under mod_wsgi. However, it wasn't without its quarks...

When I was testing the admin site under wsgi, I noticed that none of the css or other media files where showing up. I know it wasn't the project because it looked fine when using the fcgi method. I spent the better part of a day picking the site apart to try to understand why the media information wasn't displaying. I was using alias left and right and everything in between but nothing was working. After googling for a few minutes, I landed back on mod_wsgi's site on the configuration guidelines under the hosting static files anchor. The link to the actual spot is here. Well it turns out that the only time the Alias module takes precedence over WSGIScriptAlias on Apache 1.3 is when mod_wsgi is loaded before the mod_alias module. Ok, so I shoot over to the httpd.conf file to set mod_wsgi to load first when I notice that mod_alias isn't even listed in the conf file. Wait a minute, so that mean... ahh crap. mod_alias is statically compiled into OpenBSD's apache. So in order for alias to work with wsgi is to recompile apache again....shit....

Nope, ain't gonna do it just for the media files. That's just too much work for something this small. An easier, and the recommended way per the django folks, is to just setup another site or virtual host just to host these files and point to that site in the settings.py file in the project. Like so:

ADMIN_MEDIA_PREFIX='http://localhost:8180/media/'

You can set the alias with the virtual host like you would normally, just make sure you point it to the correct chroot path where the django files are kept. On OpenBSD, this should be:

/usr/local/lib/python2.5/site-packages/django/contrib/admin/media

This would probably be a better setup anyway because you don't have to worry about aliasing the media files for every django project you do. But going the mod_wsgi route on OpenBSD, it's really the only way.

With this being said, this may be the way to go in the future. However, since mod_wsgi is not in ports (yet) and considering the recompiling needed in order for it to work, you probably won't get much support as you would the fastcgi method of setting up django on Openbsd's chroot apache server. But at least the options are starting to grow...

mod_wsgi on OpenBSD

Ever since I started learning about OpenBSD a few years ago, I always tried to find easier ways of getting Python to play with the version of Apache that comes with OpenBSD out of the box. Mod_Python was a nice try but it doesn't work with Apache 1.3. So I've been settling with FastCGI for the time being. Works ok and seems to have some advantages but it can be difficult to get going, especially in a chrooted environment. Well, while browsing the django docs I noticed that the preferred setup switched from mod_python to mod_wsgi. This was a nice surprise since mod_wsgi works with Apache 1.3 and 2.0+ (if I'm reading their docs right). To top it off, I found a port entry by James Turner that looked good. It was a couple versions old but still good to try. I threw the port into an OpenBSD test environment and it installed without issues. But that's where the fun ended.

After I installed it and tried to start the Apache server, the errors started to flow. Kinda disappointing but in mod_wsgi's defense, it's difficult to account for everything type of apache setup when creating a mod. I dug around a bit and found that the issue is with the chrooting and the pthread library. In an earlier post by James Turner pointed this out and two possible ways of getting around this issue. One is by adding a line in the httpd configure file to include the pthread library during the build and the other is by using the LD_PRELOAD variable. I recently tried to do the first item but during the build process, I received building errors. I'm thinking that the error occurred due to my setup but I haven't had a chance to setup another test system since then. Option two works to a point. It's alot of text to type and if the system reboots, you'll see errors during the boot up process. You can make this easier by editing the rc and apachectl files to include the LD_PRELOAD before the command. Even this makes the upgrade process more complicated because you have to account for the version number of the pthread library. These modifications aren't the most ideal but they are not impossible to do.

All this aside, this hasn't been imported into the OpenBSD port system. Although I am not sure of the official reason behind this, I'm thinking that these problems do not help. Until it is and this little quark is fixed, I would suggest sticking with the fcgi route for Django on OpenBSD. If you are daring enough, you can try yourself, at your own risk of course.

Finally posted the script..

It's funny how time seems to fly. I was reviewing my first (and only) post here and while I was making some modifications to it, I realized that I never got around to posting my script that made the process easier. Well, I setup a google code repository for it so I can get better feedback on it. It is under the ISC License so take it and do what you wish. The google code page is located here.

I am going to see if I can start posting more on a regular basis depending on my schedule. Not that I'm popular or anything but it's a fun little hobby..

Django on OpenBSD

This is an article about setting up the Python web framework Django in OpenBSD's chrooted apache server. For those who want to learn more about Django or OpenBSD, please visit their homepages.

I've been looking through the web for awhile now trying to find a way to hook up django in OpenBSD's version of Apache with little avail. I've seen methods that incorporated installing Apache 2.2 and mod_python or through other webservers (like lighttpd) but nothing for chrooted Apache. Since the version of mod_python needed for Django only runs on Apache 2.0 or later, I needed to find out how to run Django on FastCGI in order for it to run on Apache 1.3. So I toyed around with it with the help of a couple of sites to get me started and I got it running. If anyone is interested, here are the directions I took to get things going....

Before we begin, I used OpenBSD 4.4, Python 2.5, and django 0.96.2 during my experiments. Not to say that it won't work for future or past versions but I wanted to be clear to avoid problems due to file versions listed here...

The first thing to do is to install the necessary tools on the OpenBSD system of choice. This can be done by simply typing the following command:

pkg_add -i python py-flup py-django fcgi mod_fastcgi

Depending on which database adapter you want to use, just tack it on. I'm personally biased towards Postgresql but many people use mysql so type in either:
"pkg_add -i py-psycopg2" for postgresql or "pkg_add -i py-mysql" for mysql.

The next thing to do was setup Python in a chrooted environment. It may sound daunting but it's actually not that bad. You just need to make sure that /var/www has enough space. First, copy all of the python files and associated files to the chrooted directory. Don't forget to create the necessary directories in the chroot environment (ie: /bin/sh -> /var/www/bin/sh). This includes the following files in no particular order:

• /bin/sh
• /sbin/ldconfig
• /usr/local/bin/python2.5
• /usr/local/bin/pydoc2.5
• /usr/local/bin/python2.5-config
  
• /usr/libexec/ld.so
• /usr/lib/libc.so.48.0
• /usr/lib/libcom_err.so.16.0
• /usr/lib/libcrypto.so.14.0
• /usr/lib/libm.so.3.0
• /usr/lib/libncurses.so.10.0
• /usr/lib/libpthread.so.11.0
• /usr/lib/libreadline.so.3.0
• /usr/lib/libssl.so.11.0
• /usr/lib/libstdc++.so.45.0
• /usr/lib/libutil.so.11.0
• /usr/lib/libz.so.4.1
• /usr/local/lib/libfcgi++.so.1.0
• /usr/local/lib/libfcgi.so.0.0
• /usr/local/lib/libpython2.5.so.1.0

Side note: we are copying all of the python executables into chroot as is. Since django (and possibly others) may rely on the actual "python" name as the executable, you can either copy them over without the version number (ie: python2.5 -> python) or setup symbolic links. If you go with the latter route, make sure you link them without the chroot directory. Otherwise, you will probably run into problems.

I also copied the entire /usr/local/lib/python2.5 directory over to make sure that none of the modules were missed. Some of you may want to be more cautious and only move over the files you feel are necessary. Your call..

More files to copy. If you are using postgresql, you need the following:

• /usr/local/lib/libecpg.so.7.0
• /usr/local/lib/libecpg_compat.so.4.0
• /usr/local/lib/libpgtypes.so.4.0
• /usr/local/lib/libpg.so.5.1

For mysql, you'll need:

• /usr/local/lib/libmysqlclient.so.19.0
• /usr/local/lib/libmysqlclient_r.so.19.0
• /usr/local/lib/mysql/libmysqlclient.so.19.0
• /usr/local/lib/mysql/libmysqlclient_r.so.19.0

Edit: I missed this part last time so my apologies to anyone listening. Django requires some additional nodes to run correctly. Since we are chrooting in /var/www, Django will not be able to access them. They can be recreated in the necessary area by issuing the following commands...

mknod -m 666 /var/www/dev/null c 2 2
mknod -m 644 /var/www/dev/random c 45 0
mknod -m 644 /var/www/dev/urandom c 45 2

If you want to know more about the above commands, I suggest reviewing the OpenBSD manpages since they are very detailed and clear on the commands.

Once all of the files are copied, generate a chrooted ld.hints.so file by typing the command:

chroot /var/www /sbin/ldconfig /usr/local/lib

Once this is done, you should now have a chrooted python to play with. You can confirm this by typing:

chroot /var/www /usr/local/bin/python

If a python prompt comes up, it means that python successfully chrooted in /var/www. You should be able to play with the modules, including the database adapters to make sure that they are operating correctly. I would suggest that once you get in this area, you should start playing with some of the Django modules. In particular, try something like:

import django
django.VERSION

If python errors out or if one of the database adapters doesn't import correctly, it means that a file (usually an .so library) was missed somewhere. Use the ldd command to help find out which files are missing.

Now build a django project. More detailed directions on how to do this can be found at Django's website. After the project is created, create an empty __init__.py file in the directory one level up. You'll need this in a minute. Now, in the project directory, you'll need to create an fcgi file for apache to use. You can name it anything you wish, as long as you are consistant and the file is located in the project root path. I will use mysite.fcgi just for reference. In this file, it should look something like this:

#! /usr/local/bin/python

# This file is based off of the example posted on Django's website:
# http://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/#howto-deployment-fastcgi

import os,sys

# This should be the chrooted path one level up from your Django project
RootPath = "/django"

sys.path.insert(0, RootPath)

os.environ['DJANGO_SETTINGS_MODULE'] = "(project name).settings"

from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")

Once the file is created, apache needs to be setup. Usually, it can be setup as a virtual host. I like to import virtual host setups from external files for easier management. Mine is setup like so:

# Chrooted path to Django project root path
<Directory /django/OpenDjango>
Options ExecCGI FollowSymLinks
AllowOverride all
Allow from all
Order allow,deny
</Directory>

<VirtualHost *>
AddHandler fastcgi-script fcg fcgi

ServerName django.blahblahblah.doesntexist.org

# Complete (not chrooted) path to Django project root path
DocumentRoot /var/www/django/OpenDjango

Alias /media /usr/local/lib/python2.5/site-packages/django/contrib/admin/media

RewriteEngine On
RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ /mysite.fcgi/$1 [QSA,L]
</VirtualHost>

Restart apache and check the website. You should be greeted by generic Django page saying you need to get started. I hope this helps those who were running into similar issues that I was when trying to get this thing going.

A few notes to keep in mind:

• Remember to remove the "nodev" option from the /var partition in fstab before proceeding (thanks for the reminder Thodoris)
• You should be comfortable with python, OpenBSD, and apache before attempting this
• Before I did this to my production server, I setup a test system via Qemu or VirtualBox to make sure what I was doing.
• Follow the instructions that are given to you by pkg_add when installing the packages
• FYI -When going this route, the python processes that are started run under the www user. I'm not sure of the security implications of this but it beats running external FastCGI servers under the root user.

Any questions or comments are welcome. I'm sure that this process can be improved upon...

Thanks...