Donnie’s Blog

I was pleased to note that South has recently been recommended as a good solution to the whole problem of database schema migration when using Django. I have been using South in csman for over a month now, and it has just been great. I still back up the database before applying a schema-migration to the production system (no sense in being a complete idiot), but so far every time I’ve had to make a schema change, the process has been completely flawless.

So, I can definitely recommend South too!

In general, it’s still very wise to try to change schemas as infrequently as possible, so I tend to batch up my changes, but South has almost completely removed my worries about making schema changes in my Django projects. It’s a great tool.

Using Django for csman has mostly been a positive experience, although there have been a few wrinkles along the way. I just ran into another one today, which is quite frustrating — one of my pages takes about ten seconds to render, and nobody wants to wait ten seconds! Fortunately it’s for the teachers and not the students; if it were for the students, there would be no end of whining!

This page includes a LOT of database queries, almost to the point where I should be embarrassed, except for the fact that I hacked this thing together as quickly as possible, so actually I’m proud that it has as many features as it has. But for 130 students and around 15 assignments, the server-side code issues around 2000 queries, and you don’t need to be a genius to know that this just won’t be very fast.

But surprisingly enough, only about half the time is spent in database land; the rest of the time is spent rendering the Django template. I was quite surprised by this! I thought it would be much faster than that, but evidently not. There are three nested loops in the template, iterating over a large amount of data, so in retrospect it’s no surprise, but an hour ago I was sure that I needed to optimize the database interactions, not the template rendering. Now I see that the database code is only half of the problem.

All I did to find this was to add a couple of print statements around my view code to see how much time was actually spent in the database queries. I’m glad I did; I would have been completely befuddled, otherwise. It just goes to prove that old bit of software engineering wisdom, that if you try to optimize a system before you’ve actually profiled it, you probably won’t fix the real bottleneck.

I think I can fix this issue by creating a custom template-tag that will move some of the rendering into Python code, instead of being run in the template engine. The more I can incorporate into Python, the better off I’ll probably be. This may also help with the database side of things, because if I have clever template tags, then I can do less preparation with the data, and therefore fetch larger amounts of data from the back-end in one shot.

I just wrote up a new programming note about how to do syntax-highlighted diff output using Pygments and Python’s difflib library, in a Django website. It’s available here.

I implemented this feature for csman, a homework submission and grading system I have been implementing for the last couple of months, to use with the CS1, CS4, and CS11 classes that Mike Vanier and I teach. All of these classes have a similar homework-submission model, much like the model in real software projects: you write the code, and if it’s not good enough, you have to fix it.

Of course, grading redos is always a little annoying, so if you have a tool that lets you view a diff of the old submission and the new submission, it becomes very simple to see if the student actually fixed everything you told them to fix. (Sometimes they don’t. It’s very annoying when this happens.)

Anyway, it’s a cool little feature, to be able to see syntax-highlighted diff output on csman, but it definitely still needs a lot of work. The main issue is that frequently the files are too wide to display side-by-side, unless you happen to have a nice large monitor like I do. So, the problem really requires the effective use of scrollbars, both horizontal scrollbars to allow the diff-output to be visible in a smaller window, and also vertical scrollbars since otherwise those horizontal scrollbars are going to be waaayy at the bottom of your web page.

So, I have some work to do on that problem, but for now I’ll stick with what I have. It does the job, and one day when I learn more client-side JavaScript, maybe I’ll fix that kind of stuff up to be more snazzy.