In a recent post to the Open Scriptures mailing list, it was suggested that we use md5 (or another cryptographic hash) to generate unique IDs for each token (a "token" is the fundamental unit of text (most often a word) in our API database models). Today we discussed the implementation of this on IRC, and it was fairly stimulating.

First of all, md5 is broken and deprecated, due to possible collisions (two different pieces of data can result in the same hash). Since we will be dealing with millions of tokens, we decided not to test our luck, unlikely though a problem may be. SHA-256 has no known collisions, so we decided it was best to use that algorithm.

SHA-256 is implemented in Python's standard library hashlib, so that is good. For exapmle:

>>> import hashlib
>>> hashlib.sha256("Hello world!").digest()
'\xc0S^K\xe2\xb7\x9f\xfd\x93)\x13\x05Ck\xf8\x891NJ?\xae\xc0^\xcf\xfc\xbb}\xf3\x1a\xd9\xe5\x1a'

Needless to say, such a digest would not be very good for use in a RESTful URI scheme. So, hashlib also offers a hexadecimal option:

>>> hashlib.sha256("Hello world!").hexdigest()
'c0535e4be2b79ffd93291305436bf889314e4a3faec05ecffcbb7df31ad9e51a'

That is still not the best, since that makes for a very long string. So, we have the option of using base64 encoding:

>>> import base64
>>> base64.b64encode(hashlib.sha256("Hello world!").digest())
'wFNeS+K3n/2TKRMFQ2v4iTFOSj+uwF7P/Lt98xrZ5Ro='

That is shorter, but it includes the "/" character, which is a no-no for URI design. Luckily base64 includes a function for this exact purpose: Read more »

Over the long weekend I got together with Weston, and we did some work on Open Scriptures. This resulted in a storm of commits, and a much better code base. We have been fortunate enough to be joined by Patrick Altman, whose experience with Django led to some immediate improvements in the structure and functionality of the code. This in turn has inspired other work, and the project as a whole is moving a decent pace at the moment.

We are hanging out in #openscriptures on irc.freenode.net. If you are so inclined to drop in and learn more about the project, feel free to join the room.

I suggested that we have an "OpenScripturesCon" in Portland some time. Then Weston suggested that we do it to co-incide with Django-con in September, and host it at Multnomah University. So, the OSMU (Open Scriptures Meet Up at Multnomah U) is (possibly) born. Django-con's dates are not nailed down yet, so there is no firm date for the event. It would be quite beneficial to meet with some expert Django developers (like James Tauber).  Let's keep our fingers crossed.

For the Open Scriptures API I was trying to find an elegant way to store parsing information in a language-agnostic way in a Django model. To achieve that end, I chose to use derived classes. My thought was that we could use a generic class (TokenParsing) as an abstraction layer through which the language-specific parsing models could be connected to our metadata class (TokenMeta). Those parsings could then be retrieved using this bit of magic:

TokenParsing.objects.get(tokenmeta = self)

I thought that this would return a list of all the objects subclassed TokenParsing which linked to the metadata object. However, when subclassed objects are created in Python and stored in Django, it actually creates two instances, one of the base class, and one of the derived class. So this function returns only the base class objects, which have no useful information in them. In other words, the derived, language-specific parsing class (e.g. TokenParsing_grc) can have a link to the TokenMeta object, but there is no way to traverse that link backwards from the TokenMeta object. Without that, the API would not be able to query metadata properly. I poked around at other possible methods for achieving this (e.g. abstract base classes), but none of them would achieve my goal. So I have decided to revert to the original method. But as it turns out, that is not a bad thing. In my quest to find a way to express parsings in a language-agnostic way, I overlooked one glaring problem: a language-agnostic solution would be useless. This came home when I was searching for a solution to this problem: Read more »