Arc v' June 2002

The hard "can't-be-missed" deadline was this morning, 8:00 AM. Site went live at 6:00 AM. We made it. The behind the scenes magic is still a little scary, but the end user experience works out nice. And we were on time. 24Tix. Buy your tickets for Moby's SLC performance here.

I continue to work and work and work on my current projects. Fortunately, the customer of one is moving slowly on their design decisions (the work on which is being done by coworker), leaving me time to focus on the project with The Immovable Deadline.

9AM Friday. And the design integration work hasn't even begun yet. Ugh. The next two days are going to be fun.

Fortunately, I'm armed with OmniOutliner. I'd be totally lost without it.

Hopefully I'll have a good writeup about the projects in a few days. It's interesting how fundamentally different the two are, not only in the intended end result but in development tactics and practices.

A joy moment - sitting precipitously (or at least, so it felt at times) on an iron balcony extension watching a big summer moon rise over the Wasatch, accompanied by beer (hefeweisen, decent for summer, but I really miss Pete's Summer Brew (beer + lemongrass!)), accompanies by a couple of cigarettes, and accompanied by the July Harper's.

PEP 278 is something I (and I'm sure many others) have been wanting for a very long time. Now, in Python 2.3 (most likely), you'll be able to have code like f = open(filename, 'rU'), and have ALL newlines in the file be the single, normal, '\n' character. Of interest is the new file object attribute, newlines:

A file object that has been opened in universal newline mode gets a new attribute "newlines" which reflects the newline convention used in the file. The value for this attribute is one of None (no newline read yet), "\r", "\n", "\r\n" or a tuple containing all the newline types seen.

This is especially welcome on "Mac OS X", as noted in the PEP, due to OS X's double standards - Classic and Carbon apps typically use the CR symbol to end a line, while Cocoa and Unix apps typically use LF.

I'm a little concerned about my Power Mac G4 Workstation (G4/400, 320 MB Ram). It seems much more sluggish than either my iBook (G3/500, 384 MB Ram) or my iMac (G3/400, 256 MB Ram), which are all running Mac OS 10.1.5. Granted, the Workstation tends to have a higher workload than the others. But even at a fresh startup, applications take longer to launch, and Mozilla 1.0 has a harder time keeping up with my typing on the G4 than on the iMac.

When all the current projects die down, it's time to find a good defragger and find a good RAM tester. I think one of my DIMM's is seriously mismatched. :/

Props to Joseph Kocherans for his Formulator tip yesterday.

A Zope User, a Mac OS X User, an iBook User, a Sonic Youth listener, and apparently in or near Salt Lake. The statistics for that are much higher than I ever would have expected. Then again, as I was hanging out at the opening reception for the Utah Arts Festival last night (on the top floor of the new Wells Fargo building, where I could look directly into my apartment and wonder how many other people look directly in here), I was reminded of just how surprising this town can be. The suburban sprawl and religious zealotry in the valley also produces interesting art culture, urban culture, and counter culture, all in a way to keep things (somewhat) balanced.

Late last night, I finally got Formulator under control. This happened after deciding to take a different approach to it - instead of trying to come up with a generic all-singing automatic form generating template, I took some forms I had started work on in GoLive, and used tal:replace on individual elements directly. Now, I still have my ability to mock up in "GoLive", combined with the ability to use Formulator's validation services.

A quick example:

<div class="warning"
     tal:condition="options/errors/registration|nothing">
 ... loop through validation errors here...
</div>
<form action="register.py" method="post"
      tal:define="form here/form_Register">
 Full Name:
 <input type="text" name="fullname" size="29"
        tal:replace="structure python:form.fullname.render(REQUEST=request)">

 etc...
</form>

Using this technique, I was actually able to use two Formulator Form objects within a single HTML form - one handling registration details, one handling address details (which is expected to be used quite a bit in the site). The target Python script looks something like this:

request = container.REQUEST
errors = []
try: context.form_Register.validate_all_to_request(request)
except FormValidationError, e:
    ## 'e' is an exception instance with a sequence of 
    ## validation errors
    errors.extend(e.errors)
 
try: context.form_Address.validate_all_to_request(request)
except FormValidationError, e:
    ## Now extend the list of errors with Address validation errors
    errors.extend(e.errors)
 
if errors:
    ## put the errors list in a dictionary in to distinguish which
    ## form the errors may have came from.
    return context['register.html'](request, errors={'registration':errors})

There's still one area where I have problems with Formulator, and that's in <select> .. <option> widget generation. I seem to use different option values and contents (<option value="VA">Virginia</option>) quite a bit in my sites, especially since Page Templates have made generating these even easier than in DTML. But I've had a hard time figuring out (a) how to even generate these types of option tags in Formulator (usually based off of the result of some script/method), and (b) how to do the equality "selected" testing when using a form on an existing object with a set value. It may be possible with stock Formulator, but it hasn't been obvious.

On the public side of this site, Form Validation is going to be critical. It's nice not to have to roll-my-own solution yet again.

So, if monday was an exercise in frustration, yesterday was a non-stop four hour marine corps workout. This time, the culprit was MySQL, and it's lack of sub-selects (which they are working on). I have three tables, shows, bands, and band_show_relationship. When a Band is put on a playbill for a show, an entry is put into band_show_relationship with the id's of both the show and the band, and other metadata about the relationship (ie - is the band headlining?).

Getting a list of bands playing in a particular show is really easy and straightforward SQL Joins.

Getting a list of bands NOT playing is an entirely different experience. With a sub-select, it would have been easy:

SELECT * FROM bands
WHERE id NOT IN (
  SELECT band_id FROM band_show_relationship
  WHERE show_id = :show_id
)

So, not having the ability to do sub-selects at this time, I tried doing all sorts of LEFT, RIGHT, LEFT OUTER, RIGHT OUTER variations, and failed. The main reason being that a band that's not on the playbill for a show yet won't be in the band_show_relationship table. So, the data set needs to include all bands that don't have any shows assigned, as well as ones that do but are NOT assigned to this show. I seriously thought an outer join might work, but every permutation either yielded too many results (the set of data I was trying to exclude was included) or none at all.

So, after exhausting that route on my own, I went after the idea of collecting band id's for a particular show and then passing them into the SQL Method the page for assigning/unassigning bands was collecting that information anyways. And I was hoping the great sqltest DTML tag would help me here. Nope! I found a little buglet [Collector Issue 437].

'sqltest' has a cool way of rendering "multiples". When used, the tag renders out a = b if only one value was passed in, and a in (b,c,d) if a non-string sequence is passed in. You can also specify SQL operators to use via the op attribute of the tag, so that <dtml-sqltest band_id column="id" type=nb multiple op=ne> will render a <> b if a single value is passed in. However, the 'ne' operator doesn't pass through to the list. If a sequence is passed in, it still renders a in (b,c,d) instead of a NOT in (b,c,d). Obviously, this can yield very incorrect results as passing in a single value, 'b', will exclude entries with 'b' from the result set; yet passing in a sequence of values, '(b,c,d)', will include 'b' (as well as 'c' and 'd') in the results.

Crappy. So, I got to take a trip in my time machine and go back to the earlier Principia/Aqueduct days before the specialized SQL Methods tags and write my own list generator using dtml-in, dtml-if sequence-start and dtml-if sequence-end to get the SQL that I wanted. And basically, I am now doing a sub-select because in order to get the list to place in the "not in show" method, I run my "bands in show" SQL Method and iterate over the band id's returned.

Later, a coworker who's been using MySQL for far longer than I have came up to try to help me with the do it all in a normal query approach. Even between the two of us, we couldn't get it to work. I'm starting to think that the indexes might be to blame, or my logic is just plain wrong. But, we do have a solution in place.

And it works. Elegant? no. Efficient? probably not. But, there's a big deadline and ultimately this is on a page that's not going to be used by the public-at-large, so we'll let it lie and wait for MySQL to support sub-selects. (Or I'll just sit and wish I had voted for PostgreSQL for this project when I had the chance).

Experiment: AppleScript Studio, Zope, XML-RPC. I want to try something. I think I can figure out a way to use AppleScript Studio to write an app to use XML-RPC calls to Zope to display Zope's object tree in an NSOutlineView. It initially seems like it will be pretty trivial, but I have some reading to do. Updates forthcoming... [via hoarfrost: zope, etc.]

Nice. This is something I've meant to do, but never got all that far into AppleScripting the NSOutlineView. I did get this done, which I was pretty impressed with for my first time doing that much AppleScript - I came up with a good looking (and completely frivolous) application that worked within an incredibly short timespan.

Yesterday was an interesting exercise in frustration. I keep trying to make Formulator work for me, and it comes close, and then always runs away again. I think there are a lot of undiscovered specialties in it that I haven't had the time to chase down.

Besides, working in GoLive with Page Templates makes form work deliriously easy. Even when doing boring designs, like admin screens (for CRUD (Create/Read/Update/Delete) forms), being able to count the number of items you'll need to make an edit form for and then adding in a fourteen row table and setting the left column to be <th align="left"> is super-cool. Then, it's on to dragging and dropping and - most importantly - sizing form elements. And then, a switch over to source mode (or Quick Tag Editing. Or even Outline Mode) to add in some TAL attributes, and it's done.

A beautiful satellite photo of nightfall over Europe...
Looks almost too good to not have been seriously processed, but is an awesome bit of imagery no matter what was done to it. [via bbum's rants, code & references]

We have movable Parts!. That's right. With a little bit of code, a couple of unit tests, and some surprisingly simple UI work, Parts in an SCS Document can be re-arranged.

This is yielding something I've been wanting to do (and have) for years - decent through-the-web Document editing, with drop in objects than can be reorganized easily. While I'm sure it's been done many times before, it was great seeing the results yesterday as part of my own framework and application.

The work went fairly quickly, once I moved the methods to the right class and added Unit Tests. The secret is in the _objects attribute of Zope ObjectManager based instances. _objects is a tuple of dictionaries with the keys 'id' and 'meta_type'. ObjectManager classes use this information to know what objects in the ObjectManager are being managed as subobjects, and to define their ordering. Most people never see or know of this attributes existence as its use is well-handled by the common ObjectManager API's. But one can use this list to their advantage, such as reordering.

So, in todays code we have what I used to reorder Parts in my PartContainer class. There are two important elements - finding the index of the object in question, and then swapping its place in the list. We can't use parts.index(part_id) because of the complex data structure of the list - although, if we had the object in question, we could try parts.index({'id': obj.getId(), 'meta_type': obj.meta_type}). But the following works, and could be applied to any ObjectManager based class. Note - there are already Products available to bring order to Folders in Zope, such as Ordered Folder. It's a feature that was Proposed for Zope 2.6, but was deferred..

class PartContainer(ObjectManager):
    def _findPartIndex(self, part_id, objectList=[]):
        """ Returns the index of a part in the object list """
        idx = 0
        for obj in objectList:
            if obj['id'] == part_id:
                break
            idx += 1
        else:
            ## part not found
            raise IndexError(part_id)
        return idx
 
    security.declareProtected(change_scs_documents, 'movePartUp')
    def movePartUp(self, part_id):
        """ Move a part up.  Raises IndexError if part is not found """
        ## make a copy of our _objects list, which is a part of ObjectManager.
        ## _objects is a tuple of dictionaries with keys 'meta_type' and 'id'.
        parts = list(self._objects)
        idx = self._findPartIndex(part_id, parts)
        if idx == 0:
            ## Can't move past top, just exit
            return 0
        ## swap fields, moving idx up
        parts[idx], parts[idx-1] = parts[idx-1], parts[idx]
        self._objects = tuple(parts)
        return 1
 
    security.declareProtected(change_scs_documents, 'movePartDown')
    def movePartDown(self, part_id):
        """ Move a part down.  Raises IndexError if part is not found """
        ## make a copy of our _objects list, which is a part of ObjectManager.
        ## _objects is a tuple of dictionaries with keys 'meta_type' and 'id'.
        parts = list(self._objects)
        idx = self._findPartIndex(part_id, parts)
        if idx == len(parts)-1:
            ## Can't move past bottom, so just exit
            return 0
        ## swap fields, moving idx up
        parts[idx], parts[idx+1] = parts[idx+1], parts[idx]
        self._objects = tuple(parts)
        return 1

I'm currently swamped under a double-workload, which yields little time to write about the interesting side effects, projects, and findings that go along with all of said work. The SCS Framework, an in-house component/compound document framework, is yielding a mixed bag of results that - ultimately - is for the better. It's made me understand further the need for ZCML (The Zope Configuration Markup Language, a large part of Zope 3), although I still find the syntax regrettable.

I've also noticed that since skimming through books like Extreme Programming Explained and Refactoring, my programming style has changed. The designs are more complex (factored - usually - into smaller classes and objects and interrelations), and the physical code is usually clearer (real variable and method names, etc). It's been this way for a while, but I only took notice of this recently. What's contributed to this, I think, was the finding of M-/ in XEmacs - Autocomplete!

This weekend, if there's time, I plan on updating the SCS design outline, which hasn't been touched in a couple of months.

According to this program by Jarno Virtanen, this site is a tad more zopishy than pythonishy.

So a few months ago, I started working on a little compound document system for Zope. Sortof. It was being built as an internal framework of sorts, and I noticed some interesting CMF parallels, such as separation of core concepts from default implementations. The Core of this system, SCS has a lot of usable default implementations, especially of Parts -- a document is basically made up of parts, or more precisely, a root part that contains other parts.

One of the reasons I started developing this was because of the paying project I'm currently working on. In pre-bid discussions with the customer, I realized that a more flexible compound object framework was going to be needed, and that it could potentially benefit our company in other ways as well. When we hit a lull in customers paying for my work (before I went independent), I started work on the compound system, partially as a way to cut time and costs later if we got this gig.

So, we have it. And today was the first day I finally got to try out some of the concepts. What I started on late this afternoon was a Journal. Knowing that I had the SCS, all I needed was a slight specialization of the default SCS Document that would also be CMF Content. This document class, Journal, would consist of its default SCS Root part, which would consist only of Journal Entries. A Journal Entry is a subclass of the default SCS TextPart, basically with an extra created timestamp. The Journal class has a method, listEntries(), which in turn just asks its root part for all journal/* type parts, sorts them by creation date, and reverses the sort.

This is done in relatively few lines of code. Partially because the SCS Container's default listParts() implementation is rather powerful - it's a very hopped up replacement for the more traditional Zope container objectValues() method. One can pass in an IFilter object, or a function, or some keyword values, with which to query the sub-parts of a container. This could, for example, allow usage of other Part types within a Journal to hold other information without getting in the way of entries themselves.

This is different than a folder full of entries. The paradigm is different. The paradigm of folder is just that of "something which can contain anything", but the folder itself is no real special object (it can be in Zope, but few people take advantage of that fact). SCS, on the other hand, places extreme significance on the Document, the outermost object. A document contains one container (currently), and defers all sub-content related specific stuff on to it. This allows a document class to deal with other issues, such as meta data, or interfacing with a particular indexing scheme, while letting the Parts system basically take care of itself. The document is also allowed to dictate some degree of policy, like how a part or handler registry gets looked up, without (theoretically) placing TOO much burden on surrounding objects and systems.

So far, it's working. Tomorrow, I should be able to get things to a point where I can see how well this whole Journal-as-compound-document thing works out. Hopefully, it executes as cleanly as the code currently looks.

There's a new Zope on Mac OS X portal, with links to binaries of Zope and some products (MySQL DA, TransactionAgents).

Willie Nelson: We got water, coffee, and hot chocolate. What's going to happen to our image?

Johnny Cash: As long we keep wearing black, we might be alright. I don't know.

Andrew Kuchling has published an early development revision of "What's New in Python 2.3", detailing some of the work that's already been checked in, and what's no longer bound to the __future__.

Steve Alexander points out that the ZCML file I refer to in my "Intent, and human editable XML." is rather dated, and there are newer examples in the "Zope 3" CVS tree. So, the situation may be improving, but with so many little ZCML files, it's been hard to track down exactly what's better. It still seems too verbose in the wrong places.

There are approximately 91 ZCML files in a fresh CVS update of "Zope 3". I still can can barely figure out what one of them is doing. And I still abhor the syntax. Fortunately, documentation of ZCML has started. Unfortunately, with a large project and tight deadline, I still don't have time to investigate "Zope 3" further. Why? ZCML terrifies me. It shouldn't, but it does. Part of it is that it's a moving target. Part of it is the giant question mark floating over my head and steam coming out of my ears every time I encounter a ZCML file.

Hopefully, I can use or tweak Slip to edit/generate it.

Ken Manheimer has also posted a proposal (a little bit dated) about what he calls Organization Objects, and a really good companion paper about the motivations for organization objects. The motivations paper discusses how the parent/child and table of contents relationship(s) were done for ZWiki, and how Organization Objects (which bear similarity to XLink) would have benefited the application.

A bullet point in the Organization Objects proposal that I really liked is:

• Living with only links and searches is like living in an imaginary world where every room is connected to every other by transporters - it lacks a basis for regionality, neighborhood.
  
  In other words, it lacks progressively grouped regions by which you can realize "where you are".

ZWiki is a sehr-cool beast, as far as Wikis go. A very attractive feature, seen all over the Zope.org web site is Ken Manheimer's work on page/parenting relationships. This makes it so that Wiki pages, although stored in a flat namespace (all in one folder) can appear to be hierarchical, depending on which page generated which page (parenting can be altered of course). This in turn leads to cool things, like a structured table of contents (see the Zope 3 wiki's TOC Here).

It's especially cool in that it's fairly easy to change the hierarchical structure of the Wiki - by looking at a page's backlinks, one can reparent a page to one or more of its linkers. Essentially, new logical structures can grow independent of the physical structure.

I've used a similar feature in a recent simple content system, wherein all documents were stored in a single folder, but the users entering content could classify documents in one or more categories. The main pages on the site were basically database queries (actually, catalog queries) based on the classifications. Thus, the site was built out of a simple logical structure rather than a physical one, which allowed documents to appear in more than one place if they needed to be. It's no unique concept, "Radio" and some other blogging tools allow this, much to my delight.

In the simple compound system I did for that customer, the flat namespace also allowed simple inter-document link management. The author could say "document A links to document B", and when document B was deleted, the link to it would also evaporate. For some reason, doing deep inter-object linking in "Zope" is still tricky business, but there's hope on the "Zope 3" horizon (some of which looks like it may be backported to Zope 2).

There's been an interesting debate on the "Zope" 3 development list about ZCML - the new Zope 3 configuration language.

The debate ultimately seems to be over what I view as "human readable" XML versus "machine friendly" XML. There is a lot of power and usefulness in XML, but the proliferation of namespaces and unknown attribute handling has taken the human out of it, I think. Intent gets lost.

Anyways, they're sticking with the current ZCML format, which can be seen in practice here. It's not horrible, but it could be better.

Personally, I found the Servlet Web-App Descriptor to be much easier to work with, as per the Jakarta/Tomcat sample starter file. It's more verbose, which requires more typing (it seems that programmers really hate to type), but the intent is much clearer. Intent. It's valuable. But sadly, so easy to cast off.

• eMac for everybody. While I love my current iMac (DV SE/400), an eMac could address the few issues I do have with it.
• QuickTime 6 Public Preview. Finally, the mythical QT 6 starts showing itself to the world
• As The Apple Turns is finally coming back on the air.

Google Catalog Search - try it, then sit and go "how the hell did they do that?"

Search results show scanned images of the catalog, with the search terms hilighted. And it's fast. Amazingly fast.

Nicely, this shows up in time for me to search for something obscure - like a 5Volt 3Amp power adapter.