Arc v' April 2003

Ever had that feeling that if you were just going to a restaurant to have dinner to kill time before seeing a show that you would be suddenly grabbed, have guns pointed at you, and held without lawyer on an unwarranted search? And not just you - you and everyone in that restaurant? And, ultimately, to have spent an evening going through all this for something that was a mistake?

Well now in America we can have that very feeling!

Three days later I phoned the restaurant to discover what happened. The owner was nervous and embarrassed and obviously did not want to talk about it. But I managed to ascertain that the whole thing had been one giant mistake. A mistake. Loaded guns pointed in faces, people made to crawl on their hands and knees, police officers clearly exacerbating a tense situation by kicking in doors, taunting, keeping their fingers on the trigger even after the situation was under control. A mistake. And, according to the ACLU a perfectly legal one, thanks to the Patriot Act.

And the sequel isn't even out yet. But here are some of the exciting new twists it is expected to bring us:

The Patriot Act is just the first phase of the erosion of the Fourth Amendment. From the Justice Department has emerged a draft of the Domestic Securities Enhancement Act, also known as Patriot II. Among other things, this act would allow the Justice Department to detain anyone, anytime, secretly and indefinitely. It would also make it a crime to reveal the identity or even existence of such a detainee.

Think it won't happen to you? Neither did Jason Halperin, until he and his roommate stopped in to grab dinner before a show.

America, "sacrificing our own liberties to share them with the world."

Tom Tomorrow responds to this article with the following:

About a year or so ago, I remember reading some right wing asshat on a message board insisting that the innocent have nothing to worry about, there will be no abuses of power, yadda yadda yadda. I guess that particular moron now lives in the country he deserves; it's just a shame the rest of us have to live there too.

The initial selection of the the Apple Music Store is not yet close to what I'm really looking for, but I've already found some gems. Besides - what I don't mind doing is tracking down the obscure(ish) disks in my collection. I just don't see myself buying a Kid Rock CD in my life (no matter how much I love (really!) American Bad Ass and Cowboy). Apple's store delivers on that without my having to deal with slow, arcane, annoying clients, shoddy network connections, etc.

Many articles about Apple's new music service state the fact that it won't stop people from stealing music any time soon. So what? I'm seeing a service that casual listeners and computer users will use. Many of the articles seem to keep that geek spin about technology. Apple's service, especially as it spreads to Windows (and, if rumors/talks hold true, AOL), will become a "Napster for the rest of us," combining the good aspects of Napster with a MUCH better interface and many of the other high points Jobs brought up (ensured quality, relatively free rights - certainly nothing that get in the way of most non-geek use, etc). It will be interesting to see if or when smaller labels come on board. I noticed that my favorite Dolly Parton and Rodney Crowell releases, which were all recent, were conspicuously absent from the list. But then I remember they're now on Sugar Hill Records (which has a collection of early Willie Nelson demo sessions out now too). In any case, I like Apple's new service. I see good potential. And again - it's something I could see my Dad using. He's a music lover who painfully constructs playlists for his multi-cd player. I bet he would enjoy iTunes and the ability to index and burn his own CD's from the vast amount of material he already owns.

As for me, while looking for other versions of Jobim's One Note Samba (Stereolab and Herbie Mann did a great recording of it as One Note Samba/Surfboard), I found Sergio Mendes and Brazil '66. They had One Note Samba / Spanish Flea. Wonderful as that is, I was very excited to find Bim Bom, Wave, and Look of Love. While AudioGalaxy was still alive I found these tracks attributed to Japanese supergroup Pizzicato Five, with Look of Love being titled "No Not That One!". I've held on to them, as they're spectacular songs. I even have the piano man at my primary bar sing Wave for me. So - to find the original sources and get GOOD copies of all three tracks (and One Note Samba / Spanish Flea and Cinnamon and Clove) was worth the five bucks it cost.

The DRM that's in the purchased tracks is interesting. I'm really hoping that the computer doesn't have to connect to the internet every time one wants to listen to a downloaded song to check to see if it's on an authorized machine. I wonder how it all works.

Bill Bumgarner has already decoded a lot of how the application works. Others have figured out the XML interface used. All in all, this makes iTunes 4 an interesting application. Application specific embedding of HTML/XML has long been Microsoft's wet dream and shows up almost too much in their interface. Apple does a smooth job of it and uses native widgets (column / table) when appropriate and mixes in internet media when appropriate to give a pretty smooth interface. All very interesting.

G. Beato covers Iraqbodycount.net for Alternet and wonders why none of The Networks have taken on reporting or investigating the single greatest ramification of war - the fact that people die. As the search for weapons of mass destruction continues with no substantial results, one wonders why we were there in the first place. A frequent argument was that "once we have free reign of that country, then we'll find them! Unlike those pesky U.N. Inspectors..." And while such weapons may yet turn up, one wonders why those inspectors couldn't have been given more time, potentially sparing lives and billions of dollars. (Isn't it interesting that now the U.S. has "liberated" Iraq, it's calling on everyone else to help foot the reconstruction bill? "We kicked 'em out, you clean it up. Whether you were with us or against us, pay up!") And the U.S. certainly can't pay it. I can't help but imagine what this money could have done to the schools, where hard working long-time teachers like my mother grow ever more disgruntled with bureaucracy, shrinking budgets, and expectations of teachers to do more and more work on their own dime... sigh. In any case, a handful of volunteers doing what The Networks will not:

A project like Iraqbodycount.net was tailor-made for today's 24-hour, interactive, constantly updated news cycles - and it represented a rare opportunity for news organizations to go beyond constantly recycled newzak and Pentagon ventriloquism. The fact that no professional media outlet attempted to do what a couple dozen volunteers pulled off was not a triumph of journalistic responsibility, but rather an embarassing example of journalistic complacency. ["Down for the Count", G. Beato, Alternet 23 Apr 2003; viewed 25 Apr 2003]

Separately, we have Greg Dyke, director general of the BBC:

If Iraq proved anything, it was that the BBC cannot afford to mix patriotism and journalism. This is happening in the United States and if it continues will undermine the credibility of the US electronic news media."

What credibility can our news media have when it spends more time investigating repeating the Chandra Levy story or covering the "DC Sniper" than the human (meaning not just weeping American families who have their own terrible losses to deal with) consequences of war? A genocide in Rwanda often takes fifth place to the abused news item of the week.

Playoff basketball has made it a late night. Great games - New Jersey overtook Milwaukee (dammit), and Minnesota overtook the Lakers (nice). Lots of high last minute drama.

Unfortunately, the LA/Minnesota game distracted me from much of The Daily Show, which featured a pretty interesting conversation with Fareed Zakaria, author of "The Future of Freedom: Illiberal Democracy at Home and Abroad." I wish I could have gotten a bit more of the conversation between Stewart and Zakaria, as they talked about the issues facing Iraq, the transformations going on (slowly) in Iran, why Karzai has been (relatively) successful in Afghanistan. Some interesting things to think about as I finally crawl into bed.

I think that among the better things about the new Apple, and evidence that Apple fandom is like a cult religion, is that they can make certain days become mini-holidays. Next to a Jobs-keynoted Macworld, it's events like the one coming up on monday that leave many among the faithful with christmas-eve style anticipation.

This Reuters article will surely help add fuel to the fire as the fans try to decode Steve-speak. But more interesting than that is Think Secret's notes from the same shareholders meeting:

A concern about hardware speeds was expressed. Jobs first made point that clock speed does not necessarily tell the whole story, and that even Intel is emphasizing diminishing importance of clock speed as it release new portable processors that are faster at slower clock speeds. He then acknowledged that it is a problem, and that Apple is very aware of the issue. He said "there will be a time" when Apple would speak about its relationship with Motorola, and that the particular shareholder who asked the question would be invited, if he desired.

(emphasis added by me). Now that should start the speculation machines to go into overdrive.

After an all-too-late bedtime (it's not my fault I live in a bastard time zone where The Daily Show and now Family Guy come on between midnight and 1 am!), I was awakened around 5:30 by occasional bursts of loud ringing coming from the hallway outside my apartment. By the time I figured out who I was, where I was, and when it was, I realized it was the elevator (apparently one of the oldest in Salt Lake). Someone was trapped. And apparently, no one else was doing anything about it (they were either as befuddled as I was, or figured it was Somebody Else's Problem. So, I called the emergency-maintenance number and let them know there was a stuck elevator with passenger, and told said passenger that someone was on their way. Well, by this time (around 5:45 am), sleep was pointless. I got back in bed for a short while, but decided to make an early day of it. See - I actually really like mornings. But I often miss them due to my love of nights. Afternoon - now I can do without that.

I got into the office at around 8:00 (yesterday I didn't make it into the building until 11:20) and ripped a copy of Salt Marie Celeste. It's been on a loop for much of the day now. Beautiful, and haunting.

What really got to me, however, was a sudden find of pictures of my great-grandmother (whom I knew - I said one of the prayers at her funeral in 1991) and my great-great-grandparents. Sitting around in even more family history (we found a lot on Easter, including the Western Union Telegram my great-grandmother received notifying her that one of her sons was confirmed dead after his B-17 was shot down in Europe in WW II)... I don't know how to describe it.

I finally got my copy of Nurse With Wound's latest release, "Salt Marie Celeste". What a beautiful album. It's one track, 62 minutes long. From the press release:

Similar in concept to Gavin Bryars' The Sinking of the Titanic. Salt Marie Celeste delves even deeper into the theme of dropping into unknown darkness and ultimate demise.

This release apparently expands upon last year's Music for Horse Hospital, a CD of sounds that accompanied a joint art exhibit by David Tibet (of Current 93) and Steven Stapleton (of Nurse With Wound). Salt Marie Celeste builds up a couple of chords, over and over. They're simultaneously warm and cold, some sort of haunting bellows type sound, rolling in waves under the entire piece. Already meditative and somber, this listener found himself in rocks and crags and encroaching fog at the tip of some island. As the piece continues, new sounds emerge: a rattling kind of hissing, some nearby horn. By thirty minutes in, the creaks and groans of an old ship are keeping time with the rolling drones and horn.

Of course, while listening to this in the background (and reading a great entry by Lewis H Lapham in the May 2003 Harper's), the apartment (an old 1910 unit) decides to add in its own noises. First, one of the smoke alarms has been making occasional (every 18 hours or so) chirps. I think it needs some TLC. Anyways, it chirps briefly but loudly. I go back to reading, keeping one eye towards the real clouds outside and another listening to the creeping changes in this track. It sounds like the old ship is falling apart - extra creaks and door sounds can be heard until, nearing the fifty minute mark, a bicycle bell. The creaks stop, but the drones continue. And there's something more sinister: the sound of water dropping in. Those dark dripping subterranean sounds. Around the 55 minute mark, I thought I heard a sudden (brief) new sound - like a gull or baby crying. Since there were a surprising amount of gulls flying around my window when I got home, and since the outside sounds (I forgot to mention a sudden thud which rattled much of the building, a few minutes after the fire alarm chirp) had found great timing already, I actually did stop and back up the track. Nothing. I let it go, the water was stopping, the horn was gone, and soon the drones of the bellows faded out.

It turns out there's a history to the name Mary Celeste. Originally launched under the name Amazon, she was a half-brig that apparently found itself going through many owners and accidents at sea. Purchased years later at a New York salvage auction, she was renamed "Mary Celeste". In November 1872, she departed from New York bound for Italy, but the captain, his family, and small crew were never seen again. The ship was found floating derelict, apparently missing one hastily boarded lifeboat, with no evidence of piracy or foul play. The story inspired Sir Arthur Conan Doyle to publish a fictional short story (under a pseudonym) about a derelict ship called the "Marie Celeste". Some more can be found here and here. An excerpt of the track is available from Brainwashed. I think it accurately captures the feeling of despair on a derelict ship.

Salt Marie Celeste easily finds itself fitting in with some other rainy sit-still-and-don't-move-high-above-the-pavement day music, although it shines best in evening on such a day.

Bill Bumgarner writes:

While I have an active disinterest in Zope compatibility for various reasons, any well designed persistency solution for Python should work just as well within Zope as without.

This is a design decision of Ape, and one of the reasons its name has changed (it used to be AdaptableStorage, and also the code layout has changed. I think one could start working with Ape 0.6 and start building something that didn't depend on Zope and (probably) didn't even depend on the ZODB, although the ZODB is a great persistence framework for Python on its own.

Again, the support for Zope is important to me just because I want to make sure that whatever I'm using is aware of Zope's transaction management. Also, there are issues with threads and database connections that not all designs might take into consideration. This affects Twisted as well, as it is an asynchronous framework, which puts special requirements on database access. All of these features can be met and dealt with, and I agree with Bill that a good persistence system for Python should work anywhere. There are definitely enough software patterns out there that one could write software against that transparently deal with the expectations of different environments that should be pluggable and replacable easily for new situations.

This morning, I did a more thorough reading of the design outline that comes with Ape, and I think that it could satisfy every element on my list - except one. There's still the issue of querying the data. One of the reasons we went with a relational database is the expressiveness of SQL, and the ability to tweak queries for even more performance gain. But, I'm thinking this might not be that big of an issue for one of the applications I'm dealing with, because the really complex reads are happening only on the public side of the site, and there are very very very few times when data is written to the database from the public side of the site. So I might be able to start migrating a lot of the back end to persistent objects backed by Ape serializers while keeping the optimizations and fancy queries intact for the front end of the site. The few places that actually write the data for the front end could be migrated fairly easily once the rest of the structure is in place.

Now with two well designed frameworks that abstract persistence AND work with Zope (Ape 0.6, Modeling 0.9pre4), I can start evaluating their usefulness for migrating existing applications. Two projects that I am working on could benefit from moving to such a system. In this post, I mean to just propose some of the problems/issues/questions I am looking at when evaluating such a system. It's important to keep in mind that this is to apply to existing applications AND data. For a new application with little or no existing data, the evaluation would be a bit easier. With that in mind, let's begin.

1. Does it work with Zope?
This is an important one for me. It's not just that my existing applications are written in Zope, but it's that Zope as a platform offers me many things that I take for granted, not the least of which is the built in transaction management that I've been using since early 1997 when I first used Bobo. It's important that an O-R solution be able to tie in to Zope's transaction management, threads, connections, etc.

2. Does it keep the SQL out of the App?
EOF (Enterprise Objects Framework, part of WebObjects), Modeling (which bases its design on EOF), and Ape all abstract the storage away from the business objects. Other Python frameworks like SQLObject and PyDO bring some SQL to the object by injecting words like 'table' and 'column' (and sometimes things like 'varchar') into a class definition. This can often be quick and easy to work with, but does not fit the design that I (personally) prefer. In the middle (pun not initially intended) is WebWare's MiddleKit. This evaluation point is important to me because, in programming the application, I don't want to think about how the data is stored. I care about that in a different way. I want the business objects to be business objects. It's a testament to APE that it can be put into Zope, electing to store either as normal-looking files on the file system OR as rows in MySQL or Postgres, without modifying ANY existing Zope code, even though how a Python Script, an Image, a Page Template, or anything else it has mappers for, are completely different. This is actually a testament to the ZODB system which, like MySQL, was engineered to have different storage solutions. Ape just gets in there at an incredible level. It also leads to another (smaller) wish - no generated Python code.

Note that wanting this requirement has a significant impact on existing code.

3. Querying/Fetching
This is where Ape is weaker, but considering that the ZODB has no built in query language it's a bit understandable. This is also where design of an adaptable persistence layer shows itself. Apple's Enterprise Objects Framework, since it's meant to store against any back end (mapping to relational tables is just the most common - but storage systems exist for LDAP directories and flat files as well), and that back end should be able to be switched with no impact on an applications design, excels at this. Modeling looks like it's trying to follow suit. SQL is very expressive in its query model, but it allows for read queries that don't necessarily map to objects that are easily writable (joins, joins, joins, and more joins contribute to this problem). This is where it's especially difficult to migrate a direct-SQL application to an O-R mapped one - clients (page templates, scripts, other code) and other system components are expecting a certain data layout.

It's also preferable that a query that may span multiple tables not return a single object if that data is expected to be modified in any way in the present transaction. Thus, when fetching Receipts, which reference a Purchaser table and wanting to get Receipts where Purchaser's last name is 'Shell', you should be able to perform such an operation, but get back just the root Receipt object(s). (Naturally, only the ones that match). I believe EOF/Modeling allow for this to yield the results I'm expecting (but can't explain right now). Because - naturally - it should do this in the least amount of SQL calls possible. And this all leads to the next point:

4. References
I don't just mean relationships between tables, but keeping the right relationships between objects when they're loaded into memory. You want to ensure that if Receipt A and Receipt B both point to Purchaser Bob in the database, that the following results happen (in the same transaction, without extraneous database reads):

    in: A.purchaser.name
    out: 'Bob'
   
    in: B.purchaser.name
    out: 'Bob'

    in: A.purchaser.name = 'Robert'
    
    in: B.purchaser.name
    out: 'Robert'

EOF and Modeling deal with this using EditingContexts (see "Ensuring unicity of an object"). SQLObject also offers this.

However, Zope/ZODB + Ape is a little funnier. The ZODB really seems to prefer having only one persistent reference to an object. Relationships between objects, thus, has always felt like a strong point in relational databases, while a rather weak point in object databases. This has been a problem that's plagued some Zope 2 developers (myself included). Even in Zope 3, complex relationships are handled by application level relationship services. I wonder if such relationships can be (or are) handled via weakref's. I think that Ape, even in its Zope 2 implementation, might allow designing gateways that can "do the right thing". This wouldn't be a problem (as much) under a different application design, but in my apps, sometimes a Receipt is the root object that I might be working from, sometimes it might be a Person (referring to receipts, which might refer back to that person). There are few root objects, but a lot of relations - even between the potential root objects. So being able to fetch the right ones, not pull any duplicates (different database reads of the same row of the same table), and with relative efficiency are important.

Now, in the favor of one of the applications that I have to deal with, a lot of the really complex reads are already abstracted out into special Executable Components (they were previously Python Scripts, but they've been abstracted into classes with configurable run-time behavior). These executables basically are mapped one-to-one with a particular page template to preload a lot of the data needed to keep the amount of logic (and number of queries) on the target template down to a bare minimum. These executables may be relatively easy to convert to special Gateway objects for Ape, or they may be partly moved to a Gateway object and partly kept as they are - only modified to talk to the persistence system directly instead of just firing off SQL directly.

Wrapup...for now
The Jazz are losing, the beer's settling in, it's time to wrap this up...for now. I'm really looking forward to being able to use Ape. Actually, we're already looking at using it to be able to copy code out to CVS thanks to Ape's file system storage (which nicely represents objects as naturally as possible, while still finding ways of staring properties and extra persistent data), but the real clincher for me would be to get a lot of the redundant-feeling create/read/update/delete SQL (and scripts that may show up marginally in front of it) out of the application and into a mapping layer, so that the application can focus on doing its job better with less worries about the data, that'd be great. Since Ape is pretty open about how you can write Schema objects, hopefully that means we can bring Formulator into the equation and be able to automatically generate management and administrative screens and do all that cozy little validation that Formulator provides, similar to what Zope 3's Schema/Form system gives. But it is going to be a fair amount of work to move existing straight-SQL code over.

Shane Hathaway's APE 0.6 is out. APE (short for Adaptable Persistence) just might offer me the high level object-relational AND object-file persistence system for Zope that I've been looking for. This version's pretty substantial and I've just printed out the outline that comes with the software to learn how to write custom gateways/serializers for our MySQL backed web sites.

Update: 3:04 pm: This is a pretty elaborate system. The design is really clean, but still rather specific to serializing regular Zope objects. I'm not sure if I'll have time to learn how to serialize/deserialize items in an existing MySQL database for this round of the software.

And here it is: RIVE 024, gdbodycount by Eucci & Co.. Since we erased our angry live/jam session (on accident), this was compiled instead. One of the things that was going through my mind as we redid the ELW entry, "one minute no silence, one minute suspended detonation", was a phrase from Nurse With Wound, in the liner notes to the terrific "A Missing Sense" release:

Swansong is a vocal piece which was spewed out after watching a documentary on the American A-Bomb tests off the Bikini Islands which left me sad, helpless and fucking angry - an ocean of death.

"fucking angry - an ocean of death." That phrase has been in my mind most of this week, as I found myself increasingly frustrated and even furious at the path this administration is taking. (Also, etched into the vinyl of the original release of Swansong was the phrase "You bastards won't be happy until everythings dead!"). As I've mentioned this week - when your leaders or those who whisper constantly in their ears churn out phrases like total war and bring to memory images of Goebbels, Goering, and similar: you've got a situation. This is not a defense plan - we have all but declared war on the world itself.

Direct links:

1. gd1904; my tv wife.
2. gd1905; ELW one minute no silence, one minute suspended detonation.
3. gd1906; Eucci

The Eucci piece is probably the least offensive to the ear - a rather calm and warm work actually. Visit the page for an explanation of the titling and for an up to date civilian body count.

Cycling '74 announced final versions of Max/MSP/Jitter for Mac OS X. Finally - the really beefy audio tools are making their way to OS X. Part of me is still waiting for Supercollider 3 to get a little more wrapped up though.

Funnily enough, earlier this evening I was back in OS 9 land working on some wrap-ups to rive-024 (to be released online very shortly), although most of the time was spent in Pro Tools. I haven't spent nearly as much time in the generative environments as I did a couple of years ago when I started Eucci as a branch of the ELW. Last night, in fact, it felt good to have the pedals out again for a jam performance. Mmmmm... 46 minutes of high pitched slow moving live music using nothing for a source but the number 2 key of a telephone... (really!).

According to the Detroit Free Press, the hottest rumor is that Saddam made a deal with the U.S. months ago to get out of Iraq safely. This smells of conspiracy theory, but it's a fun one (and just about as plausible as anything else right now).

It'll be interesting to see how strong this rumor gets. The strength wouldn't necessarily indicate that there's any truth, but it would indicate even more distrust of the American government.

Thanks to Evan Simpson (a former Zope Corporation colleague) for pointing me to these postings from Teresa Nielsen Hayden:

Loss, a thoughtful and saddening reflection on the looting of history in Baghdad. A poignant quote from the New York Times article cited:

Mr. Muhammad, the archaeologist, directed much of his anger at President Bush. “A country’s identity, its value and civilization resides in its history,” he said. “If a country’s civilization is looted, as ours has been here, its history ends. Please tell this to President Bush. Please remind him that he promised to liberate the Iraqi people, but that this is not a liberation, this is a humiliation. If we had stayed under the rule of Saddam Hussein, it would have been much better.”

And this is evidence of...?, citing an article in The Independent about the libraries of Baghdad being burned. From the Independent article cited:

For almost a thousand years, Baghdad was the cultural capital of the Arab world, the most literate population in the Middle East. Genghis Khan’s grandson burnt the city in the 13th century and, so it was said, the Tigris river ran black with the ink of books. Yesterday, the black ashes of thousands of ancient documents filled the skies of Iraq. Why?

To which Teresa responds with chilling speculation:

And why shouldn’t he [Donald Rumsfeld, suggesting the weapons all moved to Syria]? With so many records destroyed, in the library and elsewhere, we can claim anything about Saddam’s regime; and who can prove us wrong? If we really wanted to know about weapons of mass destruction, we wouldn’t have blown Chemical Ali’s house to smithereenies. That’s where those records were kept. Or if we did happen to blow it to smithereenies, we would have gone in and secured the site and whatever remained. It wouldn’t have been difficult; a reporter for the Daily Telegraph walked into the house while it was still being looted by local peasants and little kids. But we didn’t do that either. Nor did we secure the government offices and the homes of other high officials to see whether their paperwork held any clues. Darned if the looters didn’t destroy all that data too.

It's so much easier to rewrite history and reshape the world in our image if we destroy the past so people won't have to know the potentially terrifying future that awaits them.

APE is the rebranded (and apparently aggressively refactored) AdaptableStorage Zope product by Shane Hathaway. APE uses many of the patterns outlined in Martin Fowler's "Patterns of Enterprise Application Architecture" that deal with mapping objects to relational databases. I don't have the time to go into it now, but I want to point to this outline about the system.

A nice thing about APE, like the Enterprise Objects Framework (EOF), is that storage is heavily abstracted away from the persistent object itself. This actually allows for different storage mechanisms, such as the file system (making human-editable files) or other database systems such as MetaKit.

A recent comment on my earlier "Quick Look at WebObjects" post asked for a comparison of SQLObject and the EOF. I hope to get to this soon. I haven't actually used SQLObject, but it looks to be a useful lightweight system. It uses metaclasses to facilitate the mapping. But it really seems to be a simple (but I imagine very usable) solution. APE, Modeling, EOF are all full scale persistence systems. They all do things like faulting/ghosting (delayed fetching of objects), and should allow for conflict resolution, verification, serializing, etc. They move this code away from the object itself, which is the right thing to do in my opinion. Just like I flinch whenever I see HTML in a print statement (HTML and programming code DO NOT MIX), I flinch whenever I see words like column, table, etc in a business object.

Roundup is a good example of a system written with database abstraction in mind. It can store its data in DBM, MetaKit, Berkeley DB, SQLite, MySQL, etc. And you can (fairly) easily move from one storage system to another (basically, one just has to go through an export/import step, by my understanding). Advantages? I can tweak and work with my schema for my customized Roundup application and not worry about the database underneath. When performance becomes an issue, I can move on to a system like MetaKit or SQLite and have zero impact on my application.

Sometimes, it goes the other way around - needing to design an application around existing database data. Sometimes, that's the tricky one.

Richard Perle, humanitarian of the year:

If we let our vision of the world go forth, and we embrace it entirely, and we don't try to piece together clever diplomacy but just wage a total war, our children will sing great songs* about us years from now.

Referenced many places. By the way, this sounds like another chap:

Do you want total war? If necessary, do you want a war more total and radical than anything that we can even imagine today?

Who's this? Goebbels! We find ourselves in darker and darker company.

* By "great songs" - does that mean we'll have even more great hits by the likes of Toby Keith and Worley? Awesome (*cough*)! At least they're not traitors (according to a site that ends up being traitorous to any definition of 'freedom' that I have ever learnt outside of animal farm - a site that labels anyone against the President as traitors while completely slandering the previous vice-president. Self contradictory (and full of great aesthetics), one hopes to write it off as "they really know not what they do").

Or, will these great songs be sung by the likes of Boyd Rice?

While Michael Moore can be a bit of a loud boorish wingnut, he's a good counterweight to the some loud boorish wingnuts on the conservative side of the bully pulpit. Anyways, Moore has treated us to a story debunking the Oscar backlash:

And that, my friends, is the real point of this film that I just got an Oscar for – how those in charge use fear to manipulate the public into doing whatever they are told.

That was one of the main messages I walked away from Bowling for Columbine with. While it's a far from perfect film, it does raise some good points. The media plays up our fears, because it makes for a more captive audience. A terrific Saturday Night Live skit from a couple of years ago was the "Action 8 News Watch". The point of this skit was all of the news about the news that news shows start out with: "The President was assassinated today, but president of what? We'll tell you at the end of the hour". The rest of the skit was full of the usual "deadly common household items", which the anchors would hint at as being "very common, you or your children could be using it right now" and then say "we'll tell what it is later in the broadcast." Building arbitrary fear into the news has become a staple of American life, long before color coded alert days came into it. And now we have an administration that dishes it out daily. "This man is a menace. We must go in and stop him, before he hits us."

On a local NPR station this morning, a man called in swearing undying support of the Patriot Act, even saying it must be expanded and that yes - the government should know what you're reading and investigate you without the courts interfering. He said this is because "we are in grave times." Horseshit! The situation today is only worse than it was in 1999 due to our middle east actions. Otherwise. sigh In the ten years since the first world trade center bombing, there have only been two successful attacks on American soil, and one of them came from within. There were plenty of attempts, and they were all stopped without the need for the PATRIOT acts! Yes, one devastating attack got through, and it is indeed terrible and tragic that it happened. And it's good that the American public has woken up to the fact that yes, it is a dangerous world. But these are NOT grave times. Well, they're grave times if you're an Iraqi civilian that didn't survive the "shock and awe" and other campaigns, but they're not so grave for us. If you think they are, you are caught up in the propaganda that this administration and their media cronies are pounding you with. And that reminds me of this quote:

"Why of course the people don't want war. Why should some poor slob on a farm want to risk his life in a war when the best he can get out of it is to come back to his farm in one piece? Naturally the common people don't want war neither in Russia, nor in England, nor for that matter in Germany. That is understood. But, after all, it is the leaders of the country who determine the policy and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice, the people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked, and denounce the peacemakers for lack of patriotism and exposing the country to danger. It works the same in any country."

Who said it? Hermann Goering. My point: Be more cynical. If a quote from Hitler's appointed successor can sum up the current situation, I think it's safe to say that the current situation is fucked.

More reminders about why war sucks, especially a war that didn't need to happen:

With my own eyes I saw about fifteen civilians killed in two days. I've gone through enough wars to know that it's always dirty, that civilians are always the first victims. But the way it was happening here, it was insane.

...

I drove away a girl who had had her humerus pierced by a bullet. Enrico was holding her in his arms. In the rear, the girl's father was protecting his young son, wounded in the torso and losing consciousness. The man spoke in gestures to the doctor at the back of the lines, pleading: "I don't understand, I was walking and holding my children's hands. Why didn't you shoot in the air? Or at least shoot me?"
[Laurent Van der Stockt, interviewed by Michael Guerrin for Le Monde, Translated for CounterPunch by Norman Madarasz]

But don't worry, because now we realize that it's really Syria who's the problem, not Saddam. If we hadn't killed all these people in the first place, we never would have known that we'd get to blame someone else yet again for our own ineptitude towards capturing Osama. Does the body count have to be equal to or greater than September 11th in order for us to feel justified? Or do we just have to prove that our Gods are the best? Is Russia next on the list? The United States and Russia have done more to arm these regions/despots/wingnuts we're now soiling our spoilt god-blessed diapers over than Saddam Hussein ever did.

How long before we have the Country Song "I can't even spell Syria, but let's bomb them anyway"?

Anyways, we're still seeking out our justification for the war and having a hard time finding it. I, like others, really wouldn't be surprised if something was eventually found - but for something so dangerous that we were so certain was there, why can't we find anything?

After days of intense digging and searching at least seven suspicious sites near Nassiriya, U.S. experts have found chemical warfare protective suits, but no chemical weapons.

Chief Warrant Officer Alex Robinson, leading the U.S. search in the area, admitted on Thursday that his list of suspect sites in southern Iraq was "kind of drying up". [Croft, Adrien]

We still have a ways to go before this is all over and some stash of these things may be found, but it's certainly not looking to be the terrifying planet killers that we were constantly warned about. Don't you think a few more months of hard-lined inspections could have yielded the same results? Seriously - there are probably more chemical weapons just a few miles west of where I'm typing this than there are in Iraq (and yes - those weapons are being destroyed).

Salon reprints Tim Robbins' speech to the National Press Club (a lengthy excerpt of which is also available from Daily Kos. Full text is also available from World Net Daily).

A relative tells me that a history teacher tells his 11-year-old son, my nephew, that Susan Sarandon is endangering the troops by her opposition to the war. Another teacher in a different school asks our niece if we were coming to the school play. "They're not welcome here," said the molder of young minds. Another relative tells me of a school board decision to cancel a civics event that was proposing to have a moment of silence for those who have died in the war because the students were including dead Iraqi civilians in their silent prayer. A teacher in another nephew's school is fired for wearing a T-shirt with a peace sign on it. And a friend of the family tells of listening to the radio down South as the talk radio host calls for the murder of a prominent antiwar activist.

...

In this time when a citizenry applauds the liberation of a country as it lives in fear of its own freedom, when an administration official releases an attack ad questioning the patriotism of a legless Vietnam veteran running for Congress, when people all over the country fear reprisal if they use their right to free speech, it is time to get angry. [Robbins, Tim]

Keith Knight (of The K Chronicles fame), rewrites Natalie Maines' apology statement:

As far back as I can remember, I heard people say they were ashamed of President Clinton. I saw bumper stickers calling him everything from a pothead to a murderer. I heard people on the radio and TV like Rush Limbaugh, Pat Robertson, Newt Gingrich and Trent Lott badmouthing the President and ridiculing his wife and daughter at every opportunity. I heard LOTS of people disrespecting the President. So I guess I just assumed it was acceptable behavior.

...

And most important of all, I realize that it's wrong for a celebrity to voice a political opinion, unless they're Charlie Daniels, Clint Black, Merle Haggard, Barbara Mandrell, Loretta Lynn, Ricky Skaggs, Travis Tritt, Hank Williams Jr, Amy Grant, Larry Gatlin, Crystal Gayle, Reba McEntire, Lee Greenwood, Lorrie Morgan, Anita Bryant, Mike Oldfield, Ted Nugent, Wayne Newton, Dick Clark, Jay Leno, Drew Carey, Dixie Carter, Victoria Jackson, Charlton Heston, Fred Thompson, Ben Stein, Bruce Willis, Kevin Costner, Arnold Schwarzenegger, Bo Derek, Rick Schroeder, George Will, Pat Buchanan, Bill O'Reilly, Joe Rogan, Delta Burke, Robert Conrad or Jesse Ventura.

Remember - we have yet to hear a real, honest, genuine reason for going to war (apart from a show of American military muscle against a known weak target to keep some of the more extreme monsters, crafted with guns and kindness by the United States, in line with our expectations of the order of the world). And asking for the government to be accountable is inching closer to a hanging offense. Sing along with the government issue phrase of the day, or we won't let you watch baseball with us!

Stupid city - A Mighty Wind, the new Christopher Guest movie, opens in "select cities" today. Apparently, we're not selected.

I'm pretty excited that the great Mac OS X diagramming tool OmniGraffle 3.0 is out. I've covered OmniGraffle before (see "Sortof Agile Modeling" for the beefy post), and this new version looks to be even more intense.

One of my favorite things about OmniGraffle is that it's flexible without getting in the way. While Visio may pack enough features to do heavily detailed technical diagrams based on real data, it was never easy to just be able to pick assorted object or other shapes and just begin drawing. OmniGraffle, on the other hand, gives you plenty of default objects to be useful (with plenty more to choose from), but puts no limitations nor expectations on how you use them.

Also in the news lately is Safari Beta 2. I'm very impressed with this version, as the last official beta release (v60) and some of the unofficial ones (v64) were pretty disappointing. But now, Safari is rendering Plone sites correctly, and not having bizarre connection issues to Zope, Roundup, or a slew of other dynamic servers.

Oh, and our playoff tickets for the Jazz came today.

Inspired by Chris Petrilli (and also by looking at the URL's of some other Movable Type weblogs), I've applied an archive redesign. The new design yields nicer looking URL's, such as archives/2003/04/14/comparing_zwiki_code_bases.html instead of archives/002342.html. Granted, the second path is much shorter, but the first one is much better at conveying intent. I especially feel this is important for long entries such as the "Comparing Wiki Codebases" one.

I don't know if I'm going to bother applying the SkunkWeb stuff that Chris does to do redirects from old archive entries to the new ones. I'm awfully busy lazy.

Update: Chris applied the rewrite rules for me, so old articles will lead to new ones. This is done via SkunkWeb, and Chris writes up some details here and here.

I'm still slogging away on multiple projects, trying to bring a big mess of scripts under some more component-like control for Zope 2. It's working out, somewhat. It's nice to have some of the big pieces of code out of the ZODB and under CVS control, and it's also nice to be able to reorganize very large scripts into classes, where refactoring is easier to apply (thus breaking the big algorithm into more manageable chunks).

This week, there is some interesting code to compare: the inheritence-and-member-heavy ZWiki for Zope 2 versus the interoperable-components of ZWiki for Zope 3.

Before I begin, however, I want to discuss my, um, "prejudices" against Wikis, and other disclosures. Firstoff, one has only to visit a page like ExtremeNormalForm in the big c2.com Wiki to understand my frustrations. This page is marked as recommended on the big WikiPagesAboutRefactoring page. The WPAR page itself is a frustrating exercise, as a big batch of bulleted lists, slightly organized, with little or no descriptions next to the links. Except for a word like Recommended. "Aha!", I think, "here's a good page to keep around when thinking about database design". Wrong. I click on it, and get a poem. It's a clever poem and means well, but it wasn't nearly what I was looking for. So I click on ExtremeNormalFormCommentary. Alright! Here's some meat in the form of a discussion. Ummm, drat, it takes a while to then get to ExtremeNormalFormDefined. The first thing THIS tells me is to see ExtremeNormalFormDefinitions. Which turns out to be about as useful as ExtremeNormalFormCommentary, only less so. So, I realize that this is not at ALL what I was looking for, I have not found any good refactoring thoughts, and I'm extra cranky about the Wiki telling me this was a recommended page. What was I wanting? A real narrative. Something to tell me "this is what this really is" in a cohesive document. Instead, I'm clicking and trying to parse page after page in a circular pattern, with no indication about whether ExtremeNormalForm, ExtremeNormalFormDefined, or ExtremeNormalFormDefinitions, is going to be the real page that I'm looking for - especially because all of them seem so helpful in their title.

Now, I know that Wiki experiences don't have to be this bad. But they usually are. Some are well maintained (the Zope 3 wiki is actually kept up pretty well). But all too often, the experience lapses, and you have four pages covering a single topic, none of which give any useful information to a casual visitor that isn't in on the secret Wiki club. Page after page of negative surprises. And don't get me wrong - I've gotten some really good information out of the Portland Pattern Repository wiki, but it can often be an arduous experience.

In any case, I do know that Wikis have their purpose and some of them are quite good. And, in any case, this entry is really about an architectural comparison between the primary Wiki implementation that exists for Zope 2, and one that was written for Zope 3 (the component architecture based Zope re-implementation). A dangerous thing to perceive is that criticism of Zope 2 and software written for it, in comparison to Zope 3, means that Zope 2 is bad. We're serving some huge sites on Zope 2, and have often been able to deliver full featured applications quickly on this platform that have run for months and years. But, there are some known issues with the way that Zope 2 development has progressed and how that tends to impact software written for it.

Zope 2 often makes it easy to do the "just lump everything into a class or into a fleet of mixins" style of development, which is how much of the Zope 2 codebase is written. While Zope 2 still works quite well, experience has taught us the limits of this system. It's often desirable to defer action of to delegates and helper objects. But making registries for those objects and being able to configure/register new helpers is not an easy task in Zope 2. It's doable, but (especially without a configuration language), it can become dangerously fragile. In any case, it's interesting to compare the 'ZWikiPage' class from ZWiki for Zope 2 - with 10 direct superclasses - to the 'WikiPage' class from ZWiki for Zope 3, with only one direct superclass (Persistent). The ZWikiPage class for Zope 2 is huge - even without its superclasses - as it pushes everything needed to be a Wiki page AND a Zope Object into this one class. And if one wants to customize the class, by adding or changing the available renderers (to include reStructuredText support, for example), one is typically told "just add another method to the class and change the static class member that lists the rendering options". Well yes, it is an easy way to work, but it presents new software configuration problems. If I add a new to ZWikiPage 0.17.0, and 0.18.0 gets released, I have to manually move my changes over to the new code. Again - Zope 2 makes this mode of development easy, perhaps too easy. Much of my own code has suffered from similar problems in the past because it was the easiest thing to do.

But it seems the tide is turning against this style of development. And it has been for some time. At times, the tide seems to be against Object Oriented Programming itself. Personally, I don't think OO is bad, nor that it's going to go away any time soon, but I can see the problems people have with it. Big heavy classes, heavy inheritance trees/mixins, it all becomes too easy to do. As Todd Coram once explained to me, it's as though the craze at one time was that a juice bottle would know how to drink itself and throw itself away, before everyone realized that it really just needed to know what to do when something else drank it, and it certainly didn't need to know anything about throwing itself away.

One of the more famous Design Patterns is the Adapter. Zope 3 uses these with a vengeance. One of the principal benefits is that it's so much easier to write a business / content object as doing just what that object is interested in, and then write adapters to make other parts of the system happy with that object. Let's look at the wikipage.py module from the Zope 3 version of ZWiki again. The WikiPage class, as noted above, has "Persistent" as its only superclass. Furthermore, it has very few class members. Looking at the rest of the module, we see a lot of Adapters. One adapter makes a wiki page look like a readable file, one makes it look like a writable file. One adapts the wiki page to the expectations of the text indexing system. Another one deals with email subscriptions, and another one deals with events (and uses those events to send out emails to subscribed parties). So what's the big deal here compared with ZWikiPage.py from the ZWiki for Zope 2? Well firstoff, the code is easier to navigate. MailSubscriptions.addSubscriptions is much easier to pick out of a source code browser than trying to find addSubscriptions from a monstrously large list of methods. Secondly, I can replace any of these classes easily with my own code in my own module, using Zope 3's configuration language.

Likewise, all of the renderers for the Zope 2 implementation are thrown into the big ZWikiPage class, and short of modifying the code itself or doing some monkey patching, there's nothing you can do to add to that list easily. In the ZWiki code for Zope 3, renderers are also farmed out as separate little objects, and configured as such. If I *shudder* wanted to add WWML as a new rendering type, I could write my own module/package and register it as a renderer through the configuration language. Here, again, I wouldn't have to touch the existing ZWiki code (either directly or through monkey patching).

Well, the other big important part of a Wiki page is viewing it (and editing, I suppose). A lot of the code in ZWikiPage.py (Zope 2) version centers around this. But, for better or worse, it's intermingled with all of the other ZWikiPage responsibilities. The componentized ZWiki moves this responsibility into a collection of web browser view components, one of my favorite Zope 3 features. Sometimes, a simple template is all that's needed to make a "view" for some base business/content object. Other times, so much more is needed. Views are a special type of adapter for adapting content to various output mechanisms - web browsers, FTP clients, XML-RPC, and more. Look at the wikipage browser views - here again are a bunch of dedicated classes. Some deal with setting up the mail subscriptions, others deal with adding comments to a page, and others even provide custom Browser Widgets. And of course, there is a view object to deal with rendering a Wiki page and generating all of the Wiki links. If one looks at the view code, you realize that none of it knows about StructuredText, reStructuredText, and so on. It asks the registry that I mentioned earlier. It uses the registry both to list the available rendering options, and to get the selected renderer and use that to generate the primary HTML.

So, with all of that touched on, what are some of the downsides to Zope 3 development? Well, there's the configuration language, ZCML. ZCML is necessary, however. It's what glues all of these things together. It's how you really register all of these little classes and objects into the global framework. It's something we have to live with, and it has a very flexible design. But, at the same time, it sure is a lot of typing that needs to be done in order to put all of this stuff together. It's Zope 3's most powerful feature (because it gives total control over configuration to anyone who wants it), but I still worry that it will also be a big stumbling block for new developers.

On the upside, all of this yields a very flexible and powerful system. Through the use of Adapters and Views, you can add all sorts of new functionality to software, including adding new XML-RPC interfaces to, oh, whatever you want. You can make distributable versions of Zope that are preconfigured to run a specific application (you can do this now, but Zope 3 makes it easier through its configuration language). If other code is written to use registries, adapters, etc, it's usually easier to extend by registering new components to be used with it. There are a lot of strong possibilities.

A lot is going on, but there is little time to talk about it.

The "whole war thing" is proceeding on interesting paths. Sometimes we hear Rumsfeld spinning lines that seem meant to rival the old Iraqi Information Minister. Weapons of Mass Destruction? They were such a terrible and opposing threat, yet we can't even find any. I'm not saying that they're not there - they very well may exist. But if they were such a threat, where are they? And why is the US putting its own inspection team back in that has very little credibility? And in the meantime, what is happening on the homefront? Oh yeah - if the FBI wants to know what you're reading, the library has to tell them. And the library can NOT tell you that the FBI knows. While at times, I may just give up and say "everyone knows what I'm buying, what I'm reading, etc." (it's all an artifact of these times of immense data mines as we become ever more electric and enslaved to the convenience of credit/library/gas/access cards), I'm not comfortable with the concept that I can't be told WHO has taken an interest in my reading list. (A former roommate once checked out books on airport design after reading some article about AI used in airports in Wired. A day or two after the checkout, the FAA called. It could be coincidence, but that's the only time they've ever shown up on the caller ID, and in the springtime following the terror attacks of September 2001, it's not a coincidence that sits well in MY stomach).

What else? There's more from the programming camp, coming up...

I started off the morning looking at Bill Bumgarner's post, "What do I need in a blogging tool?" Bill's post, being written as his Radio account is due to expire in a couple of days, has a pretty good list of different weblogging tools and their strengths and weaknesses according to what he is looking for (hosting, etc). What caught me in the post was WOWebLog, a WebObjects based content management/production system.

I've always been interested in WebObjects, but have never put the time into giving it a serious examination. Partly, I have no real interest in picking up Java again (I touched on it lightly back in the 1.0.2 / 1.1 days). Mostly, I just don't have time to try something new. But WebObjects has always had my respect, and I've long viewed it as the best ideological competitor to Zope. One of WebObject's primary strengths is its use of the powerful Enterprise Objects Framework, a very powerful persistence layer specializing on object-relational mapping (but also works with flat files or LDAP directories). EOF is the most complete O-R mapper I've ever seen. It takes great pains, like a lot of the old NeXT software, to remove the mapping code from the business code. As you model your database tables and how they map to objects, EOF makes a Model that is used to define the communications between business objects and the database - SQL (or LDAP or Flat File) code does not get generated into the user code, a very different approach than what most people do. This allows you to, in theory, completely change out the Model without having impact on the code - ie, an application can be prototyped against a simple flat file system and moved to Oracle or OpenBase later without requiring a rewrite of the application.

Anyways, this document provides a nice introduction to WebObjects 5, covering more of the Enterprise Objects Framework, the Direct-to-Web Framework, the automatic state management, and other layers that make up this powerful system.

I'm so tired, but thanks to all the daylight we're saving, my internal clock isn't telling me that it's OK to just get up and leave the office.

Work on our internal configurable components simple subsystem for some Zope 2 projects is going well. To recap - this is an effort to move a lot of critical business logic out of the ZODB and into "disk-based products". We have multiple sites now running off of the same base code, but each one has had some customizations done. This makes applying critical changes more difficult, as it has to be done on almost a line-by-line basis sometimes. The CC plan is to allow for the differences to be programmed in to the base framework, and certain configuration options are introduced to set up the customer-specific rules. Now, even the configurations can be stored on disk and checked into CVS (as soon as I get comfortable with a packaging solution). This is done using ZConfig as the configuration parsing and processing framework. It took a fair bit of effort to come up with a simple ZConfig schema file for the first configurable components, but the actual config files themselves are easy to read and easy to read and use in code. The next step is to be able to generate a configuration file based on the current configuration of a site.

Tres Seaver: "Pluggable components are good; configurable components are better," from a series of slides speaking of "From CMF to Zope 3: Lessons Learned".

I've heard this line before about Zope 3, at least once (only once to my recallable memory), and it stuck out at me then, and sticks out even more now. Actually, today it hit like a bolt of lightning. It just might be in how concise the phrase is, due to it being a bullet point in a slide... No, it's more substantial than that.

Last week we hit a major performance issue with a web application that is in its transition phase. A page was taking up to twenty or more seconds to load, which is absolutely unacceptable. Theories and ideas were batted about, but it seemed to be the fault of one call that was being made a LOT of times. We started thinking about advanced caching schemes, and how to be able to start migrating some of the data access calls to read and flush the cache appropriately. But I thought "this is a query that could really be done once instead of so many times." That idea was (initially) shot down. Upon investigation, however, it seems that due to some TAL/Dreamweaver quirkiness, the data access method was being called at least THREE times for every item in a product matrix, that on the page in question featured roughly 30 items - ninety database calls for only thirty uses of the call that could be reduced to one. I found the spot in the code where this single call could be inserted and put in a simple mapping structure that the client (the template) could then access in place of the original data access method. Boom!, major speedup - especially after the erroneous TAL was cleaned up (even if it stayed, it would have had far less impact on performance).

But as I was applying this code and thinking of another site that this applied to, I realized that this script had interesting differences between the two sites (one in transition, one in early development) that used it. A problem that we've been tossing back and forth in our heads has been the issue of how best to deal with a fairly common Zope site scenario - a site built out of templates, scripts, and SQL calls, organized in folders in an object database. While this makes development and some degree of maintenance (especially emergency maintenance) delightfully easy, it's a pain when it comes to Software Configuration Management. It's fine when doing a one-off site, but becomes painful when trying to maintain many sites, with a couple in different development stages at the same time. For the past few months, I've been moving some fairly generic code into Python modules that could be imported into Python Scripts for utility purposes and to help move some common code into a real base framework (after many false starts) and into CVS for source control. But that hasn't addressed how to really start dealing with moving these massive scripts out of the ZODB and into real manageable Python code in source control. Not wanting to leap on anything until a good solution presented itself, we've waited and thought and brainstorm and waited while other more important projects took up valuable time. But todays events - reading over Tres' slides, thinking over some recent Zope 3 debates (that seem largely to focus around the need for configuration languages), seeing the release of ZConfig into the Python Package Index, and finally trying to deal with two code forks of unmanaged software - suddenly brought a solution (at least to this particular situation) into mind. A few heavy hours of coding later, it basically worked on first try. The client page of the one (fairly large) script that I "componentized" was completely unaware of the change.

So what went into this? First, I knew that I needed Formulator on my side, but wanted to define my forms in Python Code rather than through the web. Basically, I wanted a way of describing configuration properties that was closer to Zope 3 Schema's than to Zope 2's usable but not as expressive PropertyManager system. Snooping around Formulator's CVS tree, I finally came across FormulatorDemoProduct, a rudimentary Zope 2 product that shows a way of using Formulator with Python based product code. Using FormulatorDemoProduct as a base example, I came up with a small persistent object called "Configuration", that took a list of Formulator Fields as an argument. This set of fields is used, like in the demo product code, to update a dictionary (PersistentMapping in my case) upon successful validation of the form data. Using import as to shorten the name of StandardFields to the more Zope 3 style 'schema', a configuration definition can be built out of descriptions like the following:

from Products.Formulator import StandardFields as schema
LevelConfigurationFields = [
    schema.LinesField("text_data_blobs",
                      title="Text Data Blobs",
                      description=(
    "Subcategory identifiers for entries "
    "in the generic blob table that map "
    "to this particular level's id."),
                      required=0,
                      width=20,
                      height=5,
                      ),
    ]

Next, ConfigurationHolder comes into play as the base class for the service objects that will be replacing certain folders in the system that contain nothing but scripts and SQL methods. As a ConfigurationHolder, a few special methods and a Zope Management screen is made to list and access the configuration objects within each, and each configuration object has a Zope Management screen built automatically off of its Formulator data. The above example might exist as a subobject level_config. Since all of this is built purely out of descriptions (a style preferred by systems such as Naked Objects, which was brought up early during Zope 3 development and I assume contributed heavily to the power of the Schema system), adding new configurations and configuration holders should be relatively easy. We have the ability to look at certain pieces of business logic and ask "what might change here?" and make new configurations off of it without worrying about making a usable user interface for dealing with these changes.

Finally, there's the business and data access logic itself. Much of it exists in various Python scripts of various sizes, dealing with many different aspects of the system. While Zope Python Scripts are great, flexible, etc, it's easy to fall into the same maintenance nightmare that one might run into with a large application written in poorly crafted PHP/*SP pages, where large scripts dominate the landscape instead of modules and classes. Once you have a migration route out of those large scripts, which sometimes can't be found until an application is far enough along to fully understand a business problem, methods of Refactoring can finally be applied. I've done some refactoring in script land already - about as much as can be sanely done - by keeping the scripts that actually respond to HTTP requests small handlers that call into the larger "common" scripts. Usually these handlers make one call to a large script, respond to exceptions raised, or go on after a success. This allows the handlers to be tweaked to deal with UI changes without affecting the larger code of the back system, but ultimately I've been weighed down for months with the thought that "there's just too much back there". Now these scripts are being taken apart using common refactoring techniques such as Extract Method to be applied. Now code is in manageable chunks, code browsers can navigate it more easily, and performance is actually increased a little bit since it doesn't have to run in restricted python mode. And then, like the little handlers on the public side of the site, the ConfigurationHolder becomes a simple front end, or Façade to the complex business object behind it. As the interfaces of these Façades stabilize over the next few iterations of the software, I'm now (finally) confident of the software's adaptability in a way that doesn't compromise deployment and configuration management.

It's a system that I always wanted to develop in a component oriented way, but couldn't see how to do it until just now. We've had a few component based elements show up in the design already, but I think I was focused too much on trying to componentize the data (all in a relational database) without focusing on the true meat of the system - the business logic that operates on all of that data.

Interestingly, we sort of did design the system as replaceable components, hoping initially that these folders of scripts would yield to something better. It wasn't until the realization of component configuration came along that it worked out though. Bringing us back to: "Pluggable components good; configurable components better." Whew! Next stop - XML import/export of configurations so that the different component configurations for customers can be tracked by the SCM process (such as it is).