Schema Validator

I might have a Pattern here. Ever since I started implementing the system outlined in the May post, "Component Relational Mapping?" I've been employing a variety of patterns. Most have come from Martin Fowler's book Patterns of Enterprise Application Architecture, others from the venerable "gang of four" book, Design Patterns. And I've noticed some of our own patterns start to emerge.

The key pattern is something I refer to as the Schema Validator. A Schema Validator takes a raw clump of data, preferably in some sort of key/value form, and does one of two things - return a new data object with all of the values in 'correct' format/typecast, or raise an exception with a collection of all failed validations.

It's this pair of responsibilities that make the schema validator such a powerful component. The ultimate responsibility of the validator is to give comfort to the developer or integrator that when data is passed to lower levels of the system that will be responsible for storage or messaging, that data is known to be "correct." That is - integers are integers, floats are floats, dates are dates, credit card expiration dates are credit card expiration dates. More than that, any further restrictions on the data, such as an integer range, string length, etc, have already been checked. It severely cuts down on the possibility that an exception will be raised down at the storage level for attempting to insert an invalid value.

I've come to rely heavily on this patterns as I've worked to abstract the storage layers of my web applications, typically layers which speak to an external database such as MySQL or OpenLDAP. It allows me to comfortably generate dynamic SQL statements, particularly DML statements (INSERT, UPDATE), without having intimate knowledge of the schema. These lower serializers or gateways can loop over the data that's passed in to them and then format it for the target data storage, especially important when dealing with date/time storage in SQL systems (which still seems to vary wildly between databases).

To date, my Schema Validator system has basically been the Formulator product for Zope, courtesy of Infrae. Formulator also gives me HTML Widgets for HTML form generation. This aspect is very nice (it cuts down on the amount of manual HTML writing one has to do to deal with forms), but it's the validation system that I absolutely depend on. Most of the time, this allows me to describe my data model once, and use that model to generate user interfaces and to ensure that the data I depend on is relatively strong.

I think work might start soon on the next generation of our system. Formulator has been very good to me thus far, but on a recent project I started to run into two issues that I think are going to present themselves again in the future:

1. Multi-field Validation. Formulator goes on a field by field basis. In my systems, I need to add extra elements to the framework to do more complex validation that may impact/check multiple fields. It's preferable that these would execute inside the same 'validate_all()' command.
2. Context/Security Sensitive Widget Display and Validation. On any level of granularity, I need to apply an extra set of rules about who can do what to a particular field. On the display side, this entails making certain fields read-only, or maybe not displayed at all. On the validator side, this could mean ensuring that certain values that the current user does not have access to are dropped out of the 'valid data' structure.

I'm also noticing a lot of duplicated code which I haven't had time to refactor. Mostly, this concerns manually passing the same validated data block to multiple Table Gateways - the components which actually take one of these validated data structures and prepare the data to be sent to storage. Instead of multiple manual calls in the same Controller/View component, there should be one. Ape uses an 'event' object that gets passed along to its gateways, and those gateways make notifications in the event object about which elements in the data block can now be ignored by other gateways in a chain. I like this idea a lot.

Ultimately - the patterns that we've started to deploy in our Zope apps since May have really started to yield stronger bases, and much stronger user interfaces. Many of these patterns come from Zope 3, P of EAA, and other places. The ideas are already out there, and many of them are out there for good reason.