While you may order that inflatable, remote-controlled zeppelin online, the acknowledgement nevertheless comes via email, as does the receipt, and the shipping updates.

Email is the truck that moves freight – light and heavy – on the information-super-goat-trail. Plain, simple, elegant, boring, your-grandma-has-an-AOL-address-type email remains the venerable heavy lifter of the online world.

Strangely, it has also become the de facto identity management tool. It is universally used to authenticate just who we are, on everything from my bank to the myriad of social and anti-social real-time networking sites. When we forget just who we are, it’s the delivery method of choice to jog the memory or to trigger a reset — ironically, given how totally insecure it really is, likened to a postcard.]

But, the core problem with email is not security. The real problem with email is it’s really stupid. It’s dumb as a bucket of overripe bananas. I mean it. It’s really god-awful stupid. It can’t help it. It was designed that way.

When push comes to pull, with email, you really don’t get much, and that illustrates its frailty and its amazing functionality.

With email, you see, all you really get is an “envelope” (consisting of minor variations on “From,” “To” a “Subject” along with tiny little bits of routing data that nobody pays attention to) and a giant blob of undifferentiated stuff called a “message body.” What’s in that message body is anything, unstructured, and undifferentiated – a blob.

What’s in there could be secret silly croonings to your one-true love, it could be the confirmation of your getaway flight to a land without extradition, or it could be my secret recipe for the world’s best gazpacho (and hence your necessary and immediate flight from justice).

With email, the medium hides the message. (If I keep this up, I’m likely to be haunted by McLuhan.)

Since its launch in 1971, we have improved it. We’ve tweaked it and twiddled it. We’ve made it better, making it easy, for example, to stuff it with bits of binary. We’ve said goodbye and good riddance to Uuencode and its ilk. Now digital civilians needn’t know a MIME type from a mime troupe. We’ve prettied it up, too — love it or hate it — with HTML, providing that ever-so-useful ability to deliver ugly fonts, in all the sizes, shapes, and colours your little heart could desire, rendering it pretty much unreadable

[For the record: I think I sent my first in the fall of 1979, using a service called EIES. I've got a copy of it around here someplace. It was a message to the fellow at the desk next to me, suggesting we get lunch at the Burrito King (tacos al carbon); truly important stuff!]

And, so, email moves the world, moving commerce and confirmations, in the wink of an eye. We’ve filled the tubes with everything from solicitations for various dysfunctional systems (whether erectile or congressional), to orders for green tea, multiple drafts, rewrites and painful iterations of your latest annual report, and those important PDF copies of your sinister plans for global domination through puppy adoption. They’re all sent on their respective ways via email.

Email is the go-to tool for everything from “donuts in the kitchen,” to presidential elections. But, inside, it’s still one dumb blob.

Blob, meet the software equivalent of Steve McQueen: Email2DB– one magnificent tool.

Steve McQueen (saving diners) in the Blob!

Email2DB has become a necessary cog in my machinery. It lets me take those dumb blobs and ferret out the necessary bits and pieces of the message, shaping them, cleaning them up, adding value in terms of structure, and then, gently slipping that data into a giant database.

In the best of all possible worlds, I wouldn’t have to do this. In that world, I wouldn’t be tasked with figuring out what to do with thousands email messages sent willy-nilly to just about any email address, person, or inanimate object you might care to imagine. But I do.

I am charged with capturing and organizing thousands of inquiries — inquiries that arrive in every way imaginable, some via a web form, others via email, and still others via such unspeakable things as (shudder) fax. I even think a few get slipped under the door at night by pixies. They’re all important, and regardless of origination, I want them all to end up in the same place — a database. Despite their disparate origins, I want them all channeled into the waiting, eager programmatic minds for review. I am all about the smooth flow of information. My motto: Never, ever, type it twice.

Email2DB keeps me true to my motto. In a nutshell, Email2DB “deconstructs” the email. It breaks it into its constituent parts, slicing and dicing the blob, parsing not only the header, but the contents, and gently slipping those deconstructed pieces into the database of your choice – in my case, the same-same database used to capture the content entered via a fancy online web form. Derrida would be proud.

Well known parts of the email —like the TO, FROM, SUBJECT, DATE — as well as some of the arcane bits and pieces of the underlying protocol (Originating IP, MessageID, ReplyTo) are a breeze to deconstruct.

They’re “pre-programmed” into the software, and with a single mouse click, you can pull those wee bits apart and slide them into a database. It talks to all-comers: Access, SQL Server, Oracle, MySQL, Access, ODBC, yada yada yada. It’ll even write it out as a CSV if you’re living in 1996.

Once you’ve extracted the pieces you can use them for nefarious purposes: perhaps to construct a new message, sending back, for example, custom acknowledgments, or forwarding on reformatted confirmations, or simply adding them into a database for further processing. Email2DB takes email and turns the contents into fields and records in your favorite database.

The Taming of the Blob -- Smart Parsing for Dumb Blobs

Fancy stuff is easy. Within an hour I was rolling my own routines to parse more bits and pieces from the message, isolating the “First name” and “Last name” from the so-called “Friendly Name” portion of the “From” field. “Don’t stop there,” I said to myself.

So, I tackled the blob itself. With a little head-scratching, a smattering of OOP concepts under my belt (and a passing familiarity with VB), I was able to deconstruct bits of the body of the message itself, scanning through the text for familiar references that might match my mighty database elements.

With only a little fancy footwork, I was even able to detach any attachments, saving them with a unique “key” to a SharePoint document library, along with a PDF copy of the original message (also tagged with the same unique key).

The beastie will read and process messages from POP3, IMAP, and Exchange servers. It will also read and process messages directly from Outlook folders, including Exchange “Public Folders.” It has a fairly full-featured scripting language, and variables, once created are reusable.

While this is all well and good for me and mine, the beauty of this product is its universal application. There’s not a week goes by that someone on doesn’t ask me for the easy way to get information from a web site to a database. While there are a myriad of ways — some are easy and some are not. None are as easy as email.

Moreover, if it’s email generated by a web form, you control the structure. If the structure is predictable, Email2DB can easily grab that email, work with your structure, find the right bits and pieces, deconstruct them into the raw data you need, and then, easily slip that deconstructed data into an eagerly awaiting database. All is right with the world.

The requirements are minimal. The Email2DB software costs $300, $500 or $1,000, depending on features. I went with the $500 copy, as I needed the scripting engine and attachment processing. You need an email account (any will do, including those reached via SSL). Finally, I run it on a virtualized XP machine, rigged to autostart, autologin, and autorun, should anything interrupt its dedicated rounds.

Overall, the customization took about three days. I created scripts for:

• pattern matching to extract first and last names
• file-renaming to save copies of the original message and attachments to a SharePoint library
• Unique (per message) tags so that saved items could be retrieved as a group
• URL constructions so the database could include links to the original message and attachments
• Other custom flags for the source, date and time received, and type of inquiry

Whether you’re looking to manage the email meteor shower, stave off an invasion of unstoppable email blobs, or just want to turn a few dumb ones into smartly structured data, Email2DB can do it. It’s not often you can find software that will not only stop an alien invasion, but will also send you an acknowledgement when it’s done. Steve McQueen not included.

• You can chop-and-paste from one monitor to the other, keeping a browser open on one monitor for… uhm… err… research and your Great American Novel front and center on the other.
• You can set different resolutions on different monitors. This lets you quickly see through other eyes, a handy thing when designing web pages, especially if you have a penchant for extra-large (or extra small) fonts. Guilty, I am. I often forget that some people like their icons larger than a pinhead and text measured in multiple microns.
• You can run multiple flavors of browser — IE, Firefox, and Safari, maybe Opera just for grins — simultaneously making sure that nothing looks right on any of them regardless of what you do.
• Finally, for the A.D.D. amongst us, you can while away your day, in manifold multitasking, with more stuff in your face — calendar, email, task list, Facebook, ESPN and CNN, three or four or five or ten browser windows, slash-dot, iTunes, and a copy of the DMCA (just in case).

Running with two is easy to do. In fact, most modern, add-on video cards have two connectors, usually a DVI and a VGA (15pin DSUB) connector. Many now come with two DVI connectors. All you need do is connect up two monitors and click the check box in Windows (maybe twiddle with your BIOS), and — voila — you’ve got screen-space. Today’s LCD panels, like most hardware, are downright cheap too.

For folks that have to look at two things at once — such as when cataloging scanned documents, or working with document management systems — I recommend it. It’s almost a joy to view an item on one screen, whilst keying metadata on another. If you work with web pages, or graphics, or have to manipulate multiple things in multiple windows, it’s an amazing time-saver. It’s well worth the investment.

Wide monitors are also great, but having two monitors is even better. Better yet is having two wide monitors. Moreover, the first time you snap a window from one monitor to another in front of the uninitiated, the sudden gasp and resultant, “How did you do that?” is well worth the investment. People will think you’re cool and sexy. No need to tell them that all it takes is the dexterity to twist the screw connectors on a VGA cable — admittedly, that can be challenging.

Sadly, my home setup did not invoke gasps. It was, shall we say, embarrassing. Like the classic cobbler kids, my feet were unshod, my setup shameful. That shame came rushing home just a few weeks ago when a friend laughed out loud upon seeing the CRT monitor squatting on my desk like a 1950′s television. “Is that your monitor?” he snickered derisively. “It’s huge!”

“Huge” it seems, is no longer a desired attribute — at least when applied to monitors. That was it — I could stands no more — it was time to upgrade to something smaller.

Given the season, I decided I’d aim for a triptych — the holy grail of multiple monitors — the magic number three. Yep. Three monitors. It takes a wee bit more work than two, but offers a certain balance, a certain pleasing symmetry, if you will. I’m a great fan of symmetry. Besides, it looks really cool.

At home, my main monitor is a nice Dell 20-inch LCD panel. I got it cheap. It was staying. The other monitor — the behemoth, a huge 19-inch, 16 ton CRT — well, it had to go. So, I tucked it away, in my own personal white elephant graveyard (right next to the vintage Compaq Presario 526 and the Dell Optiplex G1 running OS2 Warp.)

To replace the CRT, I scavenged a two Dell 17-inch 4:3 monitors. They were homeless; abandoned. (It’s amazing how quickly wide-aspect monitors have become de rigueur and 4:3′s are now so much landfill.) The desktop space gained by removing the CRT was amazing — leaving more than enough space for the third LCD — with a little left over for a DVD-stack and miscellaneous other stuff.

Then I went to work on the box. To run the third monitor, I needed new hardware. I had run out of video connections. The box is a Dell Optiplex 745 — not fancy, but adequate. It’s stuffed with all the parts; packed with 4GB of RAM and about a half a terabyte of storage. Slot-wise, inside, the beast sports one PCI-e(xpress) x16 slot, one PCI-e x1 slot, and a couple of regular old PCI slots. I haven’t a clue what you do with PCI-e x1 — and it looked awful funny — so I concentrated on the other two types.

My current video card, an Nvidia GeForce 8500 GT made by BFG, is in the PCI-e x16 slot. It drives my 20-inch LCD via the DVI. Since that PCI-e slot was full, I figured I needed a regular PCI card. My plan was to keep the main monitor (center) on the GeForce 8500 and let the new card (whatever it might be) drive the outriggers (left and right). Hence, the new card needed to support at least two monitors.

Checking my own highly-organized inventory (AKA: my drawer full of stuff) I did find a couple of old video cards from long-gone manufacturers, but none worked. Totally irrelevant, I also found:

• Five old Cue Cats;
• A half-dozen old mice;
• About four thousand PC power cords;
• An OEM copy of WordPerfect Office for DOS;
• A Sharp “Wizard” PDA (circa April 1991), and;
• An “Ely Culbertson” mechanical card shuffler with the crumpled instructions for an “Ultrasonic Rodent Repeller” stuffed inside.

While briefly entertained by the cosmic juxtaposition of mice, Cue Cats, and “Rodent Repeller” instructions, it was immediately clear that none of this stuff was going to help in my quest. Consequently, as any geek would, I played briefly with the card shuffler, marveling at the mechanics, and then neatly stuffed it all back into the “parts” drawer, vowing to “clean it up later.”

I turned then to Google.

The collective Google geek consensus was: “Don’t mix video drivers.” In fact, said the Google, your second video card should be in the same chip family, or at least a kissing cousin. In English, this meant I needed a video board with an Nvidia 8xxx chipset, if not another actual 8500 GT. If I did that, both cards could and would (or should) use the same driver.

After a disappointing trip to the local Best Buy, where they never have what I want and it’s all overpriced, I tried a local computer hack-shack. No luck there, either. Next was Newegg. Even there, it seems, my options were limited unless I wanted to replace everything. I considered this, briefly admiring some quad-head (four monitors!) boards, but didn’t bite.

Eschewing the high-priced options and sacrificing instant gratification, I went cheap, crossed my metaphysical fingers, and ordered a PCI card with an Nvidia 8400 GS chip — I figured 8400 was close to 8500 …

Newegg, by the way, is terrific — excellent user interface, terrific prices, and good service. The board itself — a Sparkle GeForce 8400 GS 512MB GDDR2 PCI — was sixty bucks. True to form, and as promised, Newegg had the board here the day after Christmas.

Sneezing, I slipped it in to my PC, vacuumed out the dust bunnies, crossed my fingers, thought nice thoughts about churros, and slapped my head three times with a copy of Vista Premium Ultimate Galactic Omnipotent Edition and… it didn’t work.

Three Monitors: Diana Krall singing “A Case of You” and a glass of Syrah.

Yet, after a little tweaking, some irascible grumbling, and a couple of reboots — followed by a frantic yet fruitful hunt for a VGA cable without a bent pin — nirvana was mine.

My triptych was complete. I had trinity — three monitors — no muss, no fuss, no waiting. All that remained was to hook up all the USB hubs and try to gain some semblance of order in the cable chaos I had created. With three monitors, I discovered I had run out of power outlets and had to spring for another power strip — once again proving that no tech project ever comes in on budget.

The plethora of USB connections, by the way, was an un-expected bonus. I had forgotten that each monitor had its own USB hub, each offering four USB connections, two (totally unreachable) in the back and two on the side.

Unfortunately, DELL, in its cosmic wisdom, can’t seem to decide which side of a monitor to place the USB ports. On two of the monitors, the ports are on the left. On the other monitor, the ports are on the right (and reversed, back to front). This setup guarantees — no matter what way you turn the USB connection, when you try to stick it in the slot, you’ve got it backwards.

Three Card Monitor Monty

There is one more thing I should mention. Once done, you need to shuffle the monitors around on the “Display Properties” tab — making sure you’ve got them in the order you want. I wanted mine with the primary (20-inch LCD) monitor in the middle. Each monitor is numbered, so you just drag and drop the little image of the monitor where you want it. Easy as Three-card Monte.

We travelled together, she and I, bisecting France; from Brussels to Aix-en-Provence and back again. In hindsight, I couldn’t of done it without her. How I ever planned to survive, travelling those weeks without her is beyond me. I’d have been lost without her, lost.

She was my constant companion, Jane. During the day, driving, she was there. In the evenings, she’d accompany me on walks — short or long. She was never at a loss for words; she always knew just what to say. I’d hang on every word. She was worldly in ways I can’t describe. We became close in those few days together; I could sometimes even anticipate what she’s going to say next. “Oh Jane,” I said. “Run away with me. Together we’ll see the ends of the earth.” Jane, always taciturn, would say: “Go to the end of the road and turn left.” Jane had a sultry way about her, teasing yet stern, with that lilting British accent. I’d smile, knowingly. “You have reached your destination,” she’d say firmly. Then, I’d reach up and – ever so gently – pluck her from her adhesive perch on the windscreen and tuck her into the glove box. Yet, in the end, I left her. I left her in Brussels, back at the Avis counter. I had to. It was either that or lose my deposit. Jane: the GPS lady.

GPS is a liberating technology. Jane — in the form of a TomTom GPS device — was amazing. To put it nicely: driving in a strange city can be, shall we say, flummoxing. To put it accurately, it can be frustrating, irritating, and downright dangerous — to you, the other traffic, innocent and not so innocent pedestrians, and/or your assorted travelling companions.

Driving in another country quadruples that frustration and danger. Not only are you perpetually lost, but the roads are wacky, some barely wide enough for a goat (and a skinny goat at that). The pace of traffic is fast and all the signs are in a different language. [To quote Steve Martin: Those French are amazing, they have a different word for everything!] Moreover, even if the road signs are roughly approximate to English, or you happen to speak the local lingo, everything is nevertheless somehow incomprehensible.

Now what were those “three laws of robotics” again?

I’ve wandered the world, driving here and there, always with some degree of angst, some lingering anxiety about the traffic, the other drivers, where I was going, or just where the hell I was. With Jane as my co-pilot that anxiety was gone. I could concentrate on driving, either at (very) high speeds on flawless French highways or feeling like James Bond as I curved around winding trails in search of coteaux and caveau, my (rented) Audi A4 Turbo Diesel purring, Amel Bent’s Nouveaux Français blasting on the Blaupunkt.

It’s odd. Technology was supposed to be enslaving, not liberating, the enemy of democracy, not its savior. We were headed for dystopia, an Orwellian future where technology was to be a black boot on the back of the neck. I watched the year 1984 creep closer and closer, big brother looming large. It came and went, with barely a whimper.

Now, don’t get me wrong: technology can (and has) been turned to evil ends. Much has already been said about the evil ends. The possibilities for more evil abound, even for a company who’s motto is “do no evil.” (It’s just a little to newspeak for my tastes.)

The Open Secrets Effect

But I want to talk about the other side. I want to talk about Jane, and things like Jane. I want to talk about something I call the “open secrets effect,” something that has the power to save one’s marriage from map-reading malevolence, and, perhaps, the power to save democracy from itself.

While it may seem a trivial one, GPS is a good example of the open secrets effect — that magical synergy you get when you mix disaggregate information with extraordinary computational power and deliver it in new ways. In the interest of honesty, I should say that I stole the name of the effect from Larry Makinson and OpenSecrets.org, a site published by the Center for Responsive Politics (CRP). That site is the brainchild of Larry and of Ellen Miller. Ellen is the former executive director of CRP. She’s now at the Sunlight Foundation. In the interest of full disclosure, I should also note that my wife was CRP’s long-time communications director, until the fates led her down a much more fun career path. She’s the reason I get to do these fun trips.

The site, OpenSecrets.org is about campaign finance. It’s also a prime example of the power of opening the kimono, of exposing information that has been shrouded in darkness and complexity.

The Open Secret Effect is what happens when you shine a bright light on data, making it not just available — there are lots of maps of France “available” after all — but accessible, understandable, and personal. When you do that, something magic happens.

For example, originally, Open Secrets was a book. It was published by the Center for Responsive Politics (CRP). It weighed in at probably 10 pounds and was a good four inches thick. But, when CRP took that information, mixed it with a dash of database and a smidgen of internet, suddenly they had an even more powerful agent of change.

That magic mix — data, database, and internet — made the information real and powerful. People could look up their “own” politician, and see just where the money came from. There’s a new twist today, by the way, an initiative called MapLight.org. It promises to take that information to the next level, marrying campaign contributions to voting records. So called “mashups” like MapLight — where two or more previously isolated sets of data are “mashed” together — potentiate the open secrets effect. “You have reached your destination,” says Jane.

That’s the power of the open secrets effect. The mere act of opening the kimono changes behavior and changes the balance of power. Now, more than ever, U.S. political campaigns (and politicians) are dominated by big-money interests. Those interests finance the campaigns and, through those dollars and donuts, pocket the politicians that shape the fabric of our lives. Hopefully, the jig is up. With the open secrets effect of MapLight and OpenSecrets, it should be much more difficult to be bought and sold, especially when everybody knows how cheap the price.

So too, the inherent “connectedness” of the Internet is also changing the relationship of money to power as well. Big money is still there (by the bucket-full in this particular election season) but it is being somewhat counterbalanced by so-called internet campaigns, campaigns that are using the ‘net’s ability to aggregate lots of small things, in this case small contributions.

For good or for bad, campaign contributions have been ruled as constituting “free speech.” As such, more people are speaking than ever before. [Sadly, over two-thirds of those dollars — regardless of source — just end up fueling the creation of traditional one-way TV spots, designed not to inform but to obscure, enrage, or distract. One-way media must perish from this earth.]

Nevertheless, today’s innovative (dare I say social) uses of technology have had a liberating effect. Instead of robbing us of rights, they have increased our participation, restoring power to the formerly powerless. It has strengthened our democracy, not undermined it. To paraphrase Al Gore in his (absolutely terrific) book, The Assault on Reason, “a connected citizenry” is our greatest hope. The new internet is all about connections and the open secret effect.

More and more secrets are being opened. It’s a revolution in knowledge, power, and influence. Suddenly the powerless are powerful, the disenfranchised are raised up. It is something that can change the course of history or something that can get you from village “A” to vineyard “B” (and back again). “Take the roundabout, second exit,” says Jane.

Opening this secret can of worms has the potential to turn power on its head — counterbalancing previously one-sided relationships or creating strength of numbers where there was none previously. It’s directly responsible for the new realization that “consumers” are not passive patsies but active participants. Nowhere is this truer than in the development of software, for example. Software publishing has become a dynamic, interactive process where the customers participate in the product’s development, even doing the product testing.

Previously top-down, one-sided relationships are being changed — for the better IMHO. For example, let’s look at some previously one-sided relationships: between the entrepreneur and the venture capitalist, between the Fourth Estate and the public, and between grantee and grantor. These are being turned on their heads — they’re feeling the open secrets effect. They are being forced to operate in an environment where the formerly obscure is now in public view, i.e., the open secrets effect is at work.

In one of my favorite examples of truly living the Open Secrets life, Southwest Airlines actually made a television show of their inner operations called “Airline,” with a tag line of “We all have our baggage!”

Meanwhile, a site called TheFunded has turned the tables on the world of the holy venture capitalist, dishing up a place that allows entrepreneurs to rate their would-be suitors. TheFunded has changed the equation by opening up the secrets that everybody “knew” but nobody shared. TheFunded has aggregated the voice of the powerless, and in so doing, become powerful. Now the VC’s are beginning to understand the real business they’re in and the nature of their relationship with their customers, the entrepreneur.

A more Web 2.0 example is being done by Google. Google is turning the one-way-medium called “the news” into a two-way conversation by introducing a way to give the subjects of news reports a way to comment on articles written about them. What was a one-way pipe is now a two-way conversation. It chips away at the Fourth Estate’s overwhelming power to set and control the agenda.

Finally, there is a little open secrets project I’m involved with. It’s called GrantsFire. [I don't have much to show you yet about GrantsFire – but you can look at the hGrant microformat standard, if you want. Microformats are a way of marking up web pages to make them machine readable. Find information here.]

Clearly, grants are no secret, just as maps of France are easy to come by. However, GrantsFire is about seeing and presenting that information in new ways. GrantsFire is an initiative to both establish a standard for publishing machine-readable grants information on the web, and about encouraging foundations to publish such information. Once up and running, with a critical mass of participants, that information can be aggregated by one (or more) sites. People will be able to run the data through a data vegamatic, slicing and dicing by topic, type of support, geographic focus, foundation, dates, and dollars. Who knows what mashups might result. Perhaps the next time I’m careening around France, Jane at my side, she’ll pipe up and say: “You have reached your destination. There’s a vineyard on your left, a gas station on your right, and this area has received over $4 million in private grants to encourage organic farming, improve educational test scores among children, grades K-12, and to finance microenterprise.”

Secrets are now open secrets. Clive Thompson, writing for WIRED in an article entitled “The See-Through CEO,” writes:

“The Internet has inverted the social physics of information. Companies used to assume that details about their internal workings were valuable precisely because they were secret. If you were cagey about your plans, you had the upper hand; if you kept your next big idea to yourself, people couldn’t steal it. Now, billion- dollar ideas come to CEOs who give them away; corporations that publicize their failings grow stronger. Power comes not from your Rolodex but from how many bloggers link to you – and everyone trembles before search engine rankings.”

It’s a new world. Expose yourself. Come drive with Jane and see the power of the open secrets effect.

{Photograph copyright: Pam Green, 2003}

I’m back, and now I’ve knocked “sharing the DAM thing” off my “to-do” list. Read on.

Gavin’s DAM System

Finally (finally!), my brain kicked into gear. I figured out an easy way to share the DAM thing. Moreover, I re-did the whole thing in WSS just to make sure it was totally portable and didn’t have any residual MOSS attributes, etc.

I knew it should be easy. I think I was just over thinking it. Finally, one day, I, not so gently, slapped myself in the head and said (to myself): “why not just put it up on one of those free file sharing/storage systems that have sprouted up all over the place in the last few years. Might just be a good chance to take a look at how they work anyway.” A couple of aspirin later (for the headache after the slapping) I did it.

Here’s how: I closed my eyes and picked one. Once I picked one, I then puttered around with it trying to figure out how to actually share something. Then I trashed it, and tried another. Trashed that one, and tried yet another, and then another, and … finally I found one that would do what I wanted. It’s a nice service and it seems to work. It’s called File123. It seems workable.

The downside — it has a time limit. Four weeks. We’ll work around that simple enough. As long as people want it, I’ll keep resetting the date and/or posting a new copy.

So, (tad da!) you can find the completed DAM WSS template here:

http://www.file123.com/Transfer.aspx?guid=33a3935c-a4e4-4df0-975b-f066fcfe34e7

(Link updated 10/29/2007 – the new link will expire on 11/26/2007)

(12/4/2007)

I got tired of updating that link, so here’s one that won’t expire, up on Microsoft’s new (beta) Skydrive. Click here.

As I said, that URL will work for four weeks (all these systems impose some sort of time limit – I guess so you don’t use them as some sort continual, free, FTP hosting service just like I am doing). In four weeks I’ll re-share the file and reset the date.

Now: Some information:

First, I re-wrote the whole thing for WSS. My original (and working) design is in MOSS. But, I discovered that while something designed for WSS works just fine in MOSS, the reverse is not necessarily true. So, I just redid it in WSS.

Second, I completed the “Graphics” section. The photograph section was fully formed, but the Graphics section had been left wanting. I also put in some sample graphics and pictures. Of course, feel free to delete.

Third, I’ve saved it as a WSS template, instead of a “backup.” This, I believe, makes it much more portable and flexible. Everything is still there (I chose the option of saving data with the template), but templates are designed as “re-usable,” so there should be no issue with broken links or the like. I also eliminated the “search” portions. Instead, it should fall right into whatever search is appropriate for your particular installation.

Fourth, here are some basic instructions

Step 1: Add the template to your MOSS/WSS installation:

• Choose: Site Actions
• From the Galleries list, choose: Site Templates
• From the List menu, choose Upload and then upload the DAM WSS STP file into your site gallery.

Step 2: Choose to create a new site. On the site information page, choose the template under the CUSTOM tab in the Template Selection named “DAM WSS Version.”

Finally — and I say this seriously — download and have fun. However, Caveat Emptor and all that jazz. I offer no guarantees. I distribute this free of all responsibility and liability, use and/or abuse at your own risk. It should give you no problems, but if it does, well… I’ve left town, yeah, and am now living on the island of Fernando Poo, or maybe the long lost Isles of Langerhans… Yeah, that’s it.

We’re now confronted regularly with the requirement of proving our humanity before we’re allowed to comment on a blog posting, or sign up for a Yahoo! account, or send a happy note to Congress. I’ve written about that before — the Congress stuff — probably alienated a few friends in the process. (If you’re interested in my take on the Congressional move to institute so-called “logic puzzles,” look here and here.) Now that I think of it, it might be useful to have members of Congress prove that they’re human. I wonder how many would pass.

I admit it: there are days where I feel I might not pass such test. Moreover, in the grand scheme of things, I also admit there is a part of me that finds it altogether strange that we actually need such a test.

Nevertheless, tests to see if you’re “human” are now part and parcel of daily life on the internet — and appear as such things as CAPTCHA or other forms of logic puzzles. They’re all tricks, barriers to keep the ‘bots from flooding blogs, discussion groups, message boards, and web sites with auto-generated crap about drugs or body parts, or both. I’m always surprised, but apparently Christian Singles want to meet me! Obviously, they don’t know me (or my wife).

Metaphysical and political ponderings aside, tests that make you prove you’re human, officially called “Turing Tests,” are now downright commonplace.

The original “Turing Test” was developed by Alan Turing in the 1950′s. It was based on a parlor game in which a man and a woman would go into separate rooms and, remotely, using only the written word, try to convince other players in the game that they were the other person. Turing took that parlor game and tweaked it a bit, and came up with a test of a machine’s ability to demonstrate intelligence. A classic Turing Test asks whether: “a person could reliably tell the difference between a human in one room, and a computer in another, based solely on their written responses.” If (or when), a computer can fool that that human judge, computers will have passed the test, and have to be considered intelligent. [I'm here to say, right now, I want equal standing. If I can fool a judge into thinking I'm human, I want to be considered intelligent too.]

Today’s anti-spam and anti-bot tests are called “Reverse Turing Tests.” Unlike the classic test in which a human tries to tell the difference between another human and computer, a Reverse Turing Test has a computer judging if you’re a human or if you’re just another spam-generating, Viagra-pushing spam-bot.

Of such tests, CAPTCHA is probably the best known. It’s the one that asks you to use your uniquely human ability to recognize letters and numbers, even when they’re skewed, warped or wiggly. That’s something computers just don’t do well. CAPTCHA, by the way, stands for “Completely Automated Public Turing test to tell Computers and Humans Apart.” It’s trademarked by Carnegie Mellon. Here’s a sample of a CAPTCH – actually a reCAPTCHA. Let me tell you what the “re” in reCAPTCHA is all about, and why it’s so damn neat. reCAPTCHA takes the idea behind CAPTCHA and makes it work for a living.

An Example of a reCAPTCHA Turing Test

According to the reCAPTCHA web site, about 60 million CAPTCHAs are solved every day. In each case, some human takes about ten seconds to do what only humans can — recognize some funky text and translate it back into letters and numbers. Ten seconds. Not that much time, and with good effect. It stops the spambots and Christian Singles.

But, when you think about it, though, in the aggregate, those seconds add up. Supposedly they add up to over 150,000 hours a day — 150,000 hours spent closely interpreting a funky image of some text, and then entering that into a form. With this in mind, the reCAPTCHA people apparently had a bright idea: Why not use that 150,000 hours for some positive good.

reCAPTCHA does exactly that. It channels that human effort into helping verify and interpret things that scanners can’t figure out. reCAPTCHA uses this massively parallel human processing system — all those people using their human abilities to recognize badly formed letters — to correct and/or verify OCR mistakes as books are being scanned into the Internet Archive. Damn neat. A silk purse from a sow’s ear.

For all of you that immediately thought: Well, what good is that? If the system doesn’t know what the word is, how can you use it for a Turing Test? I can hear you looking for the comment button… Wait… Hold your horses, the answer here is as brilliant as it is simple. First, take a look at the sample graphic above. What do you see? Two words, not one, right?

reCAPTCHA uses word pairs, one known word, and one unknown word. Only one of the words is from the book scanning process. The reCAPTCHA system pairs up the words, and asks you to translate both. In this way, using known/unknown word pairs, reCAPTCHA allows you to both prove you’re human and extracts free labor, getting you to transcribe and verify some mystery word from a book being fed into the Internet Archive.

Even neater, they use the same mystery word ten or more times, with different people, checking and double-checking the word. Only then is the most common, and most likely correct, answer fed back into the Internet Archive. This, of course, overcomes that pesky thing called human error. After all, humans… well, they’re not to be trusted. Besides, how do we really know you’re really human, anyway…?

The trouble was even though it had a setting for “current user” buried deep in some of the “filter” options; it returned the information in the form “Domain\LOGON_USER” or the common name (Firstname Lastname). I never could figure out how to easily extract just the portions I needed and to use that portion in any of the off-the-shelf web parts or data views. As I said, perhaps it was just ignorance.

Instead, I ended up going the long way round and built out some special code to make a connectable web part that could be used to filter by username. It was a lot of work. And, of course, it broke in the transition from SPS2003 to MOSS 2007. Connectable web parts are great — but this was just too much work.

Consequently, in this version, I vowed to do something simpler. I knew it could be done; it was just a question of beating my head against it until I either:

A) beat myself senseless and forgot what I was doing, or

B) found the keys to the kingdom.

I suppose there was always option “C) Hire a consultant” but my experience with consultants, with a few notable exceptions, has not always been that satisfactory. Consultants seem to like to borrow your watch to tell you what time it is. The good ones, of course, then keep the watch.

The goal: simple, a stand-alone, self-filtering web part that displays data based upon the current user. If I did it right, the code should be reusable, or at least easy to duplicate.

I’m happy to report that I succeeded. Moreover, once I figured out where to do it, it was easy — and it opened up a whole passel of possibilities. (And I vowed to write it down so I wouldn’t forget it. I’m keeping my promise.)

About DVWPs:

Before I talk about “the how,” let me digress a minute to talk about “data views” or “data view web parts.” They’re the other half to this puzzle, and they’re real special, magical even. They are one of the more powerful SharePoint tools. Affectionately called DVWP’s, these little beasts do magic. And they’re right cute too. Like love, and like all magic, it’s kind of hard to explain unless you see it; even then, you’re not too sure how it works.

Please note: To play with DVWPs you need SharePoint Designer (MOSS 2007) or Visual Studio. The cost of admission is worth it. I use SharePoint Designer, but beware it is buggy as hell and seems to crash all the time. You get used to it, and learn not to do some things in some sequences. Such is life or Microsoft or both.

DVWP’s display data from databases such as SQL Server (or any ODBC-compliant data source) on a SharePoint page. In a nutshell, DVWPs give you a table of data, like a data grid, or a form. It’s that easy, by the way, to set up a web-front end to a database, for both viewing and editing.

If you’re looking to build some sort of dashboard, aggregating existing data, and displaying it dynamically on a web page, well, here is one easy way to do it.

Once set up, you can display data; you can shape it, mold it, and filter it. With the flick of a mouse, you can put it inside of handy dandy expanding and contracting headers, or sort it, or change the format or the display, based upon any of the values in the data itself. You can hide things or expose things, or change their color and format, based upon the values in the table. It’s just damn neat — and easy.

Here’s a simple sample (below), a DVWP showing “open” helpdesk support tickets dynamically pulled from our helpdesk system called TrackIT. Total Time to create: 2 minutes. I’ve redacted names to avoid tormenting the paranoid.

I warn you now, that while creating and tweaking DVWPs is easy as pie, the tough part is figuring out where to stop. All of a sudden, all that data that’s locked up in LOB databases can be easily displayed on a web page — without code, without anything other than (some) knowledge of the data structure and a connection to the SQL server.

As I mentioned, you need SharePoint Designer. From there, it’s easy. Just a few menu clicks, and some mousing around. I’m not going to cover them here, but, trust me, they’re phenomenally easy. Next stop, filtering. Buckle your seatbelts. The management is not responsible for headaches, eye strain, or slaps to the side of your head.

Before we go any further — and before you tell me that there are already ways to filter in MOSS — let me explain a bit more. You’re correct: built in to MOSS is a parameter called [Current User]. Unfortunately, for my purposes, it doesn’t work. It returns the current user in the form “First name Last name” (e.g., John Smith). We don’t use that. (D’oh) Where it’s appropriate, we tag data using the so-called USERNAME, in the form “JSMITH” (first initial concatenated with last name). It’s ubiquitous to all our databases. I know, because I made it so. Standards! I’m just a jeepster for standards. [You have to remember T-Rex for that to make any sense at all, not that it makes any sense even if you know the reference.]

So, I needed to work some magic — pretty simple magic if you have access to any sort of programming language — but I wanted to do this without opening the hood. How it’s done turns out to be simple, it’s done with XSLT, and I show you the code, and tell you where to put it. [I don't mean that the way it sounds.]

Personalizing a DVWP:

I’m going to use a real-life example here. I’m going to walk you thru creating a web part that displays data from our Grants Management system — specifically information about employee charitable giving, and the organization’s matching contribution. Our goal is a simple, stand-alone, portable web part that would display data filtered by the current user, in this case, me. What we want is a list of charitable gifts, sorted and grouped by year, showing the amount, the match, and the name of the organization.

What we’re not going to cover how to set up DVWPs or connections to a database. That’s pretty self explanitory.

The data is in the database, all we have to do it get it out, format it on the screen, and filter it by the Windows Server variable “LOGON_USER.” In the end, it should look like this: Again, I’ve redacted so you don’t see my gifts to folks like the Rosicrucians, the L5 Society, and the Center for Epistemological Inconsistencies and Eccentricities (I’m a charter member!).  Obviously, I’m an equal opportunity push-over.

Pretty simple really. You might note, I got a little fancy. I use some simple DVWP tricks put up a little red “new” flag if the gift date is less than a month old.

Here is how it’s done:

• Step 1:     Set up a connection to the database using the DVWP wizard.
• Step 2:    Select the table or view you want, and select the appropriate fields.
• Step 3:    Insert the data into an appropriate web part zone, and move the items around, until you get them the way you want them.
• Step 4:    Set up the groupings and headers and all that jazz.
• Step 4:    Set up the filter and sit back and admire your work.

By the way, while you’re working with a DVWP, I recommend you adjust a couple of settings right away. To get to the settings (and where you work the magic) click the little “menu” arrow thingy on the right side of the DVWP. Here’s a before and after shot, showing the task pane.

I set the various defaults as follows:

1. Click the box labled “Show with sample data” — this will populate the fields with sample data. This is important, especially if you use a filter that has no current results.
2. Set the the Data View Preview to hide all filters. Otherwise, if you filter out all the results, you have no current results.
3. The “Sort and Group” option lets you pick sort fields, and select to group on a certain field of set of fields. This is also where you set the expanding/contracting header groupings.
4. Notice the menu item labeled “Parameters.” This is where we set up the dynamic variable that pulls in the LOGON_USER.
5. Finally, we’re going to tweak the first parameter, the one labled “Filter.” This is where the magic happens.

Now we’re ready for the filter.

First we need to set up a “parameter” — click the menu item on the “Common Tasks” menu (above) labeled “Parameters.” You should see the following window:

We’re going to set up a parameter called “LogonUser” with the following attributes:

• Name:     LogonUser
• Parameter Source:    Server Variable
• Server Variable name:    Logon_User

When you’re done, it should look like the screen above. What this gets us is a dynamic parameter we can use elsewhere in the DVWP. It contains the full Windows Logon_User string, in the form “Domain\Logon_User.”

Next we’re going to turn it into something we can use by stripping off the domain and the slash (\) and converting it all to upper case. We do this right in the filter statement.

As shown above, from the “Common Data View Tasks” list, click the “Filter” menu option. It’s the first one on the list. You should see the “Filter Criteria” dialog box. It’s probably blank, and looks much like this:

Note the little check box at the bottom of the window. This is where we add the filter magic. Check the box, and then click the button to edit the XSLT filtering. Ignore the birght yellow warning that says: ”Using XSLT Filtering can reduce performance.” This is just a typical “If-you-do-this-your-naughty-bits-might-just-fall-off” warning. I just ignored it. Nothing fell off yet.

Finally we get to the filter – shown below in all its glory! Yep, that’s it. That whole line of gobbledygook. I’ll explain what it does, working from the inside out. (And quit your squinting – I put the final formula below so you can just copy and paste it into your DVWP.)

Constructing the Filter (from the inside out):

In the examples below, I show variables and formula in blue, and the current value of that statement in reddish brown. We start first in the center of the gobbledygook, with the $LogonUser parameter.

$LogonUser = Domain\JSmith

In the center of the expression is the parameter we created, called LogonUser, only now it is prefaced by a “$,” indicating that it is a parameter. This evaluates as the string “Domain\JSmith”

(Substring-After(“Domain\JSmith”,”\”) = JSmith

Surrounding the LogonUser parameter is a function to extract a substring from the parameter, using the “\” as the starting place in the string. The function extracts all the text following the “\”. We now have a parameter with just the name, e.g., JSmith.

Translate(JSmith’),’abcdefghijklmnopqrstuvwxyz’,'ABCDEFGHIJKLMNOPQRSTUVWXYZ’) = JSMITH

Now that we have the logon_user name handy, we have to convert it to upper case. XSLT is really clumsy here. Instead of a simple “UCase” statement or some such, we have to use a “Translate” function to convert every occurrence of a lower case letter to an upper case letter. Luckily I only need to do this once, and now I’ll just cut and paste it.

@Staff_ID = JSMITH

Finally, we can use an “Equals” operator to compare the variable @Staff_ID (pulled from the data source, to the LogonUser parameter we’ve constructed (or deconstructed, as the case may be).

The full statement is shown below: (note I’ve broken the line into two for display purposes). The brackets and everything else is important.

[@Staff_ID = Translate(Substring-After($LogonUser,'\'), 'abcdefghijklmnopqrstuvwxyz','ABCDEFGHIJKLMNOPQRSTUVWXYZ')]

That’s it. If you run Gifts, for example, and use the same format for STAFFID, you should be able to take this stuff verbatim and drop it into any part you set up.