“it could be bunnies”

Since I am in Bangalore for a training I dropped in on the “Adobe Flash Platform Tools Preview” this evening. The agenda promised short sessions on Flash Builder 4, Flash Catalyst and LiveCycle ES, followed by food and networking. The Flash Builder (previously known as Flex Builder) session was solid. It showed Data Centric Development, where services are defined in Flash Builder and can then easily be wired into the UI because all the code for the services and value objects is generated. (See Raghu’s blog for the demo screencast). Next up was Flash Catalyst, showing a design - development workflow where a .psd file was transformed to a .fxp, which was then imported in Flash Builder to wire the data in through the new services management. Last was LiveCycle ES. Unfortunately, but understandably considering the audience, this was all about data management and not process management. What was new for me was that apparently this now wires directly into Hibernate so you don’t need to write any server side code anymore, you can have everything generated.

The Q&A focused mainly on the designer - developer workflow with Flash catalyst. The main question that was repeated several times in different words was whether this workflow put any additional constraints on the designer. And each time the answer was that good development on the design side, including the judicious use of layers, was all it took. I think this reflected the audience of architects and project managers, from developers I would expect more technical oriented questions.

Afterward the food and networking were great. Not just the Flash team from Adobe was there, but also people from the LiveCycle team and the ColdFusion team., so I got an opportunity to thank some people in person for fixes and new features I am not allowed to mention yet. And I also met up with some of the people we do business with in India.

This Sunday at long last my first real server “Spike” died. Spike wasn’t really my server, but it was the first server that wasn’t just for my entertainment and I carried final responsibility for. Purchased in February 2001 it entered service as a shared hosting server for a not-for-profit in March 2001. In the 8 years and 1 month it ran the only real problem it had was a worn down CPU fan causing an overheated CPU, until Sunday morning at long last the primary harddisk died and it was retired from service.

With its demise I am truly saying goodbye to an era (or perhaps to a relic): Spike ran trusty old Windows NT4 SP6a with ColdFusion Enterprise 4.5.2. With it gone, the youngest production machine I have access to is a Windows 2003 system (7 years younger then NT4) with CF 7 (6 years younger then CF 4.5). Spike itself is replaced with a machine with Windows 2003 and CF 8.0.1. And the contrast between how it worked then and now it works now is quite profound. At least in the area of security configuration, CFML is sufficiently backward compatible to just drop it on the new server.

2001, the year Spike was configured, was just before the height of the ‘hackable internet’. A few months after it was taken into production we saw the release of Code Red, followed shortly by Nimda. At that time, a large part of the servers connected to internet was vulnerable to attacks. (Nowadays vulnerabilities are more of a client problem or arguably a user problem then s server problem.) And that showed itself in the way Spike was build. It took me several weeks to come up with a stable and secure configuration, with all sorts of weird constraints. To build Spike I followed the NSA guidelines for configuring a Windows NT system, which for instance meant I wasn’t supposed to install any graphical driver, because no driver was NSA certified. And the way I ended up running ColdFusion, with Sandbox Security configured to impersonate OS accounts, has once even earned me the comment from a Macromedia engineer to be the only one in the world with that configuration in production. But the result was there: even with the onslaught of Code Red and Nimda it took 7 months before there was a Windows patch that was applicable for the hardened configuration.

Contrast this with how I threw a new server online. Windows installation was a default installation, after which I had to add components instead of remove them. When installing IIS I had to add filetypes and extensions, instead of remove them. When configuring Sandbox Security for ColdFusion I could easily find anything I wanted on the subject, because there are dozens of people blogging about it. Obviously some of the ease of installing a new server is due to more experience on my side, but I think it is hard to deny that the “secure by default” mindset has made inroads.

As promised my hacks to get past the the quirks in the webservice API for the Adobe forums. I am deliberately not publishing the full application, the lack of local caching in it makes it more of a DOS tool then a forum client.

Authentication

The PermissionsService authenticate method doesn’t work since the Adobe forums do not use the standard Jive login methods, but a custom Adobe SSO login method. To get a login on the forums from AIR replay what a browser does when logging in to the forums. First visit the index page of the forums to GET a few cookies, then POST the credentials to the Adobe authentication server, then GET the index page of the forums again to allow the forums to do a call back to the Adobe authentication server to collect the user profile. So all in all it takes 3 HTTP requests to log on.
I have included the code snippet that my AIR app uses to log in to the forums below for those who wish to experiment with it. Be warned that the full sequence takes on average 15 seconds.

// General variables
private const _AdobeAuthURL:String = "http://www.adobe.com/cfusion/entitlement/index.cfm?loc=en&e=ca&returnurl=http%3A%2F%2Fforums%2Eadobe%2Ecom%2Flogin%2Ejspa";
private const _ForumRootURL:String = "http://forums.adobe.com/index.jspa";
 
private function startLoginSequence():void {
	writeLog("Starting login ");
	getForumRoot(preAuthResult);
}
private function getForumRoot(resultFn:Function):void {
	// just GET the forum root to collect cookies
	var forumRootService:HTTPService = new HTTPService();
	forumRootService.method = "GET";
	forumRootService.url = _ForumRootURL;
	forumRootService.useProxy = false;
	forumRootService.resultFormat = "text";
	forumRootService.addEventListener(FaultEvent.FAULT, faultHandler);
	forumRootService.addEventListener(ResultEvent.RESULT, resultFn);
	forumRootService.send();
}
private function preAuthResult(event:Event):void {
	// We have now collected the forum cookies, log in to the Adobe ID
	login();
}
private function login():void {
	// login does a login request to the main Adobe site
 
	// credentials Object with all name value pairs
	var credentials:Object = new Object();
	credentials['returnURL'] = 'http://forums.adobe.com%2Flogin.jspa';
	credentials['up_login'] = 'yes';
	credentials['ignore_email_validation'] = 'yes';
	credentials['up_username'] = "spam@vandieten.net";
	credentials['has_pwd'] = "true";
	credentials['up_password'] = "sihtyrt";
 
	// login Service
	var loginService:HTTPService = new HTTPService();
	loginService.method = "POST";
	loginService.url = _AdobeAuthURL;
	loginService.useProxy = false;
	loginService.resultFormat = "text";
	loginService.addEventListener(FaultEvent.FAULT, faultHandler);
	loginService.addEventListener(ResultEvent.RESULT, loginResult);
	loginService.send(credentials);
}
private function loginResult(event:ResultEvent):void {
	// check if we are really logged in
	var loggedInTestString:String = '<a href="http://www.adobe.com/cfusion/membership/logout.cfm">Sign out</a>';
	if (event.result.indexOf(loggedInTestString) != -1) {
		// Login success
		// Do a new HTTP request to the forums to propagate the login from Adobe to Jive
		getForumRoot(postAuthResult);
	} else {
		// Login failure
		throw("Username password combination incorrect.");
	}
}
private function postAuthResult(event:ResultEvent):void {
	// Fully logged in, ready to use the API
	writeLog("Login complete");
 
	// extractUserDetails(event.result);
}

Once we are logged in we can use all of the APIs to get the actual useful information from the forums. The FlexBuilder WSDL import tool works quite well with the APIs, all cases where it failed turned out to be mistakes in my programming. To get stared call getRecursiveCommunities of the CommunityService to get a list of all the communities (forums) available. Depending on how busy the forums are, internet bandwidth, traffic and the position of the moon this can take between 25 seconds and 5 minutes. From the list of forums you get you can drill down into the list of threads (ForumService getThreadsByCommunityID: up to 50 seconds to get the thread list of the DreamWeaver forum, the busiest forum with about 50K threads) and then into the list of messages per thread (ForumService getMessagesByThreadID: usually less then a second). When you get the messages you will also get the users and all things you need to display a tree of who responded to who.

Getting your own userID

Apart from the login methods in the API there appears to be another problem in the webservice API (or maybe just something I haven’t figured out). Searching for a user, either by his username or by his emailaddress, has never returned any results for me. (I do in fact get a 500 Internal server error when I try to use the UserService getUserByUsername.) But the user details are needed for all API methods to post messages. The workaround I implemented at last for that is to just do a string lookup in the HTML of the forum start page (the commented out call to extractUserDetails in the previous code snippet):

public function extractUserDetails(pageObj:Object):void {
	var page:String = pageObj as String;
	var userIDPreString:String = 'quickuserprofile.getUserProfileTooltip(';
	var userIDPreStringIndex:int = page.indexOf(userIDPreString) + 39;
	var userIDPostString:String = ')';
	var userIDPostStringIndex:int = page.indexOf(userIDPostString, userIDPreStringIndex);
	_userID = page.substring(userIDPreStringIndex, userIDPostStringIndex) as int;
}

Obviously depending on visiting some sites in a certain order to pick up cookies along the way and scraping generated HTML is rather fragile. The smallest change in the HTML or the authentication will break any application that uses this. Ideally Adobe would implement some authentication webservices on its own site to facilitate logging in from an application.

With this the basic services to list forums, retrieve threads, read messages and post your own messages can be accessed without further surprises. I intend to continue working on my ForumClient, but it will be a while. I will need a proper design for the application, I am thinking about modules to support different forum software API’s, storing configuration data in XML, message caching in a SQLite database per configured site etc. Then I need to develop the whole thing. And in the mean time real life is catching up and I am going on a training tour of Europe, so I will have little time for the next three weeks. Drop me a line if you are interested in helping out, or don’t wait for me and just get started on your own client

I managed to get a proof of concept AIR client for the Adobe forums running. It can log in, browse communities, threads, and messages and can even reply. But since it doesn’t do much with local storage yet it doesn’t remember which messages are already read. I’ll post some more on the hacks to log in to the forums soon, but for now I have screenshots.

ForumClient Screenshot

ForumClient screenshot

Just a quick update on the two avenues I am pursuing to get the new Adobe forums to open up and share their content.

Adobe is going to followup on the email headers with Jive to see what they can implement. Hopefully this will result in getting the References and / or In-Reply-To headers added to the outgoing email notifications soon so email clients can thread them properly. It should be almost trivial to implement that, because judging by the definition of the ForumMessage object each forum message knows its own ID to put in the Message-ID header and also knows its ParentMessageID to put in the References header.

The other avenue is getting access through the webservice APIs. I have been looking at the login sequence of the Adobe site and it looks like the master cookie to determine whether you are logged in is the AUID cookie for the .adobe.com domain. Subsites may add more cookies and eventually maintain state through their own cookies, but if you have the AUID cookie you can get in. So I have reduced the dozens of requests involved in a browser logging in to the forums (many of them image requests) to two requests a client must make. I have included the minimal wget testscript to simulate the login here.

REM parameters
SET USERNAME=spam@vandieten.net
SET PASSWORD=sihtyrt
SET RETURNURL=http%3A%2F%2Fforums.adobe.com%2Flogin.jspa
SET LOGINURL="https://www.adobe.com/cfusion/entitlement/index.cfm?loc=en&e=ca&returnurl=http%3A%2F%2Fforums%2Eadobe%2Ecom%2Flogin%2Ejspa"
SET FORUMURL=http://forums.adobe.com/index.jspa
 
REM remove old cookie and index
del adobe.txt
del forumindex.html
 
REM login
wget -S --no-check-certificate --save-cookies=adobe.txt --post-data "returnURL=%RETURNURL%&up_login=yes&ignore_email_validation=yes&up_username=%USERNAME%&has_pwd=true&up_password=%PASSWORD%" %LOGINURL%
REM get forum index
wget -S --load-cookies=adobe.txt -O forumindex.html "%FORUMURL%"

I need to tweak this a bit to do proper login validation, but it looks like just ignoring all the authentication mechanisms in the API and running these requests instead and then maintaining the collected cookies with all webservice calls will do the trick. Even though the entire login sequence will end up taking close to 15 seconds due to the response times of the Adobe sites, that is progress!

Two weeks ago Adobe informed us we could preview and test the new version of the online community forums. At first glance these forums offer an exciting new look with many new features. A single login system where we can use our Adobe ID to log on to the forums, RSS support, an improved search and room for attachments are great new features.

But when I started digging at what features the forums offered the excitement pretty soon turned into a feeling of unease. The old forums were an open system, accessible through NNTP, which provided two way interaction that allowed people to run their own interface and program against the forums through the NNTP ‘API’. Yet the new forums appeared to be a closed system following the old ‘ thou shalt watch me the way I want’ ways. So the questions that pretty soon bubbled up were not of excitement. Is this all? How am I going to work around this layout? How am I going to work with this workflow? And then the feeling of unease turned into a feeling of disappointment.
The new forums are state of the art. The web interface is so much better then the previous one. And yet they are a disappointment. Because they may be state of the art, but they do nothing new. Nothing we haven’t seen before. Nothing to inspire us. Adobe is merely catching up, not leading the way. And in the process Adobe is throwing away the one open interface I had to program against the forums.

You see, I am an avid NNTP user. When I start my client, it automatically goes out on the web to collect all new messages I follow. That is currently 27 newsgroups (on the Adobe, Macromedia, prerelease and other servers) with on average 40 message per day each. And when these messages are collected, they are sorted, filtered, grouped and prioritized for me. Messages get filtered out if I don’t like the sender or subject, people that post a direct response to me get priority, threads I have hidden before won’t resurface, etc. All of that takes maybe 20 seconds, and then a thousand messages have been reduced to two dozen priority messages from people who are waiting for me and have responded to me, a bunch of messages I may find interesting, and the rest that I don’t find interesting is gone. Now how is that going to work with these new forums? What is the closed system of these new forums offering that allow me to automate my workflow to the same level as NNTP offers? And I know that many other NNTP users struggle with the same questions.

Now I have done some research into possible solutions. And surprisingly enough the answer is really, really simple. It took me two hours to build a system that converted the email feed from the new forums to NNTP. All it takes is a subscription on all forums and a small script to convert email headers to NNTP headers. That means you have to replace the header “To: Joh Doe <JohnDoe@example.invalid>” with the header “Newsgroups: general.english_discussions”. And then I write that converted email to the spool folder of the Apache James NNTP server and I have an operational email to NNTP conversion. Sure, it doesn’t have avatars, author profiles, ‘mark as answer’ functionality, but as a NNTP user I really couldn’t care less about that. And the beauty of it is that this is two-way access because all I need to do is click “Reply to user” instead of “Reply to newsgroup” and the email will go to the Reply by email functionality for the forums.

Now to get the last bit of functionality implemented, I just need Adobe to fix their last bug. Email has all these hidden header fields that tell email clients how different messages relate to each other. And Adobe is filling them out incorrectly, so I can not sort messages correctly. So if Adobe could please fix the References and / or In-Reply-To headers in the outgoing email that would make me really happy. And if I am really happy, I might just open up my NNTP server for others.

And it gets better. Because the new forums could easily become inspiring. Because as it turns out, the new Adobe forums have a webservice API. And that API is enabled. And we can program against that API using Flex or more likely AIR (because of the local storage and to bypass any cross domain problems) and build their own interfaces for the forums and customize them. Write plugins for them, for instance to easily link to documentation or copy examples. Or people could write their own automation rules again, so they can sort, filter and prioritize messages the same way they can now through the NNTP ‘API’. I have a bunch of ideas for things I could do (which I probably won’t if Adobe fixes the email bugs), and I am sure there are many more people with even better ideas.
But for that to happen Adobe first needs to do something. The forums may have a webservice API right now, but it doesn’t have any documentation. Sure, there is some documentation from the vendor of the forum, but the implementation of the Adobe ID based login system of the new forums is different. And without a login no API access.

I really think Adobe has an opportunity to make these forums great. The number or forums that truly integrates web, email and NNTP is very limited. Adobe could be the first one to not only do that, but also allow unlimited AIR extensions through its webservices and empower the community to build its own tools. Or is Adobe, the company that is preaching the gospel of the Rich Internet Applications, really forcing all users to go back to a web interface?

Every year we all go on a trip together for a few days with everybody from Prisma IT. This year the trip is to Iceland. This is all of us having lunch at Schiphol airport before boarding.

Prisma IT having lunch at Schiphol airport

So for the next two days we will all be out of office. Well, not really all of us, Richard is staying in the Netherlands to answer the phone …

I am currently in the UK for a LiveCycle training and when I looked out of my hotel window the view was very nice.

Winter view from Fleet (UK)

Then just before I was going for breakfast my boss called and asked whether I still was in the hotel. Then he told me to “Go make snow dolls”. Apparently the snow is a bit more then the people in the UK are used to and is causing major travel issues. The students are from all over the country (apparently some had to drive 4 hours under normal conditions), and due to the weather the first day of the training is canceled. So now I have my first snow day ever! (In the Netherlands we only have ice days to go ice-skating, when it has frozen enough and the ice is thick enough, but no snow days.)

UPDATE: training cancelled for the whole week based on the weather forecast, but no way to go home since all flights to Amsterdam have been cancelled at Heathrow.

Today was the CF Insider Workshop in Brussels. Line-up was the same as last week in Amsterdam, Claude Englebert from Adobe and Simon Slooten and me from Prisma IT. Mark van Hedel was supposed to join us for the ColdFusion and AJAX session, but he broke his leg last week so he couldn’t be there and Simon took over his session. The event in Amsterdam was pretty well attended with 20+ attendees, in Brussels we had a few less (but they have another event in French tomorrow). But the good news is that after years of a slumber Adobe is finally talking about ColdFusion in Europe again. Some of the attendents were even Adobe employees from other departments who wanted to know what that ColdFusion thing was about.

I have uploaded my slides from the CF Insider workshop to the Prisma IT website:

What’s new in ColdFusion 8

The ColdFusion Server Monitor

Adobe and Prisma IT are organizing the Dutch and Flemish versions of the ColdFusion Insider Workshop that is touring Europe. First we will have the Dutch one on January 22th in the Adobe office in Amsterdam and then on Januari 27th the Flemish on in Brussels. There will be speakers from Adobe and Prisma IT. Attendance is free if you get an entrance voucher by filling out the registration form.

I will be doing a presentation on the ColdFusion Server Monitor and a more general one on how ColdFusion makes life easier with all the cool new features in ColdFusion 8. If you have a good suggestion I may be persuaded to add your topic to my “OMG, nobody has questions, how do I fill the awkward silence” emergency slides.