“it could be bunnies”

Moving from CF 7.02 to CF 8.01

Last week one of our big projects went online with a release of CF 8.01 final. I had personally started using CF 8 for this project in development when CF 8 was still in alfa, and while we started building EAR files against CF 8 almost a year ago, we kept using CF 7 in test / QA / production until about 5 months ago when we got the green light to move to CF 8. This move went ahead in two phases: first make the code work with CF 8 while maintaining compatibility with CF 7 and second start reaping the benefits of CF 8.

Phase one: making it work

Since I had been developing using CF 8 already we didn’t expect we needed any changes at all. So it was a bit of a disappointment when we started getting bugreports from the testing in QA. As it turned out, there were several changes in cfdocument and the client kept finding issues with alignment and the overflow of text being just a few pixels off in PDF files.  Issues that had been present in my development environemnt for well over a year, but that I had never noticed. I guess I really am more of a content person then a layout person

We had to go through several iterations to make things work, but the big break came when we got to build against CF 8.01 instead of CF 8. Our PDF alignment problems just disappeared. And we had a release that we could deploy on both CF7 and CF 8.

Phase two: reaping the benefits

When we had a converted all the infrastructure we could start to reap the benefits. My biggest priority was getting rid of the asynccfml gateway and replacing it with cfthread. Partially because it made the code simpler and more maintaineble, but increasingly because we experienced unexplained hung requests. I am still not sure what concurrency phenomenon exactly caused them and I am sure there were many other factors (ranging from database deadlocks to the dreadfull CreateUUID() performance), but I just wanted to get rid of them and for some reson cfthread did that.

The second one was replacing all occurences of CreateUUID() with something faster. ColdFusion uses UUIDs based on MAC addresses with very strong uniqueness guarantees, but atrocious performance. We had something that suited out needs better, but we needed at least Java 5 for it. And while CF 7.02 appeared to work with Java 5 in a test environment (or at least the parts we needed to work), we had to wait until it was supported before we could put it in production.

The third thing on my list is something I just realized when I was thinking about this post. It was not a deliberate priority but just evolved when writing code and it comes as somewhat of a surprise to me. What it is you may ask? The combination of creating structures inline in one line and using an attributecollection:

	<cfset variables.atts = {
		server = application.settings["SMTPserver"]
		, port=application.settings["SMTPport"]
		, username=application.settings["SMTPusername"]
		, password=application.settings["SMTPpassword"]
		} />

	<cffunction name="getSendCandidateEmail" access="private" returntype="void">
		<cfargument name="to" required="true" type="string" />
		<cfargument name="subject" required="true" type="string" />
		<cfargument name="body" required="true" type="string" />
		<cfargument name="BCC" required="false" type="boolean" default="false" />

		<cfset var localAtts = Duplicate(variables.atts) />
		<cfset localAtts.to = variables.to />
		<cfset localAtts.subject = variables.subject />
		<cfif areguments.bcc />
			<cfset localAtts.bcc = variables.bccAddress />
		</cfif>

		<cfmail attributeCollection="#localAtts#" />#arguments.body#</cfmail>
	</cffunction>

If you had asked me a year ago, neither inline structures nor the attributeCollection would have made my top 10 of favourite new features in CF 8.

Posted in ColdFusion | 2 Comments »

Reserved names for cookies

Somebody was inquiring on a mailinglist why he would have a lot of errors in his *-event.log that looked like:

06/26 14:47:27 error Cannot create cookie: domain = .xxx.yyy

Amidst a number a theories a link to a blog post No Cookie for You!! surfaced as an explanation. An explanation that couldn’t possibly be true though (for reasons I will go into later this post). So now I was interested and I set out to get to the bottom of this problem on the commute home.

Cookies

In HTTP cookies are based on 2 headers send between the server and the browser. First, when the server wants to send a cookie, it uses a Set-Cookie header:

Set-Cookie: foo=bar; path=/; domain=.xxx.yyy; expires Mon, 09-Dec-2002 13:46:00 GMT

The Set-Cookie has a name-value pair with the name of the cookie and the value, and a bunch of attributes that allow the browser to determine whether or not to send the cookie back to the server on a subsequent request. And when a  browser wants to send a cookie back to a server, it sends a HTTP Cookie header:

Cookie: foo=bar

So what is going on

There are 2 problems with the eplanation provided in No Cookie for You!!

1. the cs(Cookie) column in the webserver log does not describe the Set-Cookie header send from the server to the browser, but the Cookie header send from the browser to the server (always remember when reading HTTP logs: cs means client-to-server, sc means server-to-client). So instead of a (failed) attempt to set a cookie, it is actually is proof  that the browser sends a cookie back;
2. not allowing a cookie to be set is meaningless in terms of the HTTP protocol. HTTP is a request-response protocol: the browser does a request, the server sends a response and that is it, there is no more communication. So even if the browser doesn’t like the cookie, it has no way to communicate that back to the server.

With this in mind I set out to reproduce the problem. First I constructed my own set of HTTP headers to send to the webserver:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: name=r2; expires=Tue, 02-Jul-2007 02:02:02 GMT;

Sending this set of headers to the webserver over telnet indeed produced an error in the *-event.log:

07/03 18:25:56 error Cannot create cookie: expires = Tue, 02-Jul-2007 02:02:02 GMT

So I set out to make a minimal reproduction. Expecting that the problem was somehow caused by improper escaping of the colons in the timestamp I tried to progressively reduce the number and complexity of the cookies until I ended up with:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: expires=T;

This still produced an error. Then I tried:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: e=T;

And I didn’t get an error anymore. And that is when I realized what was going on: the problem isn’t in the value or the escaping of the value, but in the name. Expires is treated as a reserved word because it is an attribute of a Set-Cookie header. Duh!!!

Reserved names

To complete the circle I ran a test with the cfccokie tag:

<cfcookie name="expires" value="test" />

And there I got the pretty grey and blue error message:

Cookie name EXPIRES is a reserved token

Further tests confirm that the following are all reserved names for cookies:

• secure
• domain
• expires
• path

And whenever CF receives a request with a cookie with one of those names, you get a line in your *-event.log with an error. I wonder if we can get CF to ignore that error without polluting the logfile.

Posted in ColdFusion | 6 Comments »

Community Summit and Webmaniacs trip

So last weekend I came back from a two week trip to the US. I attended the Adobe Community Summit in San Jose and the Webmaniacs conference in Washington DC. As the name suggests the Community Summit is organized by Adobe for the Community. Usergroup Managers and Community Experts are invited to have a look at some cool demo’s and talk to people from different product teams. Obviously I can’t get too deep into what was discussed at that part of the Summit. After that is was 2 days of training, either CS3 or AIR. I went for the AIR training which was delivered by Leo Schuman. I had met Leo before last Februari when we were working on the materials for the TechU training program but I had never had pleasure of a two day class with him. The training itself had good materials, but was a little too much a beginners training for me since I had already worked with quite a few of the topics presented. The delivery though was excellent and made it a very enjoyable experience.

Then from San Jose it was on to Webmaniacs in Washington. I presented two sessions there for CFmaniacs. The first session was on Monday morning on J2EE clustering (slides) and it didn’t go as planned. After writing in the session abstract that clustering was easy I got challenged to demonstrate that so I set out to run a live demo in between my slides. The demo went off track when creating the cluster. That is not really unexpected, that part of setting up the cluster is temperamental, but you just delete the cluster, restart your instances and it will usually work. Then it went completely wrong when tying the cluster to IIS. As far as I have been able to reconstruct after the session there were still some leftovers from the trial run the previous night and some .dll’s were locked by IIS or something. Anyhow, after a restart of my laptop it went off without a hitch.
Then on Thuesday I had a session on N-tier ColdFusion (slides). That went much better, probably because I didn’t do any live demo. After that I was free to enjoy the rest of the conference. I didn’t follow too many sessions though because it was my first time in Washington and I took some time off to go sightseeing.

All in all it was a nice trip. The only weird thing is that I keep running into fellow Europeans in the US that I never meet in Europe.

Posted in ColdFusion, Flex | 1 Comment »

Adobe “Centaur” plans

I’m at the Webmaniacs conference just quickly writing this before I go and present my own session. Ben Forta just opened the conference with his keynote and he had 2 slides on Adobe’s plans for ColdFusion “Centaur” and I think they are the first public plans. I am not going to comment on them, just a verbatim copy.

CF Open Process Initiative

• Public bug database
• Public enhancement request system
• Custom Advisory Board
  • CFML design
  • feature definition
  • specification reviews
  • early release review

Looking Forward

• Work has already begun on “Centaur” (next major release of ColdFusion)
  • - Improved integration story
    • – Especially Flex and AIR
  • Improving the developer experience
• More details to be released at MAX 2008

Posted in ColdFusion | No Comments »

Windows file permissions for the ColdFusion account

As we all know it is a best practice to minimize the privileges of any application or account to reduce the attack surface. ColdFusion is no exception to this rule and should really be run under a dedicated user account with minimal permissions. And while on some systems such as my laptop, I am too lazy to follow that best practice, I do my best on other systems. The basic instructions for running ColdFusion under a specific user account are in Technote 17279. One of the things that bugs me about this technote is that it says ColdFusion needs Full Control permissions on the following filesystem directories to run:

• WebDocument Directory
• c:\cfusion or c:\cfusionmx (and all subdirectories)
• c:\winnt
• c:\winnt\system32

These filesystem permissions are really much more than I like them to be. As a rule I never give anything Full Control or even Change on the Windows install folder. In fact, I have been running ColdFusion servers since version 4.5 with just Read/Execute and Add on the Windows install folder (the good old NT4 days where we had Add permissions instead of FILE_APPEND_DATA and FILE_WRITE_DATA). And currently I run ColdFusion 8 multiserver JRun instances with much more restrictive permissions as well, using the following recipe:

1. Remove all permissions of the Users group on the \JRun4\servers\ folder.
2. Create a new Local account in the Local Users group (or add the Domain Account to the Local Users group).
3. Using the JRun console, create a new server instance. (Do not start the instance to test.)
4. On the filesystem, give the user you created in step 1 Change over the \JRun4\servers\<instance> directory.
5. Create a Windows Service for the new instance using jrunsvc.exe. (Do not start the service to test.)
6. Switch the new instance to the user you created in step 1 and start it. (If you started it before several files will have the wrong owner and the service will not start.)
7. Drop in a ColdFusion EAR file to replace the DEFAULT-EAR and you have a new, low privilege ColdFusion instance ready to use through the built in webserver. If you want to use ColdFusion with another webserver you need to give the ColdFusion user account permissions on the webroot and the webserver should be given Change on \JRun4\lib\wsconfig\.
8. If you want to use ColdFusion to read/write to other directories on the file system you obviously need to make sure it has the right permissions.

This recipe depends on the default permissions of a Windows installation. If you have tightened those and run into problems with your ColdFusion install, check the permissions on the \JRun4\ folder and the \windows\ folder. They should have Read, Execute (and Add for pre-MX versions of CF to accommodate temp file creation) for the user you added in step 1. Obviously this does not take into account the ODBC bridge, the Verity Services or the .NET Service.

Posted in ColdFusion | No Comments »

MS SQL Server and the ColdFusion “String format” setting

Hidden below the “Show Advanced Settings” button on the datasource settings page for MS SQL Server lies the “String format” setting. According to the manual this setting is used to “Enable this option if your application uses Unicode data in DBMS-specific Unicode datatypes, such as National Character or nchar.” In other words, this allows you to use N fields in your database without adding those pesky N’s to your SQL and without butchering your data in the character set conversion process.

What that doesn’t tell you is that this setting can have serious performance consequences. When you enable this setting all variables send to the database using cfqueryparam are sent in their N format. Even the ones where the corresponding database field is not an N field but a regular field. And when that happens you have an encoding mismatch between the parameter and the column. Not really a problem when it is part of an insert statement, but disastrous when it is part of a predicate. Because an encoding mismatch will often prevent MS SQL Server from using indexes to execute the query, the query plan you nicely tuned from the SQL Server Management Studio (without unicode) will degrade into table scans.

Same encoding:
Index Seek(OBJECT:([playground].[dbo].[testCase].[testCase_reg]), SEEK:([playground].[dbo].[testCase].[regular]=[@1]) ORDERED FORWARD)

Encoding mismatch:
Table Scan(OBJECT:([playground].[dbo].[testCase]), WHERE:(CONVERT_IMPLICIT(nvarchar(36),[playground].[dbo].[testCase].[regular],0)=[@P1]))

In a simple testcase with only 256K short records this already accounts for a speed degradation from 1 to 100 milliseconds. I suspect that in real scenario’s the performance impact may very well be magnitudes larger because wide columns will make your I/O explode and there are all sorts of concurrency penalties if you switch from index access to full table scans.

The good news is that it is pretty easy to identify whether you are having this problem. Just profile your SQL Server with the option to trace the event SHOWPLAN ALL and in all your execution plans search for the string CONVERT_IMPLICIT. The bad news is that this is hard to fix. Switching “String format” off means your data gets garbled, switching it on means your server dies from an overload, so pretty much the only solution is to switch your entire database to N fields. That makes you wonder why other databases can do without this setting. For PostgreSQL I know the answer: charsets are a database wide setting so this situation can’t happen, you just switch your entire database to UTF-8 (and since it is UTF-8 and not UCS-16 like in MS SQL Server that doesn’t have the I/O overhead). But how about MySQL, Oracle etc., are they smart enough to do the conversion once on the side of that parameter instead of for every row on the side of the query or is it something else entirely?

Download the MS SQL Server String format testcase

Posted in ColdFusion, databases | No Comments »

Browsershots: view your site in multiple browsers

It is hard to believe anybody has missed the 2 big browser announcements of today. First, Microsoft announced that “IE8 will, by default, interpret web content in the most standards compliant way it can“. Second, The Web Standards Project has released the Acid3 test.

In the torrent of messages about these 2 big announcements I found a link to a service that I hadn’t heard from before: browsershots. It allows you to select a number of browsers on different platforms and generates screenshots of the way they render. And of course somebody has submitted the Acid3 test to browsershots and we can now all see how the different browsers score.

Posted in Uncategorized | No Comments »

Localizing LiveCycle ES Workspace with FlexBuilder 3

With the recent release of FlexBuilder 3 I have been trying to work through some of the instructions for customizing the LiveCycle Workspace using FlexBuilder 3 instead of FlexBuilder 2 (if that works we don’t need to include FlexBuilder 2 in the software for the Adobe TechU). I didn’t expect too many problems because according to Customizing the LiveCycle® Workspace ES User Interface Flex SDK 2.0.1 is supported and you can pick that SDK version in FlexBuilder 3. And for the most part it did the work. The only problem I ran into was compiling the localization swf file. According to the instructions that compilation should work with the following command:

compc -locale=es -output=export/es/workspace_rb.swc -source-path
./es "C:/program files/adobe/flex builder 2/flex sdk 2/frameworks/locale/"
-include-resource-bundles alc_wks_client_msg alc_wks_client_trace
alc_wks_client_ui SharedResources collections controls core data effects
formatters logging messaging rpc skins states styles utils validators --

Now with FlexBuilder 3 this didn’t work because some of the paths to the SDK are different. With the correct paths the compile command becomes:

"C:\Program Files\Adobe\Flex Builder 3\sdks\2.0.1\bin\compc" -locale=es
  -output=spanish/workspace_rb.swc -source-path ./spanish
  "C:\Program Files\Adobe\Flex Builder 3\sdks\2.0.1\frameworks\locale"
  -include-resource-bundles alc_wks_client_msg alc_wks_client_trace
  alc_wks_client_ui SharedResources collections controls core data
  effects formatters logging messaging rpc skins states styles utils
  validators --

This didn’t compile in FlexBuilder 3 because the data properties file couldn’t be found. That is supposedly an easy fix: just copy the data.properties file from wherever FlexBuilder 2 got it to the project path in FlexBuilder 3. Only I couldn’t find a data.properties file anywhere in FlexBuilder 2. And while removing data from the compile command solved the compile time problem, it did result in a totally unacceptable runtime error. So in the end I just created my own data.properties file with the following data:

# Dummy data.properties file
foo=bar

With this file in the same directory as the localized files the compilation ran correctly and I haven’t received any runtime errors yet. It is quite likely I will get them at some point when the Workspace wants to present some message that is supposed to be in data.properties, but I think this will do until the next LiveCycle release that I expect to add support for FlexBuilder 3.

Posted in Flex, LiveCycle | 1 Comment »

CFHTTP and client certificates

I had a client with a pretty weird issue with client certificates last week and I thought I would share and maybe save somebody some time.

The client needed to consume a SOAP webservice that was protected with 2-way SSL, i.e. with both a server and a client certificate. Because of the new support for client certificates they had stepped away from the usual createObject() or cfinvoke code and moved to cfhttp, but they were getting connection errors on the ColdFusion side and HTTP 400 statuscodes in the logfiles of the webservice. That statuscode mean “Bad Request” and indeed the logfile did not show the correct resource path to the webservice but an empty path.

The first step towards solving the problem was to import the certificate chain of the server into ColdFusion. Steven Erat has an excellent article about this on his blog and sure enough the landscape changed. Instead of the HTTP 400 error the message in the cfhttp result and the webservice logfiles now was a HTTP 403 (Authentication Failed), but it did show the correct resource path. So obviously the next problem lay with the client certificates.

The client certificate appeared to be a normal client certificate with private key in PKCS#12 format, but from a different certificate chain then the server format. So I first had the client double-check that the chain was correctly installed on the server. Unfortunately it wasn’t that easy. I will spare you the details of what I tried, but in the end it turned out to be that the certificate was incorrectly encoded. These are the actual conversion steps I took to convert the certificate to a working state:

• import the certificate in the Windows Certificate store through the MMC;
• export the certificate with private key, whole chain and without strong encryption in PFX formet;
• convert the PFX encoded certificate to PEM using OpenSSL:

  openssl pkcs12 -in raw.pfx -out intermediate.pem -nodes

• re-order the certificates inside the PEM file to the following order:
  1. Identity certificate
  2. Intermediate certificate
  3. Root certificate
  4. Private Key
• convert the PEM encoded certificate to PKCS#12 using OpenSSL:

  openssl pkcs12 -export -out final.pkcs -in final.pem

It is probably possible to skip / merge a few of these steps (you probably just need to convert to PEM, reorder and convert back), but I really didn’t have time to explore that, I just know that this worked for me

Posted in ColdFusion | No Comments »

Nice overview of Database Architecture

Through the Planet PostgreQL site I found a nice paper that compares database internals. It provides an interesting read, even if you are not really into databases.

Architecture of a Database System
Joseph M. Hellerstein, Michael Stonebraker and James Hamilton

Posted in PostgreSQL, databases | No Comments »