“it could be bunnies”

One my coworkers had and issue where she couldn’t run cfreport from her site. The error from the site was rather unhelpfull:

ROOT CAUSE:
java.lang.NoClassDefFoundError: Could not initialize class coldfusion.runtime.report.Report
	at coldfusion.tagext.lang.ReportTag.doEndTag(ReportTag.java:581)
	at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2662)
	at cfprintLabels2ecfm1064228503.runPage(D:\sites\********\www\common\widgets\printLabels.cfm:49)

Initially Google only came with one hit when searching for this, and it didn’t provide any solution. Since it was a simple report to generate Avery labels in pdf I recommeded a workaround: use a pdf form template and fill out the addresses  through cfpdfform. But today when I saw a WEB-INF/cfusion/tmpCache/CFFileServlet/_cf_report/ folder on my development system I suddenly realized the problem might be caused by a sandbox issue.  The site is on a shared server and sandboxes default to the site folder and all subfolders, but nothing outside that. And as it turns out ColdFusion tries to do something with WEB-INF/cfusion/tmpCache/CFFileServlet/_cf_report/ so the sandbox will need access to that folder for cfreport to work. Once we allowed access to the Sandbox to this folder, cfreport started working again.

In case you are interested, the script to add a directory to all sandboxes is here:

<!--- For Sandbox stuff use the platform specific directory delimiter --->
<cfset cftempdir = server.ColdFusion.rootDir & "\tmpCache\-" />
<cfset adminPassword = "" />
 
<cfset administrator = CreateObject('component', 'cfide.adminapi.administrator')>
<cfset security = CreateObject('component', 'cfide.adminapi.security') />
 
<!--- Login --->
<cfif NOT administrator.login(adminPassword)>
	<cfthrow message="Login failed" detail="Unable to login to the CF adminAPI.">
</cfif>
 
<!--- Collect the defined sandboxes--->
<cfset sandBoxes = security.getSecuritySandboxes() />
<cfloop collection="#sandBoxes#" item="box">
 
	<!--- Test if it is a user-sanbox and not a system-sandbox and add the tmpCache folder --->
	<cfif Left(box, 12) IS "z:\websites\">
		<cfset security.setSecuredFolder(
				directory=box
				, folder=cftempdir
				) />
	</cfif>
 
</cfloop>

Other tags that won’t work in a sandbox

This issue appears to be new in CF 8 because aparently it had worked before in a similar sandbox on CF 7. Several more tags have problems if you typically limit sandbox access to the folder the code is in and to subfolders. My current list is

So who has discovered any other tags that need access to special directories to work properly?

Over the past months we have nearly completed a consolidation from two datacenter locations to one datacenter location. We had moved our dedicated servers to one location a while ago and two weeks ago we finally moved the last shared hosting to new hardware. And part of that last batch was our own stuff: experiments, mockups, test-sites etc. And while I was cleaning up on one of the old servers I noticed something peculiar about the webserver configuration for one of the sites. Because I would have sworn that site ran CF 7, but the IIS configuration looked like this:

If you look carefully you see that the .dbm extension is actually mapped to CF 5 and the .cfm, .cfml etc. extensions are mapped to CF MX. And if you look really carefully you will notice that the wildcard filter is mapped to wsconfig instance 2. And instance 2 had the port number in jrun.ini changed causing it to connect to CF 7 even though the connector was located in the CF MX installation directory. And since the wildcard filter runs first the site did run under CF 7.

If you use a server like this one where we hosted internal stuff and didn’t have strict change control for a long time I think it is pretty much inevitable that you end up with this sort of configuration quirks. (But that still doesn’t make it right.) But this sort of thing I have noticed before even on serious production servers from third parties (although usually with only 2 versions of CF instead of 3). And this sort of thing is why I always prefer to do upgrades through a full uninstall of the old version and then a fresh installation of the new version. It just helps keeping things clean and it really isn’t that much work. Nowadays I pretty much always work with CF multiserver and then it is just a matter of generating a new EAR file from the CF installer and deleting the old EAR file, but even when I am working with the single server edition it is just a matter of scripting the install through the silent installer and the admin API.

In the ColdFusion Administrator you can configure the logging behaviour to some extend. Where the ColdFusion logfiles are written, whether to log all email messages, long running pages etc. But you can not change the columns or masks for date formatting there, which is unfortunate because the native date mask for the logging is rather ambiguous. On my Dutch locale server the date “04/07/08″ should mean July 4th, but the log entry was written April 7th.

Fortunately, though not exposed in the ColdFusion Administrator, we can change the columns and patterns of the logfile directly in neo-logging.xml. By default, it defines the following pattern for the logging:

<var name="pattern">
    <string>"%p","%t",%d{"MM/dd/yy","HH:mm:ss"},%a,"%m%x"%n</string>
</var>

If we change the date mask in this pattern to something unambiguous, like the format defined in ISO 8601, we can now process our logfiles using tools written for our own locale:

<var name="pattern">
    <string>"%p","%t",%d{"yyyy-MM-dd","HH:mm:ss"},%a,"%m%x"%n</string>
</var>

Obviously you can change even more in the logging if you want, but if you start swapping / removing columns you may discover that the Logfile Reader that is part of the Adobe Extensions for Eclipse won’t work anymore. And like any change to a configuration file, make sure you have a backup before you start and make sure you restart the server once you have made the changes to detect any errors immediately instead of when the server won’t start at 3 AM.

Those who deploy ColdFusion on top of JRun may have noticed that the date mask in the JRun logfiles is even worse. Luckily here too we can configure the mask in a configuration file, jrun.xml:

<attribute name="format">{date yyyy-MM-dd HH:mm:ss} {log.message}{log.exception}</attribute>

The same warnings apply as with the ColdFusion log files.

Last week one of our big projects went online with a release of CF 8.01 final. I had personally started using CF 8 for this project in development when CF 8 was still in alfa, and while we started building EAR files against CF 8 almost a year ago, we kept using CF 7 in test / QA / production until about 5 months ago when we got the green light to move to CF 8. This move went ahead in two phases: first make the code work with CF 8 while maintaining compatibility with CF 7 and second start reaping the benefits of CF 8.

Phase one: making it work

Since I had been developing using CF 8 already we didn’t expect we needed any changes at all. So it was a bit of a disappointment when we started getting bugreports from the testing in QA. As it turned out, there were several changes in cfdocument and the client kept finding issues with alignment and the overflow of text being just a few pixels off in PDF files.  Issues that had been present in my development environemnt for well over a year, but that I had never noticed. I guess I really am more of a content person then a layout person

We had to go through several iterations to make things work, but the big break came when we got to build against CF 8.01 instead of CF 8. Our PDF alignment problems just disappeared. And we had a release that we could deploy on both CF7 and CF 8.

Phase two: reaping the benefits

When we had a converted all the infrastructure we could start to reap the benefits. My biggest priority was getting rid of the asynccfml gateway and replacing it with cfthread. Partially because it made the code simpler and more maintaineble, but increasingly because we experienced unexplained hung requests. I am still not sure what concurrency phenomenon exactly caused them and I am sure there were many other factors (ranging from database deadlocks to the dreadfull CreateUUID() performance), but I just wanted to get rid of them and for some reson cfthread did that.

The second one was replacing all occurences of CreateUUID() with something faster. ColdFusion uses UUIDs based on MAC addresses with very strong uniqueness guarantees, but atrocious performance. We had something that suited out needs better, but we needed at least Java 5 for it. And while CF 7.02 appeared to work with Java 5 in a test environment (or at least the parts we needed to work), we had to wait until it was supported before we could put it in production.

The third thing on my list is something I just realized when I was thinking about this post. It was not a deliberate priority but just evolved when writing code and it comes as somewhat of a surprise to me. What it is you may ask? The combination of creating structures inline in one line and using an attributecollection:

	<cfset variables.atts = {
		server = application.settings["SMTPserver"]
		, port=application.settings["SMTPport"]
		, username=application.settings["SMTPusername"]
		, password=application.settings["SMTPpassword"]
		} />
 
	<cffunction name="getSendCandidateEmail" access="private" returntype="void">
		<cfargument name="to" required="true" type="string" />
		<cfargument name="subject" required="true" type="string" />
		<cfargument name="body" required="true" type="string" />
		<cfargument name="BCC" required="false" type="boolean" default="false" />
 
		<cfset var localAtts = Duplicate(variables.atts) />
		<cfset localAtts.to = variables.to />
		<cfset localAtts.subject = variables.subject />
		<cfif areguments.bcc />
			<cfset localAtts.bcc = variables.bccAddress />
		</cfif>
 
		<cfmail attributeCollection="#localAtts#" />#arguments.body#</cfmail>
	</cffunction>

If you had asked me a year ago, neither inline structures nor the attributeCollection would have made my top 10 of favourite new features in CF 8.

Somebody was inquiring on a mailinglist why he would have a lot of errors in his *-event.log that looked like:

06/26 14:47:27 error Cannot create cookie: domain = .xxx.yyy

Amidst a number a theories a link to a blog post No Cookie for You!! surfaced as an explanation. An explanation that couldn’t possibly be true though (for reasons I will go into later this post). So now I was interested and I set out to get to the bottom of this problem on the commute home.

Cookies

In HTTP cookies are based on 2 headers send between the server and the browser. First, when the server wants to send a cookie, it uses a Set-Cookie header:

Set-Cookie: foo=bar; path=/; domain=.xxx.yyy; expires Mon, 09-Dec-2002 13:46:00 GMT

The Set-Cookie has a name-value pair with the name of the cookie and the value, and a bunch of attributes that allow the browser to determine whether or not to send the cookie back to the server on a subsequent request. And when a  browser wants to send a cookie back to a server, it sends a HTTP Cookie header:

Cookie: foo=bar

So what is going on

There are 2 problems with the eplanation provided in No Cookie for You!!

1. the cs(Cookie) column in the webserver log does not describe the Set-Cookie header send from the server to the browser, but the Cookie header send from the browser to the server (always remember when reading HTTP logs: cs means client-to-server, sc means server-to-client). So instead of a (failed) attempt to set a cookie, it is actually is proof  that the browser sends a cookie back;
2. not allowing a cookie to be set is meaningless in terms of the HTTP protocol. HTTP is a request-response protocol: the browser does a request, the server sends a response and that is it, there is no more communication. So even if the browser doesn’t like the cookie, it has no way to communicate that back to the server.

With this in mind I set out to reproduce the problem. First I constructed my own set of HTTP headers to send to the webserver:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: name=r2; expires=Tue, 02-Jul-2007 02:02:02 GMT;

Sending this set of headers to the webserver over telnet indeed produced an error in the *-event.log:

07/03 18:25:56 error Cannot create cookie: expires = Tue, 02-Jul-2007 02:02:02 GMT

So I set out to make a minimal reproduction. Expecting that the problem was somehow caused by improper escaping of the colons in the timestamp I tried to progressively reduce the number and complexity of the cookies until I ended up with:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: expires=T;

This still produced an error. Then I tried:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: e=T;

And I didn’t get an error anymore. And that is when I realized what was going on: the problem isn’t in the value or the escaping of the value, but in the name. Expires is treated as a reserved word because it is an attribute of a Set-Cookie header. Duh!!!

Reserved names

To complete the circle I ran a test with the cfccokie tag:

<cfcookie name="expires" value="test" />

And there I got the pretty grey and blue error message:

Cookie name EXPIRES is a reserved token

Further tests confirm that the following are all reserved names for cookies:

• secure
• domain
• expires
• path

And whenever CF receives a request with a cookie with one of those names, you get a line in your *-event.log with an error. I wonder if we can get CF to ignore that error without polluting the logfile.

So last weekend I came back from a two week trip to the US. I attended the Adobe Community Summit in San Jose and the Webmaniacs conference in Washington DC. As the name suggests the Community Summit is organized by Adobe for the Community. Usergroup Managers and Community Experts are invited to have a look at some cool demo’s and talk to people from different product teams. Obviously I can’t get too deep into what was discussed at that part of the Summit. After that is was 2 days of training, either CS3 or AIR. I went for the AIR training which was delivered by Leo Schuman. I had met Leo before last Februari when we were working on the materials for the TechU training program but I had never had pleasure of a two day class with him. The training itself had good materials, but was a little too much a beginners training for me since I had already worked with quite a few of the topics presented. The delivery though was excellent and made it a very enjoyable experience.

Then from San Jose it was on to Webmaniacs in Washington. I presented two sessions there for CFmaniacs. The first session was on Monday morning on J2EE clustering (slides) and it didn’t go as planned. After writing in the session abstract that clustering was easy I got challenged to demonstrate that so I set out to run a live demo in between my slides. The demo went off track when creating the cluster. That is not really unexpected, that part of setting up the cluster is temperamental, but you just delete the cluster, restart your instances and it will usually work. Then it went completely wrong when tying the cluster to IIS. As far as I have been able to reconstruct after the session there were still some leftovers from the trial run the previous night and some .dll’s were locked by IIS or something. Anyhow, after a restart of my laptop it went off without a hitch.
Then on Thuesday I had a session on N-tier ColdFusion (slides). That went much better, probably because I didn’t do any live demo. After that I was free to enjoy the rest of the conference. I didn’t follow too many sessions though because it was my first time in Washington and I took some time off to go sightseeing.

All in all it was a nice trip. The only weird thing is that I keep running into fellow Europeans in the US that I never meet in Europe.

I’m at the Webmaniacs conference just quickly writing this before I go and present my own session. Ben Forta just opened the conference with his keynote and he had 2 slides on Adobe’s plans for ColdFusion “Centaur” and I think they are the first public plans. I am not going to comment on them, just a verbatim copy.

CF Open Process Initiative

• Public bug database
• Public enhancement request system
• Custom Advisory Board
  • CFML design
  • feature definition
  • specification reviews
  • early release review

Looking Forward

• Work has already begun on “Centaur” (next major release of ColdFusion)
  • - Improved integration story
    • – Especially Flex and AIR
  • Improving the developer experience
• More details to be released at MAX 2008

As we all know it is a best practice to minimize the privileges of any application or account to reduce the attack surface. ColdFusion is no exception to this rule and should really be run under a dedicated user account with minimal permissions. And while on some systems such as my laptop, I am too lazy to follow that best practice, I do my best on other systems. The basic instructions for running ColdFusion under a specific user account are in Technote 17279. One of the things that bugs me about this technote is that it says ColdFusion needs Full Control permissions on the following filesystem directories to run:

• WebDocument Directory
• c:\cfusion or c:\cfusionmx (and all subdirectories)
• c:\winnt
• c:\winnt\system32

These filesystem permissions are really much more than I like them to be. As a rule I never give anything Full Control or even Change on the Windows install folder. In fact, I have been running ColdFusion servers since version 4.5 with just Read/Execute and Add on the Windows install folder (the good old NT4 days where we had Add permissions instead of FILE_APPEND_DATA and FILE_WRITE_DATA). And currently I run ColdFusion 8 multiserver JRun instances with much more restrictive permissions as well, using the following recipe:

1. Remove all permissions of the Users group on the \JRun4\servers\ folder.
2. Create a new Local account in the Local Users group (or add the Domain Account to the Local Users group).
3. Using the JRun console, create a new server instance. (Do not start the instance to test.)
4. On the filesystem, give the user you created in step 1 Change over the \JRun4\servers\<instance> directory.
5. Create a Windows Service for the new instance using jrunsvc.exe. (Do not start the service to test.)
6. Switch the new instance to the user you created in step 1 and start it. (If you started it before several files will have the wrong owner and the service will not start.)
7. Drop in a ColdFusion EAR file to replace the DEFAULT-EAR and you have a new, low privilege ColdFusion instance ready to use through the built in webserver. If you want to use ColdFusion with another webserver you need to give the ColdFusion user account permissions on the webroot and the webserver should be given Change on \JRun4\lib\wsconfig\.
8. If you want to use ColdFusion to read/write to other directories on the file system you obviously need to make sure it has the right permissions.

This recipe depends on the default permissions of a Windows installation. If you have tightened those and run into problems with your ColdFusion install, check the permissions on the \JRun4\ folder and the \windows\ folder. They should have Read, Execute (and Add for pre-MX versions of CF to accommodate temp file creation) for the user you added in step 1. Obviously this does not take into account the ODBC bridge, the Verity Services or the .NET Service.

Hidden below the “Show Advanced Settings” button on the datasource settings page for MS SQL Server lies the “String format” setting. According to the manual this setting is used to “Enable this option if your application uses Unicode data in DBMS-specific Unicode datatypes, such as National Character or nchar.” In other words, this allows you to use N fields in your database without adding those pesky N’s to your SQL and without butchering your data in the character set conversion process.

What that doesn’t tell you is that this setting can have serious performance consequences. When you enable this setting all variables send to the database using cfqueryparam are sent in their N format. Even the ones where the corresponding database field is not an N field but a regular field. And when that happens you have an encoding mismatch between the parameter and the column. Not really a problem when it is part of an insert statement, but disastrous when it is part of a predicate. Because an encoding mismatch will often prevent MS SQL Server from using indexes to execute the query, the query plan you nicely tuned from the SQL Server Management Studio (without unicode) will degrade into table scans.

Same encoding:
Index Seek(OBJECT:([playground].[dbo].[testCase].[testCase_reg]), SEEK:([playground].[dbo].[testCase].[regular]=[@1]) ORDERED FORWARD)

Encoding mismatch:
Table Scan(OBJECT:([playground].[dbo].[testCase]), WHERE:(CONVERT_IMPLICIT(nvarchar(36),[playground].[dbo].[testCase].[regular],0)=[@P1]))

In a simple testcase with only 256K short records this already accounts for a speed degradation from 1 to 100 milliseconds. I suspect that in real scenario’s the performance impact may very well be magnitudes larger because wide columns will make your I/O explode and there are all sorts of concurrency penalties if you switch from index access to full table scans.

The good news is that it is pretty easy to identify whether you are having this problem. Just profile your SQL Server with the option to trace the event SHOWPLAN ALL and in all your execution plans search for the string CONVERT_IMPLICIT. The bad news is that this is hard to fix. Switching “String format” off means your data gets garbled, switching it on means your server dies from an overload, so pretty much the only solution is to switch your entire database to N fields. That makes you wonder why other databases can do without this setting. For PostgreSQL I know the answer: charsets are a database wide setting so this situation can’t happen, you just switch your entire database to UTF-8 (and since it is UTF-8 and not UCS-16 like in MS SQL Server that doesn’t have the I/O overhead). But how about MySQL, Oracle etc., are they smart enough to do the conversion once on the side of that parameter instead of for every row on the side of the query or is it something else entirely?

Download the MS SQL Server String format testcase

I had a client with a pretty weird issue with client certificates last week and I thought I would share and maybe save somebody some time.

The client needed to consume a SOAP webservice that was protected with 2-way SSL, i.e. with both a server and a client certificate. Because of the new support for client certificates they had stepped away from the usual createObject() or cfinvoke code and moved to cfhttp, but they were getting connection errors on the ColdFusion side and HTTP 400 statuscodes in the logfiles of the webservice. That statuscode mean “Bad Request” and indeed the logfile did not show the correct resource path to the webservice but an empty path.

The first step towards solving the problem was to import the certificate chain of the server into ColdFusion. Steven Erat has an excellent article about this on his blog and sure enough the landscape changed. Instead of the HTTP 400 error the message in the cfhttp result and the webservice logfiles now was a HTTP 403 (Authentication Failed), but it did show the correct resource path. So obviously the next problem lay with the client certificates.

The client certificate appeared to be a normal client certificate with private key in PKCS#12 format, but from a different certificate chain then the server format. So I first had the client double-check that the chain was correctly installed on the server. Unfortunately it wasn’t that easy. I will spare you the details of what I tried, but in the end it turned out to be that the certificate was incorrectly encoded. These are the actual conversion steps I took to convert the certificate to a working state:

• import the certificate in the Windows Certificate store through the MMC;
• export the certificate with private key, whole chain and without strong encryption in PFX formet;
• convert the PFX encoded certificate to PEM using OpenSSL:

  openssl pkcs12 -in raw.pfx -out intermediate.pem -nodes

• re-order the certificates inside the PEM file to the following order:
  1. Identity certificate
  2. Intermediate certificate
  3. Root certificate
  4. Private Key
• convert the PEM encoded certificate to PKCS#12 using OpenSSL:

  openssl pkcs12 -export -out final.pkcs -in final.pem

It is probably possible to skip / merge a few of these steps (you probably just need to convert to PEM, reorder and convert back), but I really didn’t have time to explore that, I just know that this worked for me