“it could be bunnies”

Adobe just released Security Bulletin APSB08-21 for a ColdFusion privilege escalation issue. The issue described in Security Bulletin APSB08-21 is only applicable to your ColdFusion installation if you are using Sandbox Security. If you have configured a Sandbox to limit access to specific parts of the filesystem it may be possible to access information outside the Sandbox. This issue is particularly important for shared hosting servers because they are the most likely to have Sandbox Security enabled.
There are patches available for ColdFusion 7.0.2, 8.0.0 and 8.0.1. At Prisma IT we have been testing the patch for ColdFusion 8.0.1 for a while now and we have not found any side effects from applying it to our shared hosting servers.

If you are a ColdFusion user and this blog post is the first you read about this issue you really should subscribe to the Adobe Security Notification Service. You will get emails for all the important security updates from Adobe and it is an invaluable tool to staying on top of security.

By default ColdFusion will use the computer name without the domain name appended when sending email. However, some mail servers require that senders use a Fully Qualified Domain Name (FQDN) in their EHLO. If that is the case, you may get errors in your mail.log that look something like this:

Sep 18 17:22:11 mail postfix/smtpd[55543]: NOQUEUE: reject: RCPT from
prlt004[145.94.255.255]: 504 5.5.2 mail3.prisma-it.com: Helo command
rejected: need fully-qualified hostname; from=<email-address>
to=<email-address> proto=ESMTP helo=mail3.prisma-it.com

A similar problem exists with Message-IDs and spam filters and Adobe has TechNote kb400753 ColdFusion MX: Configuring cfmail to use a Fully Qualified Domain Name which describes a solution to fix the Message-ID. Luckily a similar solution works for changing the FQDN used in the EHLO. Just add a mail.smtp.localhost to your jvm.config with the right FQDN as value, restart your ColdFusion instance and you can send email again. My jvm.config uses:

-Dmail.host=jochem.vandieten.net -Dmail.smtp.localhost=jochem.vandieten.net

One my coworkers had and issue where she couldn’t run cfreport from her site. The error from the site was rather unhelpfull:

ROOT CAUSE:
java.lang.NoClassDefFoundError: Could not initialize class coldfusion.runtime.report.Report
	at coldfusion.tagext.lang.ReportTag.doEndTag(ReportTag.java:581)
	at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2662)
	at cfprintLabels2ecfm1064228503.runPage(D:\sites\********\www\common\widgets\printLabels.cfm:49)

Initially Google only came with one hit when searching for this, and it didn’t provide any solution. Since it was a simple report to generate Avery labels in pdf I recommeded a workaround: use a pdf form template and fill out the addresses  through cfpdfform. But today when I saw a WEB-INF/cfusion/tmpCache/CFFileServlet/_cf_report/ folder on my development system I suddenly realized the problem might be caused by a sandbox issue.  The site is on a shared server and sandboxes default to the site folder and all subfolders, but nothing outside that. And as it turns out ColdFusion tries to do something with WEB-INF/cfusion/tmpCache/CFFileServlet/_cf_report/ so the sandbox will need access to that folder for cfreport to work. Once we allowed access to the Sandbox to this folder, cfreport started working again.

In case you are interested, the script to add a directory to all sandboxes is here:

<!--- For Sandbox stuff use the platform specific directory delimiter --->
<cfset cftempdir = server.ColdFusion.rootDir & "\tmpCache\-" />
<cfset adminPassword = "" />
 
<cfset administrator = CreateObject('component', 'cfide.adminapi.administrator')>
<cfset security = CreateObject('component', 'cfide.adminapi.security') />
 
<!--- Login --->
<cfif NOT administrator.login(adminPassword)>
	<cfthrow message="Login failed" detail="Unable to login to the CF adminAPI.">
</cfif>
 
<!--- Collect the defined sandboxes--->
<cfset sandBoxes = security.getSecuritySandboxes() />
<cfloop collection="#sandBoxes#" item="box">
 
	<!--- Test if it is a user-sanbox and not a system-sandbox and add the tmpCache folder --->
	<cfif Left(box, 12) IS "z:\websites\">
		<cfset security.setSecuredFolder(
				directory=box
				, folder=cftempdir
				) />
	</cfif>
 
</cfloop>

Other tags that won’t work in a sandbox

This issue appears to be new in CF 8 because aparently it had worked before in a similar sandbox on CF 7. Several more tags have problems if you typically limit sandbox access to the folder the code is in and to subfolders. My current list is

So who has discovered any other tags that need access to special directories to work properly?

Over the past months we have nearly completed a consolidation from two datacenter locations to one datacenter location. We had moved our dedicated servers to one location a while ago and two weeks ago we finally moved the last shared hosting to new hardware. And part of that last batch was our own stuff: experiments, mockups, test-sites etc. And while I was cleaning up on one of the old servers I noticed something peculiar about the webserver configuration for one of the sites. Because I would have sworn that site ran CF 7, but the IIS configuration looked like this:

If you look carefully you see that the .dbm extension is actually mapped to CF 5 and the .cfm, .cfml etc. extensions are mapped to CF MX. And if you look really carefully you will notice that the wildcard filter is mapped to wsconfig instance 2. And instance 2 had the port number in jrun.ini changed causing it to connect to CF 7 even though the connector was located in the CF MX installation directory. And since the wildcard filter runs first the site did run under CF 7.

If you use a server like this one where we hosted internal stuff and didn’t have strict change control for a long time I think it is pretty much inevitable that you end up with this sort of configuration quirks. (But that still doesn’t make it right.) But this sort of thing I have noticed before even on serious production servers from third parties (although usually with only 2 versions of CF instead of 3). And this sort of thing is why I always prefer to do upgrades through a full uninstall of the old version and then a fresh installation of the new version. It just helps keeping things clean and it really isn’t that much work. Nowadays I pretty much always work with CF multiserver and then it is just a matter of generating a new EAR file from the CF installer and deleting the old EAR file, but even when I am working with the single server edition it is just a matter of scripting the install through the silent installer and the admin API.

In the ColdFusion Administrator you can configure the logging behaviour to some extend. Where the ColdFusion logfiles are written, whether to log all email messages, long running pages etc. But you can not change the columns or masks for date formatting there, which is unfortunate because the native date mask for the logging is rather ambiguous. On my Dutch locale server the date “04/07/08″ should mean July 4th, but the log entry was written April 7th.

Fortunately, though not exposed in the ColdFusion Administrator, we can change the columns and patterns of the logfile directly in neo-logging.xml. By default, it defines the following pattern for the logging:

<var name="pattern">
    <string>"%p","%t",%d{"MM/dd/yy","HH:mm:ss"},%a,"%m%x"%n</string>
</var>

If we change the date mask in this pattern to something unambiguous, like the format defined in ISO 8601, we can now process our logfiles using tools written for our own locale:

<var name="pattern">
    <string>"%p","%t",%d{"yyyy-MM-dd","HH:mm:ss"},%a,"%m%x"%n</string>
</var>

Obviously you can change even more in the logging if you want, but if you start swapping / removing columns you may discover that the Logfile Reader that is part of the Adobe Extensions for Eclipse won’t work anymore. And like any change to a configuration file, make sure you have a backup before you start and make sure you restart the server once you have made the changes to detect any errors immediately instead of when the server won’t start at 3 AM.

Those who deploy ColdFusion on top of JRun may have noticed that the date mask in the JRun logfiles is even worse. Luckily here too we can configure the mask in a configuration file, jrun.xml:

<attribute name="format">{date yyyy-MM-dd HH:mm:ss} {log.message}{log.exception}</attribute>

The same warnings apply as with the ColdFusion log files.

Last week one of our big projects went online with a release of CF 8.01 final. I had personally started using CF 8 for this project in development when CF 8 was still in alfa, and while we started building EAR files against CF 8 almost a year ago, we kept using CF 7 in test / QA / production until about 5 months ago when we got the green light to move to CF 8. This move went ahead in two phases: first make the code work with CF 8 while maintaining compatibility with CF 7 and second start reaping the benefits of CF 8.

Phase one: making it work

Since I had been developing using CF 8 already we didn’t expect we needed any changes at all. So it was a bit of a disappointment when we started getting bugreports from the testing in QA. As it turned out, there were several changes in cfdocument and the client kept finding issues with alignment and the overflow of text being just a few pixels off in PDF files.  Issues that had been present in my development environemnt for well over a year, but that I had never noticed. I guess I really am more of a content person then a layout person

We had to go through several iterations to make things work, but the big break came when we got to build against CF 8.01 instead of CF 8. Our PDF alignment problems just disappeared. And we had a release that we could deploy on both CF7 and CF 8.

Phase two: reaping the benefits

When we had a converted all the infrastructure we could start to reap the benefits. My biggest priority was getting rid of the asynccfml gateway and replacing it with cfthread. Partially because it made the code simpler and more maintaineble, but increasingly because we experienced unexplained hung requests. I am still not sure what concurrency phenomenon exactly caused them and I am sure there were many other factors (ranging from database deadlocks to the dreadfull CreateUUID() performance), but I just wanted to get rid of them and for some reson cfthread did that.

The second one was replacing all occurences of CreateUUID() with something faster. ColdFusion uses UUIDs based on MAC addresses with very strong uniqueness guarantees, but atrocious performance. We had something that suited out needs better, but we needed at least Java 5 for it. And while CF 7.02 appeared to work with Java 5 in a test environment (or at least the parts we needed to work), we had to wait until it was supported before we could put it in production.

The third thing on my list is something I just realized when I was thinking about this post. It was not a deliberate priority but just evolved when writing code and it comes as somewhat of a surprise to me. What it is you may ask? The combination of creating structures inline in one line and using an attributecollection:

	<cfset variables.atts = {
		server = application.settings["SMTPserver"]
		, port=application.settings["SMTPport"]
		, username=application.settings["SMTPusername"]
		, password=application.settings["SMTPpassword"]
		} />
 
	<cffunction name="getSendCandidateEmail" access="private" returntype="void">
		<cfargument name="to" required="true" type="string" />
		<cfargument name="subject" required="true" type="string" />
		<cfargument name="body" required="true" type="string" />
		<cfargument name="BCC" required="false" type="boolean" default="false" />
 
		<cfset var localAtts = Duplicate(variables.atts) />
		<cfset localAtts.to = variables.to />
		<cfset localAtts.subject = variables.subject />
		<cfif areguments.bcc />
			<cfset localAtts.bcc = variables.bccAddress />
		</cfif>
 
		<cfmail attributeCollection="#localAtts#" />#arguments.body#</cfmail>
	</cffunction>

If you had asked me a year ago, neither inline structures nor the attributeCollection would have made my top 10 of favourite new features in CF 8.

Somebody was inquiring on a mailinglist why he would have a lot of errors in his *-event.log that looked like:

06/26 14:47:27 error Cannot create cookie: domain = .xxx.yyy

Amidst a number a theories a link to a blog post No Cookie for You!! surfaced as an explanation. An explanation that couldn’t possibly be true though (for reasons I will go into later this post). So now I was interested and I set out to get to the bottom of this problem on the commute home.

Cookies

In HTTP cookies are based on 2 headers send between the server and the browser. First, when the server wants to send a cookie, it uses a Set-Cookie header:

Set-Cookie: foo=bar; path=/; domain=.xxx.yyy; expires Mon, 09-Dec-2002 13:46:00 GMT

The Set-Cookie has a name-value pair with the name of the cookie and the value, and a bunch of attributes that allow the browser to determine whether or not to send the cookie back to the server on a subsequent request. And when a  browser wants to send a cookie back to a server, it sends a HTTP Cookie header:

Cookie: foo=bar

So what is going on

There are 2 problems with the eplanation provided in No Cookie for You!!

1. the cs(Cookie) column in the webserver log does not describe the Set-Cookie header send from the server to the browser, but the Cookie header send from the browser to the server (always remember when reading HTTP logs: cs means client-to-server, sc means server-to-client). So instead of a (failed) attempt to set a cookie, it is actually is proof  that the browser sends a cookie back;
2. not allowing a cookie to be set is meaningless in terms of the HTTP protocol. HTTP is a request-response protocol: the browser does a request, the server sends a response and that is it, there is no more communication. So even if the browser doesn’t like the cookie, it has no way to communicate that back to the server.

With this in mind I set out to reproduce the problem. First I constructed my own set of HTTP headers to send to the webserver:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: name=r2; expires=Tue, 02-Jul-2007 02:02:02 GMT;

Sending this set of headers to the webserver over telnet indeed produced an error in the *-event.log:

07/03 18:25:56 error Cannot create cookie: expires = Tue, 02-Jul-2007 02:02:02 GMT

So I set out to make a minimal reproduction. Expecting that the problem was somehow caused by improper escaping of the colons in the timestamp I tried to progressively reduce the number and complexity of the cookies until I ended up with:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: expires=T;

This still produced an error. Then I tried:

GET /index.cfm HTTP/1.1
Host: localhost
Cookie: e=T;

And I didn’t get an error anymore. And that is when I realized what was going on: the problem isn’t in the value or the escaping of the value, but in the name. Expires is treated as a reserved word because it is an attribute of a Set-Cookie header. Duh!!!

Reserved names

To complete the circle I ran a test with the cfccokie tag:

<cfcookie name="expires" value="test" />

And there I got the pretty grey and blue error message:

Cookie name EXPIRES is a reserved token

Further tests confirm that the following are all reserved names for cookies:

• secure
• domain
• expires
• path

And whenever CF receives a request with a cookie with one of those names, you get a line in your *-event.log with an error. I wonder if we can get CF to ignore that error without polluting the logfile.

So last weekend I came back from a two week trip to the US. I attended the Adobe Community Summit in San Jose and the Webmaniacs conference in Washington DC. As the name suggests the Community Summit is organized by Adobe for the Community. Usergroup Managers and Community Experts are invited to have a look at some cool demo’s and talk to people from different product teams. Obviously I can’t get too deep into what was discussed at that part of the Summit. After that is was 2 days of training, either CS3 or AIR. I went for the AIR training which was delivered by Leo Schuman. I had met Leo before last Februari when we were working on the materials for the TechU training program but I had never had pleasure of a two day class with him. The training itself had good materials, but was a little too much a beginners training for me since I had already worked with quite a few of the topics presented. The delivery though was excellent and made it a very enjoyable experience.

Then from San Jose it was on to Webmaniacs in Washington. I presented two sessions there for CFmaniacs. The first session was on Monday morning on J2EE clustering (slides) and it didn’t go as planned. After writing in the session abstract that clustering was easy I got challenged to demonstrate that so I set out to run a live demo in between my slides. The demo went off track when creating the cluster. That is not really unexpected, that part of setting up the cluster is temperamental, but you just delete the cluster, restart your instances and it will usually work. Then it went completely wrong when tying the cluster to IIS. As far as I have been able to reconstruct after the session there were still some leftovers from the trial run the previous night and some .dll’s were locked by IIS or something. Anyhow, after a restart of my laptop it went off without a hitch.
Then on Thuesday I had a session on N-tier ColdFusion (slides). That went much better, probably because I didn’t do any live demo. After that I was free to enjoy the rest of the conference. I didn’t follow too many sessions though because it was my first time in Washington and I took some time off to go sightseeing.

All in all it was a nice trip. The only weird thing is that I keep running into fellow Europeans in the US that I never meet in Europe.

I’m at the Webmaniacs conference just quickly writing this before I go and present my own session. Ben Forta just opened the conference with his keynote and he had 2 slides on Adobe’s plans for ColdFusion “Centaur” and I think they are the first public plans. I am not going to comment on them, just a verbatim copy.

CF Open Process Initiative

• Public bug database
• Public enhancement request system
• Custom Advisory Board
  • CFML design
  • feature definition
  • specification reviews
  • early release review

Looking Forward

• Work has already begun on “Centaur” (next major release of ColdFusion)
  • - Improved integration story
    • – Especially Flex and AIR
  • Improving the developer experience
• More details to be released at MAX 2008

As we all know it is a best practice to minimize the privileges of any application or account to reduce the attack surface. ColdFusion is no exception to this rule and should really be run under a dedicated user account with minimal permissions. And while on some systems such as my laptop, I am too lazy to follow that best practice, I do my best on other systems. The basic instructions for running ColdFusion under a specific user account are in Technote 17279. One of the things that bugs me about this technote is that it says ColdFusion needs Full Control permissions on the following filesystem directories to run:

• WebDocument Directory
• c:\cfusion or c:\cfusionmx (and all subdirectories)
• c:\winnt
• c:\winnt\system32

These filesystem permissions are really much more than I like them to be. As a rule I never give anything Full Control or even Change on the Windows install folder. In fact, I have been running ColdFusion servers since version 4.5 with just Read/Execute and Add on the Windows install folder (the good old NT4 days where we had Add permissions instead of FILE_APPEND_DATA and FILE_WRITE_DATA). And currently I run ColdFusion 8 multiserver JRun instances with much more restrictive permissions as well, using the following recipe:

1. Remove all permissions of the Users group on the \JRun4\servers\ folder.
2. Create a new Local account in the Local Users group (or add the Domain Account to the Local Users group).
3. Using the JRun console, create a new server instance. (Do not start the instance to test.)
4. On the filesystem, give the user you created in step 1 Change over the \JRun4\servers\<instance> directory.
5. Create a Windows Service for the new instance using jrunsvc.exe. (Do not start the service to test.)
6. Switch the new instance to the user you created in step 1 and start it. (If you started it before several files will have the wrong owner and the service will not start.)
7. Drop in a ColdFusion EAR file to replace the DEFAULT-EAR and you have a new, low privilege ColdFusion instance ready to use through the built in webserver. If you want to use ColdFusion with another webserver you need to give the ColdFusion user account permissions on the webroot and the webserver should be given Change on \JRun4\lib\wsconfig\.
8. If you want to use ColdFusion to read/write to other directories on the file system you obviously need to make sure it has the right permissions.

This recipe depends on the default permissions of a Windows installation. If you have tightened those and run into problems with your ColdFusion install, check the permissions on the \JRun4\ folder and the \windows\ folder. They should have Read, Execute (and Add for pre-MX versions of CF to accommodate temp file creation) for the user you added in step 1. Obviously this does not take into account the ODBC bridge, the Verity Services or the .NET Service.