“it could be bunnies”

ColdFusion 8 includes the very nice server monitor that allows you to see what is happening in real time. The problem with the server monitor is that it runs in the same process as ColdFusion so when ColdFusion becomes unresponsive there is a chance that the server monitor is inaccessible as well. And of course the server monitor is Enterprise Edition only.

But some of the functionality in the server monitor is built on native functionality of the JVM and there are some tools in the JDK that can extract some of the same information from a running ColdFusion server. The most important one is jstack. Jstack is a program that can connect to any running Java program and create a stack dump of all running threads and some memory statistics. Jstack is not included with the JVM that Adobe uses for ColdFusion 8 so you have to get a full JDK from Sun to get a copy. (While you are at it, why not install JDK 6 update 10 or later that solves the class loading bug in earlier JDK 6 versions?) Once you have that you can simply getting a stack trace by running the command “jstack <pid>” where pid is the process identifier of the ColdFusion (JRun) process you want to investigate.

The complication is that you need to do this from the same useraccount as the account the process you want to analyze is running. On Linux that is simply a matter of su, but on Windows this is more complicated. If you are running ColdFusion in the default configuration where  it runs under the SYSTEM account you need to start another program as SYSTEM, and that is not as easy as it may seem because you can not use the runas command to switch to that account. Some time ago a solution for that was posted at the MSDN blogs that involved creating a temporary service and starting that. I have packaged that solution as the following batchfile:

sc create testsvc binpath= "cmd /K start" type= own type= interact
sc start testsvc
sc delete testsvc

When I run this on Windows I get a nice command prompt under the system account. If you want to run this over a remote desktop session, make sure you are connected to the console session (use “mstsc /admin /v:servername” to start your remote desktop session). From that command line I can now run jstack, jmap and all the other JDK tools to analyze a running Java program.

In the first part we have set the stage for this series: the goal is to protect one shared hosting customer from an ‘inside attack’ by another shared hosting customer on the same ColdFusion instance.

If no precautions have been taken, attacking is a simple job. The starting point is a template that does a directory listing of every drive from “a:\” to “z:\” using cfdirectory. When you find something interesting, like the “coldfusion8″ directory, the “jrun4″ directory or a “customers” directory, just drill down from there. Not elegant, but very effective. The only way to hide from it is to use netwerk shares with hard to guess names, but even then it is just a matter of drilling down from the #server.coldfusion.rootdir# to the logfiles and see what the directory paths in some of the mappings and logged errors are.
Hiding (i.e. obscurity) provides no effective hurdle against an inside job and hence no real security.

To get real security against this reading of directories we need to enable Sandbox Security. Sandbox Security allows us to define a directory on the filesystem as a Sandbox and subject every request that starts from that Sandbox to a set of constraints. These constraints can include which tags are allowed, i.e. forbid cfregistry outright, or which resources can be accessed. Typically each Sandbox is defined at the root of a customers FTP and / or WWW directory and then allows for access of only some directories and datasources. Setting up the allowed resources and tags in a Sandbox can occasionally be a bit counterintuitive, for instance to allow a file to be used in a cfinclude it needs execute permissions and several extra directories need to be accessible for some tags.

The thing with Sandbox Security is that it is a feature that is only available in Enterprse Edition. ColdFusion 8 Standard Edition has the ability to restrict the usage of tags, functions and resources as well, but everybody operates in the same Sandbox. So while we can disallow customer A from reading the files of customer B through cfdirectory, that would also disallow customer B from reading his own files. And disabling tags and functions won’t be foolproof either because there is always a way around that. For instance, even if you can’t read a certain file with cffile that doesn’t stop you from mailing it to yourself using cfmailparam.

So here we see the first issue with shared hosting and security: in order to combine them and get a system that is even remotely securable, the hoster needs to invest significantly in a ColdFusion 8 Enterprise Edition license and needs to figure out how to configure Sandbox Security properly. Obviously (if the hoster even decides to bother with all that in the first place) that expense gets charged to the hosted customers, making ColdFusion hosting more expensive then for instance PHP hosting.

A frequent cause for complaints among ColdFusion developers is that shared hosting for ColdFusion is hard to find, expensive and/or limited in functionality. I am not really affected by that since I don’t develop applications that are intended to be deployed in a shared hosting environment. But I do get to see the other side of the medal: at Prisma IT we offer some shared hosting (it is not an area we focus on) and I am responsible for the architecture of the platform. So in this and some followup articles I will dive into some of the complications of securing a shared hosting platform running Adobe ColdFusion 8 on Windows.

Setting the stage

These articles will specifically be about ‘inside jobs’: securing ColdFusion shared hosting customers from each other, while they run on the same ColdFusion instance. This is not about securing servers from outside threats, the starting point is some customer who can FTP his CFML templates to the server and wants to look at the templates and the database records of another customer on the same system.
For this discussion I will largely ignore the complications of combining ColdFusion with other scripting languages. While it is most certainly an interesting subject, the issues involved are too diverse for some general blogs.

Topics

The topics aren’t set in stone yet, but for now I am writing on the following blogs:

• Filesystem access and Sandbox Security
• Memory access
• ColdFusion internals
• Java, COM and .NET
• Microsoft Access

Feel free to suggest more / other content in your comments.

I’m sure that if you have never worked with the LiveCycle training material you have no idea what that title is about. During the LiveCycle Building Applications (LCBA) training a mortgage workflow for the fictional company Fin@nceCorp is build step by step. It starts with a PDF form to submit a mortgage application, then through a workflow process with reviews by loan officers through PDF and Flex forms it ends with a rejection or acceptance letter. Along the way it introduces the key concepts of LiveCycle and when the course is complete the students should be able to build workflows in LiveCycle.

Last week I was in Lisbon to teach this course to a group of people who are about to use LiveCycle to build workflows for generating contracts in PDF. Monday and Tuesday I covered LiveCycle Developing Forms (LCDV) with a group of up to 13 people, both backend developers and forms developers. Luckily this is a somewhat normal schedule for this course, because having flown in straight from the MAX in San Francisco I was somewhat jet-lagged.

Then Wednesday and Thursday with just the backend developers we had the absolute madness of squeezing 4 days of training plus some custom content on LiveCycle installation and webservice invocation in two days. Luckily the client knew exactly what he wanted so we could skip several chapters and I just demonstrated several of the walkthroughs instead of everybody completing them, but I think this was still a little too much.

Then Friday we switched gears completely. First I wrapped up the webservices content and then in a brainstorm session we designed a process that matched a simplified version of their business case. We ended up with a master process with 7 subprocesses on the whiteboard. After some more discussion we had the in and out variables of each subprocess and I put everybody to work on a subprocess. As soon as people had defined their process with variables I wired them all together in the main process on the projector and then I helped people with issues with their individual subprocess.

I have rarely had so much fun in a training. I had to run around and was answering questions all day (I am very glad the Portugese take their time for their lunch), but it is fun to be challenged as a trainer.

What is more, I think that the students got more out of training this way then if we had followed just the regular schedule. Doing this group building process not only what is taught is really put in practice, but the additional aspects of application design, decoupling processes and API design become much clearer. It is a good thing I had some really smart people in the class because I would never have been able to write the XSLT they did in such a short time, but that just shows how even in such a short exercise skills can complement eachother.

So here is my recommendation: if you want to get the 4 day LCBA training for your team, make sure you get a trainer with field-experience with LiveCycle and tag on an extra day to design and build your first application together. It will prepare your team so much better.

With the MS SQL Server 2008 obstacle out of the way (I am back to MS SQL Server 2005) I am finally finishing the setup of my LiveCycle ES development environment. The applications I need are:

• LiveCycle ES with PDF Generator native application support
• LiveCycle Workbench with LiveCycle Designer
• Acrobat Professional
• FlexBuilder
• the usual browser, AIR, FlashPlayer etc.

In order to get LiveCycle ES PDFG Native Application support to work I need to install Acrobat Pro before LiveCycle ES first. For the Workbench, I need to have FlashPlayer 9 (somehow it won’t work with FlashPlayer 10), but I can not have Acrobat Pro installed. With that, the installation order becomes:

1. uninstall all FlashPlayer versions.
2.  install Flash Player 9 ActiveX (from the LiveCycle ES download).
3. install LiveCycle Workbench.
4. install Acrobat Pro fom CS4. It will generate an error and tell you it didn’t install correctly. When you run the install log analysis tool, it will tell you that yes an error did occur, but it is an unknown error with no known resolution. Ignore it, just test that Acrobat Pro works.
5. install LiveCycle ES with PDFG and native application support.
6. install FlexBuilder.
7. update your FlashPlayer to version 10.

That’s it. All dependencies are now satisfied and all applications now work. Except perhaps Acrobat Pro, I still don’t know what that error was about and it will probably crash horribly in the middle of the next LiveCycle training. And it only took me 6 tries to get there.

So I have a shiny new laptop (Lenovo Thinkpad T61p) that I have been configuring for the last 2 weeks now. I have installed all the office applications and most of my development tools. ColdFusion 8.0.1, Eclipse 3.4, FlexBuilder, CS4 and MS SQL Server Express 2008. verything worked reasonably well. Except that I had to update the JDBC drivers from Microsoft to version 2 CTP, and then I had to switch to the JDBC 4 drivers because the other driver is supposedly incompatible with an 1.6 JVM. (It worked just fine with the 1.x versions of the drivers.)

And that is where the trouble started, many queries suddenly gave datatype resolution errors in ColdFusion. After some testing it turned out that problem queries are typically similar to:

	SELECT
		T.credits
		, PT.pageText AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

In a subsequent QoQ there would be an error if the datatype of PT.pageText was an N-datatype. What fixes the problem is to change the datatype using an explicit cast:

	SELECT
		T.credits
		, CAST(PT.pageText AS VARCHAR) AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

Not something that I fancy changing in all queries, but since this version of the driver is still a technology preview not something that really bothers me either.

Next on the list of applications to install was LiveCycle ES. Installation was a bit troublesome, the installer doesn’t recognize Flash 10, the LC Designer from LC ES is incompatible with the LC Designer from CS4 and the configuration instructions haven’t been updated for CS4 yet, but in the end I got it all installed. But not running. LC Designer crashed consistently within 3 seconds of being started. After lots of digging and searching I finally stumbled upon an explanation of the problem.

SQL Server installs a newer version of the MFC DLL on top of the one we install, and the new version has a critical bug.

Other people have reported there are lots of other applications that suddenly start crashing as soon as MS SQL Server 2008 is installed. Or maybe it is not really MS SQL Server 2008 but on of the dependencies, like the .NET Framework 3.5. I don’t know, I don’t care, I think MS SQL Server 2008 has wasted enough of my time and I have uninstalled it completely.

Adobe just released Security Bulletin APSB08-21 for a ColdFusion privilege escalation issue. The issue described in Security Bulletin APSB08-21 is only applicable to your ColdFusion installation if you are using Sandbox Security. If you have configured a Sandbox to limit access to specific parts of the filesystem it may be possible to access information outside the Sandbox. This issue is particularly important for shared hosting servers because they are the most likely to have Sandbox Security enabled.
There are patches available for ColdFusion 7.0.2, 8.0.0 and 8.0.1. At Prisma IT we have been testing the patch for ColdFusion 8.0.1 for a while now and we have not found any side effects from applying it to our shared hosting servers.

If you are a ColdFusion user and this blog post is the first you read about this issue you really should subscribe to the Adobe Security Notification Service. You will get emails for all the important security updates from Adobe and it is an invaluable tool to staying on top of security.