“it could be bunnies”

ColdFusion 8 includes the very nice server monitor that allows you to see what is happening in real time. The problem with the server monitor is that it runs in the same process as ColdFusion so when ColdFusion becomes unresponsive there is a chance that the server monitor is inaccessible as well. And of course the server monitor is Enterprise Edition only.

But some of the functionality in the server monitor is built on native functionality of the JVM and there are some tools in the JDK that can extract some of the same information from a running ColdFusion server. The most important one is jstack. Jstack is a program that can connect to any running Java program and create a stack dump of all running threads and some memory statistics. Jstack is not included with the JVM that Adobe uses for ColdFusion 8 so you have to get a full JDK from Sun to get a copy. (While you are at it, why not install JDK 6 update 10 or later that solves the class loading bug in earlier JDK 6 versions?) Once you have that you can simply getting a stack trace by running the command “jstack <pid>” where pid is the process identifier of the ColdFusion (JRun) process you want to investigate.

The complication is that you need to do this from the same useraccount as the account the process you want to analyze is running. On Linux that is simply a matter of su, but on Windows this is more complicated. If you are running ColdFusion in the default configuration where  it runs under the SYSTEM account you need to start another program as SYSTEM, and that is not as easy as it may seem because you can not use the runas command to switch to that account. Some time ago a solution for that was posted at the MSDN blogs that involved creating a temporary service and starting that. I have packaged that solution as the following batchfile:

sc create testsvc binpath= "cmd /K start" type= own type= interact
sc start testsvc
sc delete testsvc

When I run this on Windows I get a nice command prompt under the system account. If you want to run this over a remote desktop session, make sure you are connected to the console session (use “mstsc /admin /v:servername” to start your remote desktop session). From that command line I can now run jstack, jmap and all the other JDK tools to analyze a running Java program.

In the first part we have set the stage for this series: the goal is to protect one shared hosting customer from an ‘inside attack’ by another shared hosting customer on the same ColdFusion instance.

If no precautions have been taken, attacking is a simple job. The starting point is a template that does a directory listing of every drive from “a:\” to “z:\” using cfdirectory. When you find something interesting, like the “coldfusion8″ directory, the “jrun4″ directory or a “customers” directory, just drill down from there. Not elegant, but very effective. The only way to hide from it is to use netwerk shares with hard to guess names, but even then it is just a matter of drilling down from the #server.coldfusion.rootdir# to the logfiles and see what the directory paths in some of the mappings and logged errors are.
Hiding (i.e. obscurity) provides no effective hurdle against an inside job and hence no real security.

To get real security against this reading of directories we need to enable Sandbox Security. Sandbox Security allows us to define a directory on the filesystem as a Sandbox and subject every request that starts from that Sandbox to a set of constraints. These constraints can include which tags are allowed, i.e. forbid cfregistry outright, or which resources can be accessed. Typically each Sandbox is defined at the root of a customers FTP and / or WWW directory and then allows for access of only some directories and datasources. Setting up the allowed resources and tags in a Sandbox can occasionally be a bit counterintuitive, for instance to allow a file to be used in a cfinclude it needs execute permissions and several extra directories need to be accessible for some tags.

The thing with Sandbox Security is that it is a feature that is only available in Enterprse Edition. ColdFusion 8 Standard Edition has the ability to restrict the usage of tags, functions and resources as well, but everybody operates in the same Sandbox. So while we can disallow customer A from reading the files of customer B through cfdirectory, that would also disallow customer B from reading his own files. And disabling tags and functions won’t be foolproof either because there is always a way around that. For instance, even if you can’t read a certain file with cffile that doesn’t stop you from mailing it to yourself using cfmailparam.

So here we see the first issue with shared hosting and security: in order to combine them and get a system that is even remotely securable, the hoster needs to invest significantly in a ColdFusion 8 Enterprise Edition license and needs to figure out how to configure Sandbox Security properly. Obviously (if the hoster even decides to bother with all that in the first place) that expense gets charged to the hosted customers, making ColdFusion hosting more expensive then for instance PHP hosting.

A frequent cause for complaints among ColdFusion developers is that shared hosting for ColdFusion is hard to find, expensive and/or limited in functionality. I am not really affected by that since I don’t develop applications that are intended to be deployed in a shared hosting environment. But I do get to see the other side of the medal: at Prisma IT we offer some shared hosting (it is not an area we focus on) and I am responsible for the architecture of the platform. So in this and some followup articles I will dive into some of the complications of securing a shared hosting platform running Adobe ColdFusion 8 on Windows.

Setting the stage

These articles will specifically be about ‘inside jobs’: securing ColdFusion shared hosting customers from each other, while they run on the same ColdFusion instance. This is not about securing servers from outside threats, the starting point is some customer who can FTP his CFML templates to the server and wants to look at the templates and the database records of another customer on the same system.
For this discussion I will largely ignore the complications of combining ColdFusion with other scripting languages. While it is most certainly an interesting subject, the issues involved are too diverse for some general blogs.

Topics

The topics aren’t set in stone yet, but for now I am writing on the following blogs:

• Filesystem access and Sandbox Security
• Memory access
• ColdFusion internals
• Java, COM and .NET
• Microsoft Access

Feel free to suggest more / other content in your comments.

I’m sure that if you have never worked with the LiveCycle training material you have no idea what that title is about. During the LiveCycle Building Applications (LCBA) training a mortgage workflow for the fictional company Fin@nceCorp is build step by step. It starts with a PDF form to submit a mortgage application, then through a workflow process with reviews by loan officers through PDF and Flex forms it ends with a rejection or acceptance letter. Along the way it introduces the key concepts of LiveCycle and when the course is complete the students should be able to build workflows in LiveCycle.

Last week I was in Lisbon to teach this course to a group of people who are about to use LiveCycle to build workflows for generating contracts in PDF. Monday and Tuesday I covered LiveCycle Developing Forms (LCDV) with a group of up to 13 people, both backend developers and forms developers. Luckily this is a somewhat normal schedule for this course, because having flown in straight from the MAX in San Francisco I was somewhat jet-lagged.

Then Wednesday and Thursday with just the backend developers we had the absolute madness of squeezing 4 days of training plus some custom content on LiveCycle installation and webservice invocation in two days. Luckily the client knew exactly what he wanted so we could skip several chapters and I just demonstrated several of the walkthroughs instead of everybody completing them, but I think this was still a little too much.

Then Friday we switched gears completely. First I wrapped up the webservices content and then in a brainstorm session we designed a process that matched a simplified version of their business case. We ended up with a master process with 7 subprocesses on the whiteboard. After some more discussion we had the in and out variables of each subprocess and I put everybody to work on a subprocess. As soon as people had defined their process with variables I wired them all together in the main process on the projector and then I helped people with issues with their individual subprocess.

I have rarely had so much fun in a training. I had to run around and was answering questions all day (I am very glad the Portugese take their time for their lunch), but it is fun to be challenged as a trainer.

What is more, I think that the students got more out of training this way then if we had followed just the regular schedule. Doing this group building process not only what is taught is really put in practice, but the additional aspects of application design, decoupling processes and API design become much clearer. It is a good thing I had some really smart people in the class because I would never have been able to write the XSLT they did in such a short time, but that just shows how even in such a short exercise skills can complement eachother.

So here is my recommendation: if you want to get the 4 day LCBA training for your team, make sure you get a trainer with field-experience with LiveCycle and tag on an extra day to design and build your first application together. It will prepare your team so much better.

MAX 2008 is over so I though I would add my list of highlights and random thoughts.

C/C++ to AS3 conversion

Last year at MAX Adobe talked a bit about converting c/c++ to AS3 during the sneak peaks. This year there was some more information and there was a session on Project Alchemy. The way it works is that Adobe has a bunch of library code that mimics the standard C libraries, a toolchain to configure and compile and then you can call your library with a bunch of glue code from AS3. This is far from finished, it is just a research project and not even a pre-product, but it is available on labs.

The demo’s were pretty impressive and showed Quake, crypto libraries (OpenSSL)  and sound code. Most of the technical stuff was way over my head, but one of the comments I did understand was that there were no OS libraries and no socket code. So I guess we will have to wait for a while before we can compile C++ database drivers to AS3 and build native database tools (unless you want to try beta drivers for MySQL).

Adobe Enterprise Developer Program (AEDP)

The AEDP is a new program to help Enterprise developers (Flex, AIR and LiveCycle ES) with direct access to Adobe support engineers. For an annual fee Adobe commits to providing tools, software and most importantly support with a guaranteed 1 business day response time. It seems tobe mostly geared towards Adobe LiveCycle ES customers.

Looks like the support is not for the other Enterprise product, ColdFusion. I guess it isn’t really needed there anyway with the engineering team being regulars in the community and on mailinglists.

Flash Catalyst

Formerly known as Project Thermo, Flash Catalyst will bring designer - developer workflows for Flex.  Designers working with Flash Catalyst can work in the same project as developers working with FlexBuilder without having to go through a whole manual export from CS4 - import into FB cycle after every change. The current Beta on labs is Mac only at the moment.

AIR 1.5 released

And it immediately fixed several crashing issues I had with the previous version.

ColdFusion IDE announced

At last Adobe announced a new IDE for ColdFusion: Bolt. The name is a tribute to the lightning bolt that used to be the ColdFusion logo. Apart from a fleeting mention of support for the native ORM due in Centaur and support for community frameworks no further information is available at the moment.

And for the random thoughts.

Which labs not to choose

To start with the bad stuff I have come up with a way to identify which labs are a waste of time. If a lab is:

• presented by somebody from Adobe Consultancy,
• who is not a native speaker,
• and does not have an Instructor certification,

don’t go. It is a waste of time. The lab will be written without proper definition of learning targets, without a delivery strategy and will end with a ten minute monologue that is more about “look how much I know about Maven/EJB/whatever” then about teaching.

Adobe should wake up and realize that labs should be taught by certified instructors and evangelists.

In the interest of full disclosure, yes, I am a certified instructor. And I am not the easiest one for my students, because I want to train them to the level where if they were on my development team I would feel that I could assign tasks to them and they could complete them (as opposed to a teaching strategy where you leave nobody behind even if that slows the entire group down). But some of the labs during MAX were outright ridiculous. And not just this year, the same happens every year.

Where was LiveCycle?

As a product manager of another product said: “With more then 80 products not every product can get a mention.” But LiveCycle is Adobe’s flagship, multi-billion dollar Enterprise suite and it was nowhere. In three days the only mention LiveCycle got on the main stage was as a hidden part in a movie production workflow during a sneak peak. There were only 2 advanced sessions on the LiveCycle servers. I know LiveCycle isn’t the sexiest product around, but this is a bit extreme.

On the upside there were people at the support lab that knew about LiveCycle and they are going to open a case to investigate my installation problems.

Cloud Computing?

One of the themes of the keynote was “Client + Cloud”. So what is the progress on taking Adobe’s products to the cloud? Does ColdFusion have any licensing form to allow it to run on the cloud for a per CPU-hour price? No. Does LiveCycle have any licensing form at all to allow it to run on the cloud for a per CPU-hour price? No.

The only progress on taking the existing products to the cloud is that LiveCycle will be put on the cloud by Adobe and people can get 10 hours of cloud-time for a testdrive. “Getting our toes wet with cloud computing” was what Adobe called it. If you are going to do that, you might as well install the developer editin of LiveCycle and take it for a 1 year testdrive.

With the MS SQL Server 2008 obstacle out of the way (I am back to MS SQL Server 2005) I am finally finishing the setup of my LiveCycle ES development environment. The applications I need are:

• LiveCycle ES with PDF Generator native application support
• LiveCycle Workbench with LiveCycle Designer
• Acrobat Professional
• FlexBuilder
• the usual browser, AIR, FlashPlayer etc.

In order to get LiveCycle ES PDFG Native Application support to work I need to install Acrobat Pro before LiveCycle ES first. For the Workbench, I need to have FlashPlayer 9 (somehow it won’t work with FlashPlayer 10), but I can not have Acrobat Pro installed. With that, the installation order becomes:

1. uninstall all FlashPlayer versions.
2.  install Flash Player 9 ActiveX (from the LiveCycle ES download).
3. install LiveCycle Workbench.
4. install Acrobat Pro fom CS4. It will generate an error and tell you it didn’t install correctly. When you run the install log analysis tool, it will tell you that yes an error did occur, but it is an unknown error with no known resolution. Ignore it, just test that Acrobat Pro works.
5. install LiveCycle ES with PDFG and native application support.
6. install FlexBuilder.
7. update your FlashPlayer to version 10.

That’s it. All dependencies are now satisfied and all applications now work. Except perhaps Acrobat Pro, I still don’t know what that error was about and it will probably crash horribly in the middle of the next LiveCycle training. And it only took me 6 tries to get there.

So I have a shiny new laptop (Lenovo Thinkpad T61p) that I have been configuring for the last 2 weeks now. I have installed all the office applications and most of my development tools. ColdFusion 8.0.1, Eclipse 3.4, FlexBuilder, CS4 and MS SQL Server Express 2008. verything worked reasonably well. Except that I had to update the JDBC drivers from Microsoft to version 2 CTP, and then I had to switch to the JDBC 4 drivers because the other driver is supposedly incompatible with an 1.6 JVM. (It worked just fine with the 1.x versions of the drivers.)

And that is where the trouble started, many queries suddenly gave datatype resolution errors in ColdFusion. After some testing it turned out that problem queries are typically similar to:

	SELECT
		T.credits
		, PT.pageText AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

In a subsequent QoQ there would be an error if the datatype of PT.pageText was an N-datatype. What fixes the problem is to change the datatype using an explicit cast:

	SELECT
		T.credits
		, CAST(PT.pageText AS VARCHAR) AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

Not something that I fancy changing in all queries, but since this version of the driver is still a technology preview not something that really bothers me either.

Next on the list of applications to install was LiveCycle ES. Installation was a bit troublesome, the installer doesn’t recognize Flash 10, the LC Designer from LC ES is incompatible with the LC Designer from CS4 and the configuration instructions haven’t been updated for CS4 yet, but in the end I got it all installed. But not running. LC Designer crashed consistently within 3 seconds of being started. After lots of digging and searching I finally stumbled upon an explanation of the problem.

SQL Server installs a newer version of the MFC DLL on top of the one we install, and the new version has a critical bug.

Other people have reported there are lots of other applications that suddenly start crashing as soon as MS SQL Server 2008 is installed. Or maybe it is not really MS SQL Server 2008 but on of the dependencies, like the .NET Framework 3.5. I don’t know, I don’t care, I think MS SQL Server 2008 has wasted enough of my time and I have uninstalled it completely.

Adobe just released Security Bulletin APSB08-21 for a ColdFusion privilege escalation issue. The issue described in Security Bulletin APSB08-21 is only applicable to your ColdFusion installation if you are using Sandbox Security. If you have configured a Sandbox to limit access to specific parts of the filesystem it may be possible to access information outside the Sandbox. This issue is particularly important for shared hosting servers because they are the most likely to have Sandbox Security enabled.
There are patches available for ColdFusion 7.0.2, 8.0.0 and 8.0.1. At Prisma IT we have been testing the patch for ColdFusion 8.0.1 for a while now and we have not found any side effects from applying it to our shared hosting servers.

If you are a ColdFusion user and this blog post is the first you read about this issue you really should subscribe to the Adobe Security Notification Service. You will get emails for all the important security updates from Adobe and it is an invaluable tool to staying on top of security.

By default ColdFusion will use the computer name without the domain name appended when sending email. However, some mail servers require that senders use a Fully Qualified Domain Name (FQDN) in their EHLO. If that is the case, you may get errors in your mail.log that look something like this:

Sep 18 17:22:11 mail postfix/smtpd[55543]: NOQUEUE: reject: RCPT from
prlt004[145.94.255.255]: 504 5.5.2 mail3.prisma-it.com: Helo command
rejected: need fully-qualified hostname; from=<email-address>
to=<email-address> proto=ESMTP helo=mail3.prisma-it.com

A similar problem exists with Message-IDs and spam filters and Adobe has TechNote kb400753 ColdFusion MX: Configuring cfmail to use a Fully Qualified Domain Name which describes a solution to fix the Message-ID. Luckily a similar solution works for changing the FQDN used in the EHLO. Just add a mail.smtp.localhost to your jvm.config with the right FQDN as value, restart your ColdFusion instance and you can send email again. My jvm.config uses:

-Dmail.host=jochem.vandieten.net -Dmail.smtp.localhost=jochem.vandieten.net

So yesterday I wrote about installing 2 versions of LiveCycle ES, but with the limitation that you can’t run them simultaneously. I didn’t really intend to try to run them simultaneously because I thought it would be a lot of work, but it did feel unfinished and that kept nagging. So a little over an hour ago I decided to give it a shot anyway. Not the way you are supposed to do it, but using a brute force approach. The first step was using “netstat -ano” to register all the ports in use by the JBoss and MySQL processes. After filtering that produced the following list:

Active Connections
 
  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:1098           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:1099           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:1100           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:1101           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:3306           0.0.0.0:0              LISTENING       2392
  TCP    0.0.0.0:3528           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:4444           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:4445           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:4446           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:4447           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:8083           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:8093           0.0.0.0:0              LISTENING       1032
  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       1032
  UDP    0.0.0.0:1161           *:*                                    1032
  UDP    0.0.0.0:1162           *:*                                    1032
  UDP    192.168.0.121:1496     *:*                                    1032
  UDP    192.168.0.121:1498     *:*                                    1032
  UDP    192.168.0.121:1501     *:*                                    1032

The next step was a search of all the configuration files to find all occurences of these port numbers. All the configuration files means everything with the extensions .properties, .xml and .ini in all the subfolders of “c:\Adobe\LiveCycle8.3\jboss\” and “c:\Adobe\LiveCycle8.3\mysql\”. I did some fuzzy selection on which of these files were really relevant and changed the port numbers to a number exactly 1000 higher in the following files:

C:\Adobe\LiveCycle8.2\jboss\server\all\conf\jacorb.properties
C:\Adobe\LiveCycle8.2\jboss\server\all\conf\jboss-minimal.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\conf\jboss-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\cluster-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\httpha-invoker.sar\META-INF\jboss-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\jboss-ws4ee.sar\META-INF\jboss-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\jbossweb-tomcat55.sar\server.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\jms\hajndi-jms-ds.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\juddi-service.sar\juddiws.war\WEB-INF\juddi.properties
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\snmp-adaptor.sar\META-INF\jboss-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy-hasingleton\jms\uil2-service.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\adobe-ds.xml
C:\Adobe\LiveCycle8.2\jboss\server\all\deploy\mysql-ds.xml
C:\Adobe\LiveCycle8.2\mysql\my.ini

This appears to work (for some value of work). Anytime now I will get an alarm because the system overheated and imploded (it has only 2 GB of RAM), but the server started, the server.log is clean and I am logged on to both the LiveCycle 8 Admin on port 8080 and the LiveCycle 8.2 Admin on port 9080.So as long as you’re willing to bypass all the rules on how it should be done and just go for the brute force approach it appears to be pretty easy to run 2 versions of LiveCycle ES simultaneously.