“it could be bunnies”

I have been hit by a mysterious msvcr71.dll not found error when upgrading the JDK on a CF 8 install on an on and off basis. Today I finally got annoyed enough with it to start digging into the cause. Apparently this is a known issue with Java 6:

Windows Java SE 6 applications using custom launchers must be installed with msvcr71.dll in the same directory as the launcher executable. According to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_CRT_C_Run.2d.Time_Libraries.asp, this is the new Microsoft C runtime distribution model:

“An application should use and redistribute msvcr71.dll, and it should avoid placing a copy or using an existing copy of msvcr71.dll in the system directory. Instead, the application should keep a copy of msvcr71.dll in its application directory with the program executable. Any application built with Visual C++ .NET using the /MD switch will necessarily use msvcr71.dll.”

This also explains the on and off nature of the error. The way I understand it, you need to go through the following steps before you experience the issue:

1. you have deployed a portable WAR/EAR file(the portable WAR/EAR files do not contain all the .dll files the Windows specific ones do);
2. or you have created extra instances and adapted your java.library path in jvm.config;
3. you perform a JDK upgrade;
4. you have not installed any application that placed msvcr71.dll somewhere on the path.

I think that explains why I have been experiencing this issue in some cases and not in others. Anyway, Sun’s recommendation to drop msvcr71.dll in the executable folder with jrun.exe solves the issue. (Tempting as it may be to drop it in the JDK folder, it is better to put it in the JRun folder: that way it still works when you upgrade your JDK next time.) I have reported the issue with Adobe so it should be solved in Centaur.

Last February I posted on how storing UUIDs using the native UUID datatype in PostgreSQL was about 15% faster then using a VARCHAR(35) datatype. In March I posted about the performance effects of the “String Format” setting for MS SQL Server databases when using cfqueryparam. Recently I have been working on checking the performance effects when combining different datatypes for UUIDs with the string formats settings for a MS SQL Server 2005 database. (Or to be more exact, I have been checking the performance effects of different datatypes for storing 128-bit integers in the database.)

The starting point is an existing database for an application. That database stores UUIDs using a VARCHAR(35) datatype. From that database I extracted the CREATE statements for the schema using SQL Server Management Studio and build a script from there that follows the following pattern:

1. create database
2. create tables including:
   • primary key constraints
   • indexes
   • check constraints
3. load data (and transform where needed)
4. create foreign key constraints

Due to the ordering of the load script where all data is loaded before the foreign keys are created the order in which the tables are created and loaded doesn’t really matter: there will never be foreign key constraint errors, not even with circular foreign keys.

For the different datatypes I then manipulated the script a little bit. The first change was a replace to change all occurrences of VARCHAR(35) to NVARCHAR(35). The second was to change all occurrences of VARCHAR(35) with UNIQUEIDENTIFIER. For the last one I then wrote a function to convert the ColdFusion UUID format to the standard format the the UNIQUEIDENTIFIER datatype expects. This function looks like:

CREATE  function var_to_UUID(@cfUUID VARCHAR(35)) RETURNS uniqueidentifier AS
BEGIN
  Return
    CASE
      WHEN Len(@cfUUID) = 35 THEN Substring(@cfUUID,1,23) + '-' + Substring(@cfUUID,24,12)
      ELSE NULL
    END;
END

Next I had to manipulate the data loading script to use this function to convert all the UUIDs. That required some manual editing to make all the insert statements look like:

INSERT INTO [testUUID].[dbo].[language](
  [languageID]
  ,[language]
  )
SELECT
  testUUID.dbo.var_to_uuid([languageID])
  ,[language]
FROM
  [source_DB].[dbo].[language]

The database I used for these tests is on the one hand far from trivial: 80+ tables with normal and composite primary keys, 120+ foreign keys and many check constraints. On the other hand it is very trivial: just one stored procedure and one UDF. To make sure that I am comparing apples to apples I ran the existing database through the create and load sequence to establish my base for comparison and I ran all databases through the example index optimization script from MSDN. The numbers I got after just that sequence are:

As you can see the database size correlates very well to the storage size of the different datatypes. Apparently we have about 26.5 GB of non-UUID data in the database (which for this database is mostly in NTEXT fields in 2 tables of 18 GB and 6 GB data size respectively). For the tables most and least affected by the change the statistics are:

So as you can see there may be good reason to pick the right datatype: if your schema has many UUID columns you can save upto a factor three in storage space (or hardly any space at al if you have few UUIDs). Looking at it from the MS SQL Sever perspective the preferred storage format is UNIQUEIDENTIFIER, followed by VARCHAR(35), and lastly NVARCHAR(35). But MS SQL Server is only half of the equation, ColdFusion is the other half and in the next installment I will show you why the order from the ColdFusion side is radically different.

In the first part we have set the stage for this series: the goal is to protect one shared hosting customer from an ‘inside attack’ by another shared hosting customer on the same ColdFusion instance. In the second part we have gone into the reason why we need ColdFusion Enterprise Edition to secure the filesystem from direct access through cffile and cfdirectory. In this part we will see why we need to take additional steps to secure the filesystem against the other languages we can use in / call from ColdFusion.

Java

Since ColdFusion is written in Java the integration with Java is pretty tight and we can easily use Java code from ColdFusion. So what happens if we try not to use a CFML tag to access a file from some other customer, but use Java directly. You don’t need any Java knowledge to do so, if you Google for some code you will find plenty of examples. For this I downloaded the code from Ben Nadel’s Java Exploration in ColdFusion: java.io.LineNumberReader blogpost, changed the ExpandPath(”./data.txt”) to my template name and ran the code. It displayed the sourcecode of the template, so the code was good. Next I changed the code so the file it pointed to was “c:\\test\\secret.txt”, which is outside my Sandbox. And the code promptly returned an error.

So far so good: our Sandbox protects against reading files directly through Java.

CFEXECUTE

So if we can’t read files from elsewhere on the filesystem directly, perhaps we can copy them to our Sandbox first  using the native functionality of the operating system, and then read them once they are in our Sandbox. (And if you don’t know which files you need, just use xcopy and copy them all recursively.) For that we turn to the cfexecute tag. In the default form that tag would be used like this:

<cfexecute name="c:\windows\system32\cmd.exe" arguments="/c copy #source# #target#" />

But because we are in a Sandbox that will most likely not work  because your Sandbox will not allow you to access the c:\windows\system32\cmd.exe file. To circumvent that, simply upload a copy of cmd.exe to your Sandbox and use that one. With that the code to copy files and then read them through cffile becomes:

So by now it should be clear that CFEXECUTE needs to be disabled to maintain security in a shared hosting environment. The bottom line is that with CFEXECUTE you start a process, that process runs as the same user as ColdFusion, the ColdFusion user by definition has access to the templates of other customers, so the process you started has access to them as well. And this means that useful functionality (albeit used by a limited number of users) needs to bes disabled.

COM, .NET and Java again

The problem is that CFEXECUTE is not the only way to start a new process. COM, .NET and Java can do so as well. Below is a simple code example that shows how to dynamically generate a batchfile, write that to the filesystem inside the Sandbox, execute it through Java, and read the result.

<!--- target file outside our Sandbox --->
<cfset secretFile = server.coldfusion.rootdir & "\lib\neo-datasource.xml" />
<!--- batch file inside our Sandbox --->
<cfset batchFile = ExpandPath("./copy.bat") />
<!--- location where we copy the file inside our Sandbox --->
<cfset readableCopyOfSecretFile = ExpandPath("./readableCopyOfSecretFile.xml") />
<!--- copy command for in our batchfile --->
<cfset batchFileContent = 'cmd /c "copy #secretfile# #readableCopyOfSecretFile#"' />
 
<!--- write the batchfile inside our Sandbox --->
<cffile action="write" file="#batchFile#" output="#batchFileContent#" />
 
<!--- execute the batchfile --->
<cfset CreateObject("java", "java.lang.Runtime").getRuntime().exec(batchFile) />
 
<!--- read the copy of file inside the sandbox --->
<cffile action="read" file="#readableCopyOfSecretFile#" variable="fileContent" />
<!--- display the contects of the file --->
<cfoutput>#HTMLCodeFormat(fileContent)#</cfoutput>

We can prevent this by modifying the Sandbox configuration and disabling the execute permission from the Sandbox, but since ColdFusion itself needs execute permissions to run CFML that will kill ColdFusion as well.  Executing processes from COM works the same as from Java. From .NET it works slightly different because the actual .NET service runs in another process and could potentially have different permissions from the ColdFusion account, but the inability to differentiate the permissions between different users remains.

So the hoster is left with a hard choice: disable CFEXECUTE, CFOBJECT, CreateObject(.NET), CreateObject(COM) and CreateObject(JAVA) or accept that there is no security whatsoever in the shared hosting configuration. If you disable these tags a lot of applications and frameworks won’t work anymore. For instance Transfer ORM needs Java access, so any application build on top of it will not work in a secured shared hosting environment.

Warning

A long time ago I had some other code examples online that could be used to extend the functionality of a shared hosting environment. I have taken these examples down not just because with the arrival of the Admin API they are obsolete, but also because I was informed that somebody had wrecked a shared hosting server with them. So please before you start uploading and running these examples on your shared host, please take a moment to consider the following points:

• If you want to look further into this code, why not do that on your own development environment?
• In most jurisdictions accessing other peoples data / code is illegal.
• Some of the code examples (especially if you replace copy with xcopy) can consume lots of resources.
• You will never be able to test whether your hosting account is protected from others. You will only be able to test if others are protected from you. So really, why not on your own development environment?

ColdFusion 8 includes the very nice server monitor that allows you to see what is happening in real time. The problem with the server monitor is that it runs in the same process as ColdFusion so when ColdFusion becomes unresponsive there is a chance that the server monitor is inaccessible as well. And of course the server monitor is Enterprise Edition only.

But some of the functionality in the server monitor is built on native functionality of the JVM and there are some tools in the JDK that can extract some of the same information from a running ColdFusion server. The most important one is jstack. Jstack is a program that can connect to any running Java program and create a stack dump of all running threads and some memory statistics. Jstack is not included with the JVM that Adobe uses for ColdFusion 8 so you have to get a full JDK from Sun to get a copy. (While you are at it, why not install JDK 6 update 10 or later that solves the class loading bug in earlier JDK 6 versions?) Once you have that you can simply getting a stack trace by running the command “jstack <pid>” where pid is the process identifier of the ColdFusion (JRun) process you want to investigate.

The complication is that you need to do this from the same useraccount as the account the process you want to analyze is running. On Linux that is simply a matter of su, but on Windows this is more complicated. If you are running ColdFusion in the default configuration where  it runs under the SYSTEM account you need to start another program as SYSTEM, and that is not as easy as it may seem because you can not use the runas command to switch to that account. Some time ago a solution for that was posted at the MSDN blogs that involved creating a temporary service and starting that. I have packaged that solution as the following batchfile:

sc create testsvc binpath= "cmd /K start" type= own type= interact
sc start testsvc
sc delete testsvc

When I run this on Windows I get a nice command prompt under the system account. If you want to run this over a remote desktop session, make sure you are connected to the console session (use “mstsc /admin /v:servername” to start your remote desktop session). From that command line I can now run jstack, jmap and all the other JDK tools to analyze a running Java program.

In the first part we have set the stage for this series: the goal is to protect one shared hosting customer from an ‘inside attack’ by another shared hosting customer on the same ColdFusion instance.

If no precautions have been taken, attacking is a simple job. The starting point is a template that does a directory listing of every drive from “a:\” to “z:\” using cfdirectory. When you find something interesting, like the “coldfusion8″ directory, the “jrun4″ directory or a “customers” directory, just drill down from there. Not elegant, but very effective. The only way to hide from it is to use netwerk shares with hard to guess names, but even then it is just a matter of drilling down from the #server.coldfusion.rootdir# to the logfiles and see what the directory paths in some of the mappings and logged errors are.
Hiding (i.e. obscurity) provides no effective hurdle against an inside job and hence no real security.

To get real security against this reading of directories we need to enable Sandbox Security. Sandbox Security allows us to define a directory on the filesystem as a Sandbox and subject every request that starts from that Sandbox to a set of constraints. These constraints can include which tags are allowed, i.e. forbid cfregistry outright, or which resources can be accessed. Typically each Sandbox is defined at the root of a customers FTP and / or WWW directory and then allows for access of only some directories and datasources. Setting up the allowed resources and tags in a Sandbox can occasionally be a bit counterintuitive, for instance to allow a file to be used in a cfinclude it needs execute permissions and several extra directories need to be accessible for some tags.

The thing with Sandbox Security is that it is a feature that is only available in Enterprse Edition. ColdFusion 8 Standard Edition has the ability to restrict the usage of tags, functions and resources as well, but everybody operates in the same Sandbox. So while we can disallow customer A from reading the files of customer B through cfdirectory, that would also disallow customer B from reading his own files. And disabling tags and functions won’t be foolproof either because there is always a way around that. For instance, even if you can’t read a certain file with cffile that doesn’t stop you from mailing it to yourself using cfmailparam.

So here we see the first issue with shared hosting and security: in order to combine them and get a system that is even remotely securable, the hoster needs to invest significantly in a ColdFusion 8 Enterprise Edition license and needs to figure out how to configure Sandbox Security properly. Obviously (if the hoster even decides to bother with all that in the first place) that expense gets charged to the hosted customers, making ColdFusion hosting more expensive then for instance PHP hosting.

A frequent cause for complaints among ColdFusion developers is that shared hosting for ColdFusion is hard to find, expensive and/or limited in functionality. I am not really affected by that since I don’t develop applications that are intended to be deployed in a shared hosting environment. But I do get to see the other side of the medal: at Prisma IT we offer some shared hosting (it is not an area we focus on) and I am responsible for the architecture of the platform. So in this and some followup articles I will dive into some of the complications of securing a shared hosting platform running Adobe ColdFusion 8 on Windows.

Setting the stage

These articles will specifically be about ‘inside jobs’: securing ColdFusion shared hosting customers from each other, while they run on the same ColdFusion instance. This is not about securing servers from outside threats, the starting point is some customer who can FTP his CFML templates to the server and wants to look at the templates and the database records of another customer on the same system.
For this discussion I will largely ignore the complications of combining ColdFusion with other scripting languages. While it is most certainly an interesting subject, the issues involved are too diverse for some general blogs.

Topics

The topics aren’t set in stone yet, but for now I am writing on the following blogs:

• Filesystem access and Sandbox Security
• Memory access
• ColdFusion internals
• Java, COM and .NET
• Microsoft Access

Feel free to suggest more / other content in your comments.

MAX 2008 is over so I though I would add my list of highlights and random thoughts.

C/C++ to AS3 conversion

Last year at MAX Adobe talked a bit about converting c/c++ to AS3 during the sneak peaks. This year there was some more information and there was a session on Project Alchemy. The way it works is that Adobe has a bunch of library code that mimics the standard C libraries, a toolchain to configure and compile and then you can call your library with a bunch of glue code from AS3. This is far from finished, it is just a research project and not even a pre-product, but it is available on labs.

The demo’s were pretty impressive and showed Quake, crypto libraries (OpenSSL)  and sound code. Most of the technical stuff was way over my head, but one of the comments I did understand was that there were no OS libraries and no socket code. So I guess we will have to wait for a while before we can compile C++ database drivers to AS3 and build native database tools (unless you want to try beta drivers for MySQL).

Adobe Enterprise Developer Program (AEDP)

The AEDP is a new program to help Enterprise developers (Flex, AIR and LiveCycle ES) with direct access to Adobe support engineers. For an annual fee Adobe commits to providing tools, software and most importantly support with a guaranteed 1 business day response time. It seems tobe mostly geared towards Adobe LiveCycle ES customers.

Looks like the support is not for the other Enterprise product, ColdFusion. I guess it isn’t really needed there anyway with the engineering team being regulars in the community and on mailinglists.

Flash Catalyst

Formerly known as Project Thermo, Flash Catalyst will bring designer - developer workflows for Flex.  Designers working with Flash Catalyst can work in the same project as developers working with FlexBuilder without having to go through a whole manual export from CS4 - import into FB cycle after every change. The current Beta on labs is Mac only at the moment.

AIR 1.5 released

And it immediately fixed several crashing issues I had with the previous version.

ColdFusion IDE announced

At last Adobe announced a new IDE for ColdFusion: Bolt. The name is a tribute to the lightning bolt that used to be the ColdFusion logo. Apart from a fleeting mention of support for the native ORM due in Centaur and support for community frameworks no further information is available at the moment.

And for the random thoughts.

Which labs not to choose

To start with the bad stuff I have come up with a way to identify which labs are a waste of time. If a lab is:

• presented by somebody from Adobe Consultancy,
• who is not a native speaker,
• and does not have an Instructor certification,

don’t go. It is a waste of time. The lab will be written without proper definition of learning targets, without a delivery strategy and will end with a ten minute monologue that is more about “look how much I know about Maven/EJB/whatever” then about teaching.

Adobe should wake up and realize that labs should be taught by certified instructors and evangelists.

In the interest of full disclosure, yes, I am a certified instructor. And I am not the easiest one for my students, because I want to train them to the level where if they were on my development team I would feel that I could assign tasks to them and they could complete them (as opposed to a teaching strategy where you leave nobody behind even if that slows the entire group down). But some of the labs during MAX were outright ridiculous. And not just this year, the same happens every year.

Where was LiveCycle?

As a product manager of another product said: “With more then 80 products not every product can get a mention.” But LiveCycle is Adobe’s flagship, multi-billion dollar Enterprise suite and it was nowhere. In three days the only mention LiveCycle got on the main stage was as a hidden part in a movie production workflow during a sneak peak. There were only 2 advanced sessions on the LiveCycle servers. I know LiveCycle isn’t the sexiest product around, but this is a bit extreme.

On the upside there were people at the support lab that knew about LiveCycle and they are going to open a case to investigate my installation problems.

Cloud Computing?

One of the themes of the keynote was “Client + Cloud”. So what is the progress on taking Adobe’s products to the cloud? Does ColdFusion have any licensing form to allow it to run on the cloud for a per CPU-hour price? No. Does LiveCycle have any licensing form at all to allow it to run on the cloud for a per CPU-hour price? No.

The only progress on taking the existing products to the cloud is that LiveCycle will be put on the cloud by Adobe and people can get 10 hours of cloud-time for a testdrive. “Getting our toes wet with cloud computing” was what Adobe called it. If you are going to do that, you might as well install the developer editin of LiveCycle and take it for a 1 year testdrive.

So I have a shiny new laptop (Lenovo Thinkpad T61p) that I have been configuring for the last 2 weeks now. I have installed all the office applications and most of my development tools. ColdFusion 8.0.1, Eclipse 3.4, FlexBuilder, CS4 and MS SQL Server Express 2008. verything worked reasonably well. Except that I had to update the JDBC drivers from Microsoft to version 2 CTP, and then I had to switch to the JDBC 4 drivers because the other driver is supposedly incompatible with an 1.6 JVM. (It worked just fine with the 1.x versions of the drivers.)

And that is where the trouble started, many queries suddenly gave datatype resolution errors in ColdFusion. After some testing it turned out that problem queries are typically similar to:

	SELECT
		T.credits
		, PT.pageText AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

In a subsequent QoQ there would be an error if the datatype of PT.pageText was an N-datatype. What fixes the problem is to change the datatype using an explicit cast:

	SELECT
		T.credits
		, CAST(PT.pageText AS VARCHAR) AS displayName
	FROM
		test
			INNER JOIN
		pageText PT ON T.nameTextID = PT.pageTextID

Not something that I fancy changing in all queries, but since this version of the driver is still a technology preview not something that really bothers me either.

Next on the list of applications to install was LiveCycle ES. Installation was a bit troublesome, the installer doesn’t recognize Flash 10, the LC Designer from LC ES is incompatible with the LC Designer from CS4 and the configuration instructions haven’t been updated for CS4 yet, but in the end I got it all installed. But not running. LC Designer crashed consistently within 3 seconds of being started. After lots of digging and searching I finally stumbled upon an explanation of the problem.

SQL Server installs a newer version of the MFC DLL on top of the one we install, and the new version has a critical bug.

Other people have reported there are lots of other applications that suddenly start crashing as soon as MS SQL Server 2008 is installed. Or maybe it is not really MS SQL Server 2008 but on of the dependencies, like the .NET Framework 3.5. I don’t know, I don’t care, I think MS SQL Server 2008 has wasted enough of my time and I have uninstalled it completely.

Adobe just released Security Bulletin APSB08-21 for a ColdFusion privilege escalation issue. The issue described in Security Bulletin APSB08-21 is only applicable to your ColdFusion installation if you are using Sandbox Security. If you have configured a Sandbox to limit access to specific parts of the filesystem it may be possible to access information outside the Sandbox. This issue is particularly important for shared hosting servers because they are the most likely to have Sandbox Security enabled.
There are patches available for ColdFusion 7.0.2, 8.0.0 and 8.0.1. At Prisma IT we have been testing the patch for ColdFusion 8.0.1 for a while now and we have not found any side effects from applying it to our shared hosting servers.

If you are a ColdFusion user and this blog post is the first you read about this issue you really should subscribe to the Adobe Security Notification Service. You will get emails for all the important security updates from Adobe and it is an invaluable tool to staying on top of security.

By default ColdFusion will use the computer name without the domain name appended when sending email. However, some mail servers require that senders use a Fully Qualified Domain Name (FQDN) in their EHLO. If that is the case, you may get errors in your mail.log that look something like this:

Sep 18 17:22:11 mail postfix/smtpd[55543]: NOQUEUE: reject: RCPT from
prlt004[145.94.255.255]: 504 5.5.2 mail3.prisma-it.com: Helo command
rejected: need fully-qualified hostname; from=<email-address>
to=<email-address> proto=ESMTP helo=mail3.prisma-it.com

A similar problem exists with Message-IDs and spam filters and Adobe has TechNote kb400753 ColdFusion MX: Configuring cfmail to use a Fully Qualified Domain Name which describes a solution to fix the Message-ID. Luckily a similar solution works for changing the FQDN used in the EHLO. Just add a mail.smtp.localhost to your jvm.config with the right FQDN as value, restart your ColdFusion instance and you can send email again. My jvm.config uses:

-Dmail.host=jochem.vandieten.net -Dmail.smtp.localhost=jochem.vandieten.net