Friday, June 11, 2010

HowTo unload JAR files loaded by ClassLoader

Dealing with application that executes thousands of batch jobs in the form of executable JAR files within single JVM (prior 7), I've found JVM issue:
  • any of ClassLoader implementations don't have method like close() for closing opened libraries and resources
  • garbage collector doesn't close JAR files opened by ClassLoader, even if ClassLoader was garbage collected
On the Linux machine where maximum number of open file descriptors is limited (ulimit -n shows 1024 for example), execution thousands of JARs causes following error:
java.lang.RuntimeException: java.io.FileNotFoundException: /some-file.txt (Too many open files)
since all possible file descriptors were allocated for loaded JARs.

Google led me to this post, where the same issue described, but proposed solution is based on calling classes from sun.* packages via reflection. This solution limits application to be executed on SUN JVM only. Fortunately this blog post contains link to the original source of the issue at SUN bug-tracking system, where more graceful workaround was proposed: keep track of all the JAR files used in classpath by means of URL or JarUrlConnection and close JAR files when you need by executing jarUrlConnection .getJarFile().close()

Simple test, showing the idea of unloading JAR files:

import java.net.JarURLConnection;
import java.net.URL;
import java.net.URLClassLoader;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.zip.ZipEntry;
import java.util.jar.JarOutputStream;

public class Test {

    private static File makeTestJar() throws IOException {
        File jarLibFile = File.createTempFile("classloader-test", ".jar");
        jarLibFile.deleteOnExit();

        JarOutputStream jstream =
          new JarOutputStream(new FileOutputStream(jarLibFile));

        jstream.putNextEntry(new ZipEntry("fooboo.txt"));
        jstream.closeEntry();
        jstream.close();

        return jarLibFile;
    }

    public static void main(String[] args) throws Exception
    {
        File f = makeTestJar();
        ArrayList list = new ArrayList();
        byte[] buf = new byte[(int)f.length()];

        FileInputStream in = null;
        boolean unload = args.length > 0 && "unload".equalsIgnoreCase(args[args.length-1]);

        try {
            in = new FileInputStream(f);
          in.read(buf);
        } finally {
          if (in != null) in.close();
        }

        try {
          for (int j=0;j<20;j++) {
              for (int i=0;i<100;i++) {
                    // clone our test JAR
                  File tmpFile = File.createTempFile(f.getName(), ".jar");
                  FileOutputStream out = null;
                  try {
                    out = new FileOutputStream(tmpFile);
                    out.write(buf);
                  } finally {
                    if (out != null) out.close();
                  }
                  tmpFile.deleteOnExit();

                    // open cached connection to just created clone of test JAR
                  URL jarUrl = new URL("jar", "", -1, tmpFile.toURI().toString() + "!/");
                  URLConnection c = jarUrl.openConnection();
                  c.setUseCaches(true);

                  ClassLoader cLoader = new URLClassLoader(new URL[] {jarUrl}, null);
                    // make sure ClassLoader loads our test JAR
                    cLoader.getResource("fooboo.txt");

                  list.add(c);
              }

              if (unload)
              for (URLConnection c:list)
                ((JarURLConnection)c).getJarFile().close();

            list.clear();
          }
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }
}