Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Diagnosing Tomcat memory‑leak warnings and the Abandoned Connection Cleanup thread

Tech 1

Symptom

Tomcat fails to start or restarts with severe and warning messages about filters not initializing, JDBC drivers not being unregistered, and a lingering background thread called Abandoned connection cleanup thread.

Example log excerpt:

02-Apr-2021 10:12:41.915 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. See the container log for details
02-Apr-2021 10:12:41.916 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
02-Apr-2021 10:12:42.037 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it on shutdown. The driver has been forcibly deregistered to prevent a memory leak.
02-Apr-2021 10:12:42.038 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it on shutdown. The driver has been forcibly deregistered to prevent a memory leak.
02-Apr-2021 10:12:42.038 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Abandoned connection cleanup thread] and has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
    com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    java.lang.Thread.run(Thread.java:748)

Why this happens

  • JDBC drivers (e.g., MySQL, H2, Druid) are registered by the webapp’s classloader and not cleaned up on webapp stop/redeploy, leaving strong references that Tomcat flags as potential leaks.
  • Background threads (notably MySQL’s AbandonedConnectionCleanupThread) are started by the application or JDBC driver and not terminated before the webapp classloader is discarded.
  • Multiple Tomcat proecsses may be running concurrently, causing inconsistent startup/shutdown behavior and stale resources.
  • Misconfigured memory settings in the JVM can exacerbate failures during classloading or bean initialization, surfacing as filter startup errors.

Remediation strategies

1) Ensure only one Tomcat process is running

  • Check for stray processes and terminate them before starting:
ps -ef | grep '[o]rg.apache.catalina.startup.Bootstrap'
# or, for a named instance
ps -ef | grep 'catalina.*your-instance'

# kill by PID once verified
kill -9 <PID>

2) Clean shutdown of JDBC drivers and MySQL cleanup thread

Add a ServletContextListener (or an ApplicationListener) to explicitly release resources on application stop:

import jakarta.servlet.ServletContextEvent;
import jakarta.servlet.ServletContextListener;
import java.sql.Driver;
import java.sql.DriverManager;
import java.util.Enumeration;

public class AppCleanupListener implements ServletContextListener {
    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        // Stop MySQL abandon cleanup thread (class name differs by driver version)
        try {
            // MySQL 8+
            Class<?> cls = Class.forName("com.mysql.cj.jdbc.AbandonedConnectionCleanupThread");
            cls.getMethod("checkedShutdown").invoke(null);
        } catch (Throwable ignore) {
            try {
                // MySQL 5.x
                Class<?> cls = Class.forName("com.mysql.jdbc.AbandonedConnectionCleanupThread");
                cls.getMethod("shutdown").invoke(null);
            } catch (Throwable ignored) { }
        }

        // Deregister JDBC drivers loaded by this webapp
        ClassLoader cl = Thread.currentThread().getContextClassLoader();
        Enumeration<Driver> drivers = DriverManager.getDrivers();
        while (drivers.hasMoreElements()) {
            Driver driver = drivers.nextElement();
            if (driver.getClass().getClassLoader() == cl) {
                try { DriverManager.deregisterDriver(driver); } catch (Exception ignored) { }
            }
        }

        // Ensure pools are closed (example: Druid)
        try {
            com.alibaba.druid.pool.DruidDataSourceStatManager.clearDataSources();
        } catch (Throwable ignored) { }
    }
}

Register the listener in web.xml or via Spring Boot’s ServletContextInitializer equivalent.

3) Verify data source lifecycle

  • Make sure connection pools are singletons and are closed during shutdown (e.g., implement DisposableBean in Spring or define a destroyMethod for the DataSource bean).
  • If multiple apps share a JDBC driver, consider placing the driver jar in TOMCAT_HOME/lib so the server classloader owns it, reducing per-webapp registration churn.

4) JVM memory configuration

Insufficient or misaligned heap/metadata sizes can fail classloading and filter initialization. Adjust according to your environment:

  • For legacy JDK 7 and earlier (PermGen):
JAVA_OPTS='-server -Xms5120m -Xmx10240m -XX:PermSize=256m -XX:MaxPermSize=5120m -XX:MaxNewSize=256m'
  • For JDK 8+ (Metaspace replaces PermGen):
JAVA_OPTS='-server -Xms4g -Xmx8g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1g'

Tune based on actual usage and monitoring.

5) Tomcat leak-prevention listener (not recommended to disable)

Tomcat’s leak-prevention logs issues that GC cannot fix automatically. If you must suppress this detection for debugging, comment the listener in conf/server.xml:

<!-- Use with caution: hides useful diagnostics -->
<!--
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/>
-->

Prefer fixing the underlying resource lifecycle instead of disabling detection.

6) Iterative isolation when the cause isn’t obvious

  1. Deploy a known-good baseline build that previously worked.
  2. If Tomcat starts cleanly, the container is healthy; the regression is in the application changes.
  3. Create a temporary branch from the baseline and migrate changes in small, testable increments.
  4. After each increment, redeploy. When the warning reappears, focus on the last change set and inspect resources started at init time (filters, listeners, schedulers, JDBC config).

Example field note: injection behavior in a security filter

An application filter created during startup referenced Spring services but exhibited unexpected wiring behavior depending on the injection annotation used. Inconsistent injection can result in early failures or background resources not being tied to the Spring context lifecycle.

Problematic setup (names changed):

import lombok.extern.slf4j.Slf4j;
import org.apache.shiro.web.filter.authc.FormAuthenticationFilter;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;

@Slf4j
@Component
public class ApiAuthFilter extends FormAuthenticationFilter {

    @Resource
    DeviceRegistryService deviceRegistryService;

    @Resource
    AccountService accountService;

    // ...
}

Service implementation:

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

import javax.annotation.Resource;

@Slf4j
@Service
public class DeviceRegistryServiceImpl implements DeviceRegistryService {

    @Resource
    CoreStorageService coreStorageService;

    // ...
}

If the filter is not created by Spring (e.g., instantiated by the web container or a sceurity framework), field injection may be inconsistent. Use constructor injection and ensure the filter itself is registered as a Spring-managed bean and wired into the filter chain via Spring:

import org.springframework.beans.factory.annotation.Autowired;

@Component
public class ApiAuthFilter extends FormAuthenticationFilter {
    private final DeviceRegistryService deviceRegistryService;
    private final AccountService accountService;

    @Autowired
    public ApiAuthFilter(DeviceRegistryService deviceRegistryService,
                         AccountService accountService) {
        this.deviceRegistryService = deviceRegistryService;
        this.accountService = accountService;
    }
}

Alternatively, if using @Resource, verify the bean names match or switch to @Autowired with constructor-based wiring to avoid nulls. Miswiring at this layer can surface as filter startup failures that cascade into the memory-leak warnings during redeploys.

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.