common-pool剖析

common-pool是什么

apache commons-pool是apache基金会的一个开源对象池组件，常用的数据库连接池dpcp和redis的java客户端jedis都是基于commons-pool实现的。

什么是对象池

单例模式 singleton：全局只有一个对象，供所有线程使用。

原型模式 propotype：每次使用对象时都新建，用完之后释放。

对象池模式 pool：一个池子，里面 维护 多个对象。调用方从对象池 借取 对象，使用之后 归还 给对象池。

使用对象池的利与弊

弊：

1. 增加复杂度。正常对象的生命周期由使用者和JVM维护： 使用者创建（JVM分配内存） → 销毁 → GC。对象池对象的生命周期更复杂，需要引入多个状态。 IDLE（空闲）、ALLOCATED（分配）、EVICTION（驱逐）、VALIDATION（验证）、INVALID（无效）、RETURNING（返回中）；
2. 多线程资源竞争。common-pool内部实现大量使用了synchronized以及其他juc原子类，可能会造成锁竞争，影响程序性能；
3. 参数配置失误可能会带来严重问题。

利：

1. 复用池中对象,消除创建对象、回收对象 所产生的内存开销、cpu开销以及(若跨网络)产生的网络开销common-pool参数配置详解

适用场景

1.  受限的, 不需要可伸缩性的环境(cpu\内存等物理资源有限): cpu性能不够强劲, 内存比较紧张, 垃圾收集, 内存抖动会造成比较大的影响, 需要提高内存管理效率, 响应性比吞吐量更为重要;
2. 数量受限的资源, 比如数据库连接;
3. 创建成本高昂的对象。

common-pool核心类

1. PooledObjectFactory【对象工厂】，用于产生一个新对象

public interface PooledObjectFactory<T> {     /**      * 产生一个池对象      */     PooledObject<T> makeObject() throws Exception;        /**      * 销毁一个池对象      */     void destroyObject(PooledObject<T> p) throws Exception;        /**      * 校验对象是否可用      */     boolean validateObject(PooledObject<T> p);        /**      * 激活一个对象      */     void activateObject(PooledObject<T> p) throws Exception;        /**      * 钝化一个对象      */     void passivateObject(PooledObject<T> p) throws Exception; }

2. ObjectPool【链接池】，用于存放链接对象的一个池子，提供接口用于对象的借取和归还。

2.1 GenericObjectPool【ObjectPool的具体实现类】，这就是包里用来作为默认的链接池对象的实现类

GenericObjectPool中主要的方法包括borrowObject，returnObject，create， clear，close，destroy，ensureIdle，evict等。

borrowObject： 用于从对象池中取出一个对象；

returnObject：把取出来的对象归还给对象池；

create：调用PooledObjectFactory的makeObject方法新建对象

clear：清空对象池中的对象；

close：停止驱逐任务运行，清空对象池对象，注销对象池的Mbean，中断当前等待从池中获取对象的线程；

destroy：空闲对象队列中remove指定对象，并且全部对象队列中remove该对象，销毁对象数量+1，新建对象-1。PooledObjectFactory的实现类destroy具体的对象；

ensureIdle：确保有充足的空闲对象；

evict：驱逐无效的对象。

2.2 BaseObjectPoolConfig【链接池主要配置】，提供了链接池的主要配置，例如进出策略，最大链接，空闲链接，超时时间，回收策略等等一系列丰富的对于链接池的管理。

详见common-pool参数配置详解

2.3 AbandonedConfig【丢弃策略】，提供了针对链接池内链接的丢弃策略，主要防止内存泄漏

3. PooledObject【链接对象】，链接池内的对象。

common-pool参数配置详解

GenericObjectPoolConfig

参数名	类型	描述	默认值
maxTotal	int	最大对象数	8
maxIdle	int	最大空闲对象数	8
minIdle	int	最小空闲对象数	0
lifo	boolean	双端队列中空闲对象是否后进先出	true
fairness	boolean	队列锁使用的是否公平锁	false
maxWaitDuration	Duration	最大等待时间 -1L
minEvictableIdleDuration	Duration	最小空闲驱逐时间  1000L * 60L * 30L
evictorShutdownTimeoutDuration	Duration	驱逐停止超时时间  10L * 1000L
softMinEvictableIdleDuration	Duration	最小安全驱逐空闲时间 -1L
numTestsPerEvictionRun	int	每次驱逐运行的测试数量	3
evictionPolicy	EvictionPolicy<T>	驱逐策略	DefaultEvictionPolicy
evictionPolicyClassName	String	驱逐策略名称	org.apache.commons.pool2.impl.DefaultEvictionPolicy
testOnCreate	boolean	创建池对象时检测池对象是否有效，无效则剔除，并尝试继续获取新池对象	false
testOnBorrow	boolean	取出池对象时检测池对象是否有效，无效则剔除，并尝试继续获取新池对象	false
testOnReturn	boolean	归还对象时检测对象是否有效，无效则不归还	false
testWhileIdle	boolean	检测是否空闲，空闲则调用evict方法驱逐对象	false
durationBetweenEvictionRuns	Duration	空闲对象驱除任务间隔时间 -1L
blockWhenExhausted	boolean	值【true】maxWaitMillis生效，【false】对象池池没资源时，立马抛异常	true
jmxEnabled	boolean	是否开启Java管理扩展功能	true
jmxNamePrefix	String	jmx名称前缀	”pool“
jmxNameBase	String	jmx名称base	null

common-pool实现SFTP连接池demo

SFTPProperties 配置类

public class SFTPProperties {     private String username;     private String password;     private String privateKey;     private String host;     private Integer port;     private Integer maxTotal;     private Integer maxIdle;     private Integer minIdle;     ... }

ChannelSftpFactory SFTP连接池工厂

channelSftp.getSession().disconnect() 如果不关闭连接将一直占用

public class ChannelSftpFactory extends BasePooledObjectFactory<ChannelSftp> {     private SFTPProperties properties;     public ChannelSftpFactory(SFTPProperties properties) {         this.properties = properties;     }     public ChannelSftp create() {         return SFTPOperationHelper.login(this.properties);     }     public PooledObject<ChannelSftp> wrap(ChannelSftp channelSftp) {         return new DefaultPooledObject(channelSftp);     }     public void destroyObject(PooledObject<ChannelSftp> p) {         ChannelSftp channelSftp = (ChannelSftp)p.getObject();         channelSftp.disconnect();         try {             channelSftp.getSession().disconnect();         } catch (JSchException e) {             e.printStackTrace();         }     }     ... }

ChannelSftpPool SFTP连接池

public class ChannelSftpPool extends GenericObjectPool<ChannelSftp> {     private String name;     public ChannelSftpPool(PooledObjectFactory<ChannelSftp> factory) {         super(factory);     }     public ChannelSftpPool(PooledObjectFactory<ChannelSftp> factory, GenericObjectPoolConfig config) {         super(factory, config);     }     ... }

ChannelSftpPoolHelper SFTP连接池操作辅助类

public class ChannelSftpPoolHelper {     private static Map<String, ChannelSftpPool> customerSFTPPools = new ConcurrentHashMap();     private static final Integer DEFAULT_MAX_TOTAL = 5;     private static final Integer DEFAULT_MAX_IDLE = 4;     private static final Integer DEFAULT_MIN_IDLE = 1;     private static final Long IDLE_EVICT_TIME = 20000L;     private static final Integer MAX_WAIT_TIME = 3000;     public ChannelSftpPoolHelper() {     }     /**      * 新增连接池      * @param userName      * @param sftpPool      */     public static void addPool(String userName, ChannelSftpPool sftpPool) {         customerSFTPPools.put(userName, sftpPool);     }     /**      * 获取连接池      * @param userName      * @return      */     public static ChannelSftpPool getPool(String userName) {         return (ChannelSftpPool)customerSFTPPools.get(userName);     }     /**      * 获取连接池如果不存在则新建      * @param properties      * @return      */     public static ChannelSftpPool getAndCreatePool(SFTPProperties properties) {         String userName = properties.getUsername();         ChannelSftpPool pool = getPool(userName);         if (pool == null) {             pool = createPool(properties);             if (pool != null) {                 customerSFTPPools.put(userName, pool);             }         }         return pool;     }     /**      * 从连接池中获取一个对象      * @param pool      * @return      */     public static ChannelSftp borrowObject(ChannelSftpPool pool) {         try {             if (pool != null) {                 ChannelSftp channelSftp = (ChannelSftp)pool.borrowObject();                 if (channelSftp.isClosed()) {                     pool.invalidateObject(channelSftp);                     channelSftp = (ChannelSftp)pool.borrowObject();                 }                 return channelSftp;             }         } catch (Exception var2) {             System.err.println("连接池获取异常");         }         return null;     }     /**      * 从连接池中获取一个对象      * @param userName      * @return      */     public static ChannelSftp borrowObject(String userName) {         ChannelSftpPool pool = getPool(userName);         return borrowObject(pool);     }     /**      * 从连接池中获取一个对象      * @param properties      * @return      */     public static ChannelSftp borrowObject(SFTPProperties properties) {         ChannelSftpPool pool = getAndCreatePool(properties);         return borrowObject(pool);     }     /**      * 归还对象给连接池      * @param userName      * @param channelSftp      */     public static void returnObject(String userName, ChannelSftp channelSftp) {         if (channelSftp != null) {             ChannelSftpPool pool = getPool(userName);             pool.returnObject(channelSftp);         }     }     /**      * 新建连接池      * @param properties      * @return      */     public static ChannelSftpPool createPool(SFTPProperties properties) {         ChannelSftpFactory factory = new ChannelSftpFactory(properties);         return createPool(properties, factory);     }     /**      * 新建连接池      * @param sftpProperties      * @param channelSftpFactory      * @return      */     public static ChannelSftpPool createPool(SFTPProperties sftpProperties, ChannelSftpFactory channelSftpFactory) {         GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();         Integer maxIdle = (Integer)Optional.ofNullable(sftpProperties.getMaxIdle()).orElse(DEFAULT_MAX_IDLE);         poolConfig.setMaxIdle(maxIdle);         Integer minIdle = (Integer)Optional.ofNullable(sftpProperties.getMinIdle()).orElse(DEFAULT_MIN_IDLE);         poolConfig.setMinIdle(minIdle);         Integer maxTotal = (Integer)Optional.ofNullable(sftpProperties.getMaxTotal()).orElse(DEFAULT_MAX_TOTAL);         poolConfig.setMaxTotal(maxTotal);         poolConfig.setBlockWhenExhausted(true);         poolConfig.setTestOnCreate(true);         poolConfig.setTestOnBorrow(false);         poolConfig.setTestOnReturn(true);         poolConfig.setTestWhileIdle(true);         poolConfig.setTimeBetweenEvictionRuns(Duration.ofMillis(5000L));         poolConfig.setMinEvictableIdleTime(Duration.ofMillis(1000L));         poolConfig.setMaxWait(Duration.ofMillis((long)MAX_WAIT_TIME));         poolConfig.setJmxEnabled(false);         ChannelSftpPool pool = new ChannelSftpPool(channelSftpFactory, poolConfig);         pool.setName(sftpProperties.getUsername());         addPool(sftpProperties.getUsername(), pool);         return pool;     }     /**      * 获取所有连接池      * @return      */     public static Map<String, ChannelSftpPool> getPools() {         return customerSFTPPools;     } }

SFTPOperationHelper SFTP操作辅助父类  

public class SFTPOperationHelper {     public SFTPOperationHelper() {     }     /**      * 登录SFTP      * @param properties      * @return      */     public static ChannelSftp login(SFTPProperties properties) {         try {             JSch jsch = new JSch();             if (StringUtils.hasText(properties.getPrivateKey())) {                 jsch.addIdentity("idname", properties.getPrivateKey().getBytes(), (byte[])null, (byte[])null);             }             Session sshSession = jsch.getSession(properties.getUsername(), properties.getHost(), properties.getPort());             if (StringUtils.hasText(properties.getPassword())) {                 sshSession.setPassword(properties.getPassword());             }             Properties sshConfig = new Properties();             sshConfig.put("StrictHostKeyChecking", "no");             sshConfig.put("PreferredAuthentications", "publickey,keyboard-interactive,password");             sshSession.setConfig(sshConfig);             sshSession.connect();             ChannelSftp channel = (ChannelSftp)sshSession.openChannel("sftp");             channel.connect();             return channel;         } catch (JSchException var5) {             System.err.println("连接SFTP服务器={}异常");             return null;         }     }     /**      * 下载成InputStream      * @param dir 目录      * @param name 文件名      * @param sftp      * @return      */     public static InputStream downloadToStream(String dir, String name, ChannelSftp sftp) {         try {             sftp.cd(dir);             return sftp.get(name);         } catch (SftpException var4) {             System.err.println("SFTP下载文件={}{}异常");             return null;         }     }     /**      * 下载成byte      * @param dir 目录      * @param name 文件名      * @param sftp      * @return      */     public static byte[] downloadToBytes(String dir, String name, ChannelSftp sftp) {         try {             sftp.cd(dir);             InputStream inputStream = sftp.get(name);             return IOUtils.toByteArray(inputStream);         } catch (IOException | SftpException var4) {             System.err.println("SFTP下载文件={}{}异常");             return null;         }     }     /**      * 上传      * @param dir 目录      * @param name 文件名      * @param in      * @param sftp sftp      * @throws SftpException      */     public static void upload(String dir, String name, InputStream in, ChannelSftp sftp) throws SftpException {         mkdirs(dir, sftp);         sftp.cd(dir);         sftp.put(in, name);     }     /**      * 删除      * @param dir      * @param name      * @param sftp      * @throws SftpException      */     public static void delete(String dir, String name, ChannelSftp sftp) throws SftpException {         sftp.cd(dir);         sftp.rm(name);     }     /**      * 新建目录      * @param dir      * @param sftp      * @throws SftpException      */     public static void mkdirs(String dir, ChannelSftp sftp) throws SftpException {         String[] folders = dir.split("/");         sftp.cd("/");         String[] var3 = folders;         int var4 = folders.length;         for(int var5 = 0; var5 < var4; ++var5) {             String folder = var3[var5];             if (folder.length() > 0) {                 try {                     sftp.cd(folder);                 } catch (Exception var8) {                     sftp.mkdir(folder);                     sftp.cd(folder);                 }             }         }     } }

SFTPPoolOperationHelper SFTP操作辅助子类 

public class SFTPPoolOperationHelper extends SFTPOperationHelper {     public SFTPPoolOperationHelper() {     }     /**      * 下载文件byte      * @param dir      * @param name      * @param pool      * @return      * @throws Exception      */     public static byte[] downloadToBytes(String dir, String name, ChannelSftpPool pool) throws Exception {         ChannelSftp sftp = ChannelSftpPoolHelper.borrowObject(pool);         byte[] var4;         try {             var4 = downloadToBytes(dir, name, sftp);         } finally {             pool.returnObject(sftp);         }         return var4;     }     /**      * 上传文件      * @param dir      * @param name      * @param in      * @param pool      * @throws Exception      */     public static void upload(String dir, String name, InputStream in, ChannelSftpPool pool) throws Exception {         ChannelSftp sftp = (ChannelSftp)pool.borrowObject();         try {             upload(dir, name, in, sftp);         } finally {             pool.returnObject(sftp);         }     }     /**      * 删除文件      * @param dir      * @param name      * @param pool      * @throws Exception      */     public static void delete(String dir, String name, ChannelSftpPool pool) throws Exception {         ChannelSftp sftp = (ChannelSftp)pool.borrowObject();         try {             delete(dir, name, sftp);         } finally {             pool.returnObject(sftp);         }     }     /**      * 新建目录      * @param dir      * @param pool      * @throws Exception      */     private static void mkdirs(String dir, ChannelSftpPool pool) throws Exception {         ChannelSftp sftp = (ChannelSftp)pool.borrowObject();         try {             mkdirs(dir, sftp);         } finally {             pool.returnObject(sftp);         }     } }

DmpCrowdFtpUtils SFTP util类 

public class DmpCrowdFtpUtils {     private final static String FTP_USER_NAME = "squidxu";     private final static String FTP_PASSWORD = "squidyu.1";     private final static String FTP_HOST = "10.1.18.75";     private final static Integer FTP_PORT = 22;     /**      * 文件分隔符      */     private static final String SEPARATOR = "/";     /**      * 获取ftp文件路径      *      * @param taskId      */     public static String getFtpTargetPath(Integer taskId) {         return SEPARATOR + "download" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + taskId + SEPARATOR; //        return SEPARATOR + "in" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + date + SEPARATOR + taskId + SEPARATOR;     }     /**      * 将本地文件上传至ftp文件服务器      *      * @param file      * @param taskId      * @param fileName      * @throws Exception      */     public static String uploadFile(File file, Integer taskId, String fileName) throws Exception {         String ftpTargetPath = getFtpTargetPath(taskId);         InputStream content = new FileInputStream(file);         ChannelSftpPool channelSftpPool = getChannelSftpPool();         SFTPPoolOperationHelper.upload(ftpTargetPath, fileName, content, channelSftpPool);         return ftpTargetPath + fileName;     }     public static void main(String[] args) {         for (int i = 0; i < 10; i++) {             File file = new File("C:\\Users\\squidxu\\Desktop\\周报模板.xlsx");             try {                 uploadFile(file, 1000 + i, "周报模板" + i + ".xlsx");             } catch (Exception e) {                 e.printStackTrace();             }         }     }     /**      * 从ftp下载文件      *      * @param fullPath      * @return      */     public static InputStream downloadStream(String fullPath) throws Exception {         return new ByteArrayInputStream(downloadFile(fullPath));     }     /**      *      * 从fastdfs下载文件      * @param path      * @param localPath      * @return      */     public static void download2Local(String path, String localPath) throws Exception {         BufferedReader buffReader = null;         BufferedWriter buffWriter = null;         try {             buffReader = new BufferedReader(new InputStreamReader(downloadStream(path)));             buffWriter = new BufferedWriter(new OutputStreamWriter(                     new FileOutputStream(localPath,true)));             String line = null;             while ((line = buffReader.readLine()) != null) {                 buffWriter.write(line + "\n");             }         } finally {             flush(buffWriter);             close(buffReader);             close(buffWriter);         }     }     private static void flush(BufferedWriter bufferedWriter) {         if (bufferedWriter != null) {             try {                 bufferedWriter.flush();             } catch (IOException e) {                 System.err.println("close error, message={}"+ e.getMessage());             }         }     }     /**      * 从ftp下载文件      *      * @param fullPath      * @return      */     public static byte[] downloadFile(String fullPath) throws Exception {         String ftpTargetPath = StringUtils.substringBeforeLast(fullPath, SEPARATOR) + SEPARATOR;         String ftpFileName = StringUtils.substringAfterLast(fullPath, SEPARATOR);         long start = System.currentTimeMillis();         ChannelSftpPool channelSftpPool = getChannelSftpPool();         System.out.println("channelSftpPool:" + channelSftpPool.hashCode());         long end = System.currentTimeMillis();         System.out.println("getChannelSftpPool" + (end - start) + "ms");         return SFTPPoolOperationHelper.downloadToBytes(ftpTargetPath, ftpFileName, channelSftpPool);     }     /**      * 获取pool      *      * @param      * @return      */     private static ChannelSftpPool getChannelSftpPool() {         SFTPProperties properties = new SFTPProperties();         properties.setUsername(FTP_USER_NAME);         properties.setPassword(FTP_PASSWORD);         properties.setHost(FTP_HOST);         properties.setPort(FTP_PORT);         properties.setMaxTotal(5);         properties.setMaxIdle(1);         properties.setMinIdle(1);         final ChannelSftpPool pool = ChannelSftpPoolHelper.getAndCreatePool(properties);         System.out.println("object size" + pool.getNumActive());         System.out.println("MaxIdle size" + pool.getMaxIdle());         System.out.println("MaxIdle NumIdle" + pool.getNumIdle());         if (pool == null) {             throw new RuntimeException("获取pool错误");         }         return pool;     }     /**      * 关闭流      *      * @param      * @return      */     private static void close(BufferedReader reader) throws IOException {         if (reader != null) {             reader.close();         }     }     /**      * 关闭流      *      * @param      * @return      */     private static void close(BufferedWriter writer) throws IOException {         if (writer != null) {             writer.close();         }     } }