前段时间公司项目运行一段时间 cpu 就占用100%,然后服务就不可用了, 但是那段时间并发也没有升高,数据库,缓存也很正常,弄了很久都没有头绪。于是领导让我来解决这个问题。
登陆服务器 先用top 命令查看cpu 占用
1top
发现 java 进程确实占用cpu 很高,继续查看java 内线程的cpu 占用
1top -H -p 4536
找到两个线程占用cpu 很高,然后打印java 程序的线程,找到这两个线程的信息(线程号需要转成16进制)
1jstack 4536> 4536.txt
发现这两个线程都是 gc 操作。有可能是jvm 内存占用满了,频繁full gc 导致的。
下载 阿里的 arthas 来看一下gc 的情况
https://github.com/alibaba/arthas/blob/master/README_CN.md
发现老年代都占用100%了,果然是频繁的full gc 导致的。而且full gc 都无法降低老年代的内存
打印一下 堆的信息
1
2jmap -heap 4536 jstack 4536> 1.log
实时打印gc 情况
1jstat -gc 4536 3000
手动fullgc 也不行
1jmap -histo:live 4536
发现有一个user对象占用很多,比较诡异,然后把堆的dump 对象下载下来,在本地用mat 查看
1jmap -dump:live,format=b,file=4536 4536
https://www.eclipse.org/mat/
右键char[] 打开char[],查看里面的对象
发现不仅user 对象比较多,而且char[] 里面有一个 对象也比较多,带有cookie 和 deleteMe 字段,这个是shiro 框架里面的内容。 我们使用了 shiro 和shiro-redis 作为权限控制框架。 有可能是这两个框架导致的。最后在网上找的果然是shiro-redis 导致的问题
http://www.findsrc.com/java/detail/8688
我们在github找到框架的源码
https://github.com/alexxiyang/shiro-redis
发现果然用到了threadlocal,作者的说法是shiro 在鉴权的时候频繁读取session 信息,所以用到了threadlocal,减小redis 的读取压力。所以可能就是这个原因导致的内存泄漏。
1
2
3
4
5
6/** * doReadSession be called about 10 times when login. * Save Session in ThreadLocal to resolve this problem. sessionInMemoryTimeout is expiration of Session in ThreadLocal. * The default value is 1000 milliseconds (1s). * Most of time, you don't need to change it. */
最终解决办法。
升级shiro-redis 至 3.2.3
shiroconfig在使用redissessiondao 时禁用threadLocal
1
2
3
4
5
6
7
8
9@Bean public RedisSessionDAO sessionDAO(){ RedisSessionDAO sessionDAO = new RedisSessionDAO(); // crazycake 实现 sessionDAO.setSessionInMemoryEnabled(false); sessionDAO.setRedisManager(redisManager()); sessionDAO.setSessionIdGenerator(sessionIdGenerator()); // Session ID 生成器 return sessionDAO; }
以下为shiro-redis源码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283package org.crazycake.shiro; import org.apache.shiro.session.Session; import org.apache.shiro.session.UnknownSessionException; import org.apache.shiro.session.mgt.eis.AbstractSessionDAO; import org.crazycake.shiro.exception.SerializationException; import org.crazycake.shiro.serializer.ObjectSerializer; import org.crazycake.shiro.serializer.RedisSerializer; import org.crazycake.shiro.serializer.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.Serializable; import java.util.*; public class RedisSessionDAO extends AbstractSessionDAO { private static Logger logger = LoggerFactory.getLogger(RedisSessionDAO.class); private static final String DEFAULT_SESSION_KEY_PREFIX = "shiro:session:"; private String keyPrefix = DEFAULT_SESSION_KEY_PREFIX; private static final long DEFAULT_SESSION_IN_MEMORY_TIMEOUT = 1000L; /** * doReadSession be called about 10 times when login. * Save Session in ThreadLocal to resolve this problem. sessionInMemoryTimeout is expiration of Session in ThreadLocal. * The default value is 1000 milliseconds (1s). * Most of time, you don't need to change it. */ private long sessionInMemoryTimeout = DEFAULT_SESSION_IN_MEMORY_TIMEOUT; private static final boolean DEFAULT_SESSION_IN_MEMORY_ENABLED = true; private boolean sessionInMemoryEnabled = DEFAULT_SESSION_IN_MEMORY_ENABLED; // expire time in seconds private static final int DEFAULT_EXPIRE = -2; private static final int NO_EXPIRE = -1; /** * Please make sure expire is longer than sesion.getTimeout() */ private int expire = DEFAULT_EXPIRE; private static final int MILLISECONDS_IN_A_SECOND = 1000; private IRedisManager redisManager; private RedisSerializer keySerializer = new StringSerializer(); private RedisSerializer valueSerializer = new ObjectSerializer(); private static ThreadLocal sessionsInThread = new ThreadLocal(); @Override public void update(Session session) throws UnknownSessionException { this.saveSession(session); if (this.sessionInMemoryEnabled) { this.setSessionToThreadLocal(session.getId(), session); } } /** * save session * @param session * @throws UnknownSessionException */ private void saveSession(Session session) throws UnknownSessionException { if (session == null || session.getId() == null) { logger.error("session or session id is null"); throw new UnknownSessionException("session or session id is null"); } byte[] key; byte[] value; try { key = keySerializer.serialize(getRedisSessionKey(session.getId())); value = valueSerializer.serialize(session); } catch (SerializationException e) { logger.error("serialize session error. session id=" + session.getId()); throw new UnknownSessionException(e); } if (expire == DEFAULT_EXPIRE) { this.redisManager.set(key, value, (int) (session.getTimeout() / MILLISECONDS_IN_A_SECOND)); return; } if (expire != NO_EXPIRE && expire * MILLISECONDS_IN_A_SECOND < session.getTimeout()) { logger.warn("Redis session expire time: " + (expire * MILLISECONDS_IN_A_SECOND) + " is less than Session timeout: " + session.getTimeout() + " . It may cause some problems."); } this.redisManager.set(key, value, expire); } @Override public void delete(Session session) { if (session == null || session.getId() == null) { logger.error("session or session id is null"); return; } try { redisManager.del(keySerializer.serialize(getRedisSessionKey(session.getId()))); } catch (SerializationException e) { logger.error("delete session error. session id=" + session.getId()); } } @Override public Collection<Session> getActiveSessions() { Set<Session> sessions = new HashSet<Session>(); try { Set<byte[]> keys = redisManager.keys(this.keySerializer.serialize(this.keyPrefix + "*")); if (keys != null && keys.size() > 0) { for (byte[] key:keys) { Session s = (Session) valueSerializer.deserialize(redisManager.get(key)); sessions.add(s); } } } catch (SerializationException e) { logger.error("get active sessions error."); } return sessions; } @Override protected Serializable doCreate(Session session) { if (session == null) { logger.error("session is null"); throw new UnknownSessionException("session is null"); } Serializable sessionId = this.generateSessionId(session); this.assignSessionId(session, sessionId); this.saveSession(session); return sessionId; } @Override protected Session doReadSession(Serializable sessionId) { if (sessionId == null) { logger.warn("session id is null"); return null; } if (this.sessionInMemoryEnabled) { Session session = getSessionFromThreadLocal(sessionId); if (session != null) { return session; } } Session session = null; logger.debug("read session from redis"); try { session = (Session) valueSerializer.deserialize(redisManager.get(keySerializer.serialize(getRedisSessionKey(sessionId)))); if (this.sessionInMemoryEnabled) { setSessionToThreadLocal(sessionId, session); } } catch (SerializationException e) { logger.error("read session error. settionId=" + sessionId); } return session; } private void setSessionToThreadLocal(Serializable sessionId, Session s) { Map<Serializable, SessionInMemory> sessionMap = (Map<Serializable, SessionInMemory>) sessionsInThread.get(); if (sessionMap == null) { sessionMap = new HashMap<Serializable, SessionInMemory>(); sessionsInThread.set(sessionMap); } removeExpiredSessionInMemory(sessionMap); SessionInMemory sessionInMemory = new SessionInMemory(); sessionInMemory.setCreateTime(new Date()); sessionInMemory.setSession(s); sessionMap.put(sessionId, sessionInMemory); } private void removeExpiredSessionInMemory(Map<Serializable, SessionInMemory> sessionMap) { Iterator<Serializable> it = sessionMap.keySet().iterator(); while (it.hasNext()) { Serializable sessionId = it.next(); SessionInMemory sessionInMemory = sessionMap.get(sessionId); if (sessionInMemory == null) { it.remove(); continue; } long liveTime = getSessionInMemoryLiveTime(sessionInMemory); if (liveTime > sessionInMemoryTimeout) { it.remove(); } } } private Session getSessionFromThreadLocal(Serializable sessionId) { if (sessionsInThread.get() == null) { return null; } Map<Serializable, SessionInMemory> sessionMap = (Map<Serializable, SessionInMemory>) sessionsInThread.get(); SessionInMemory sessionInMemory = sessionMap.get(sessionId); if (sessionInMemory == null) { return null; } long liveTime = getSessionInMemoryLiveTime(sessionInMemory); if (liveTime > sessionInMemoryTimeout) { sessionMap.remove(sessionId); return null; } logger.debug("read session from memory"); return sessionInMemory.getSession(); } private long getSessionInMemoryLiveTime(SessionInMemory sessionInMemory) { Date now = new Date(); return now.getTime() - sessionInMemory.getCreateTime().getTime(); } private String getRedisSessionKey(Serializable sessionId) { return this.keyPrefix + sessionId; } public IRedisManager getRedisManager() { return redisManager; } public void setRedisManager(IRedisManager redisManager) { this.redisManager = redisManager; } public String getKeyPrefix() { return keyPrefix; } public void setKeyPrefix(String keyPrefix) { this.keyPrefix = keyPrefix; } public RedisSerializer getKeySerializer() { return keySerializer; } public void setKeySerializer(RedisSerializer keySerializer) { this.keySerializer = keySerializer; } public RedisSerializer getValueSerializer() { return valueSerializer; } public void setValueSerializer(RedisSerializer valueSerializer) { this.valueSerializer = valueSerializer; } public long getSessionInMemoryTimeout() { return sessionInMemoryTimeout; } public void setSessionInMemoryTimeout(long sessionInMemoryTimeout) { this.sessionInMemoryTimeout = sessionInMemoryTimeout; } public int getExpire() { return expire; } public void setExpire(int expire) { this.expire = expire; } public boolean getSessionInMemoryEnabled() { return sessionInMemoryEnabled; } public void setSessionInMemoryEnabled(boolean sessionInMemoryEnabled) { this.sessionInMemoryEnabled = sessionInMemoryEnabled; } public static ThreadLocal getSessionsInThread() { return sessionsInThread; } }
最后
以上就是俏皮指甲油最近收集整理的关于一次java 内存泄漏问题的解决过程的全部内容,更多相关一次java内容请搜索靠谱客的其他文章。
发表评论 取消回复