今天solr遇到的问题。

线上 15:32:36  搜索出现问题:

 

 

INFO: [feed] webapp=/solr path=/select params={fl=id&sort=last_replied_datetime+desc&start=0&q=(*)+_val_:"div(photo_size,+sum(product(photo_width,+pho
to_height),1))"&wt=standard&fq=category:9+OR+category:1&fq=photo_size:[1+TO+1048576]&fq=-pop_score:0&fq=-status:6&fq=-photo_id:0&fq=buyable:0+OR+buyab
le:11+OR+buyable:14&fq=add_datetime:[2012-07-03T13:46:06Z+TO+*]&fq=favorite_count:[1+TO+*]&rows=30&version=2.2} hits=5814 status=0 QTime=11342

 

Search .execute time is:11007ms
Search .execute time is:17759ms
Search .execute time is:17904ms
Search .execute time is:11955ms
Search .execute time is:11648ms
Search .execute time is:11958ms
Search .execute time is:19494ms
Search .execute time is:11346ms
Search .execute time is:19519ms
Search .execute time is:13948ms
Search .execute time is:8309ms
Search .execute time is:14111ms
Search .execute time is:14007ms
Search .execute time is:19588ms
Search .execute time is:8438ms
Search .execute time is:13767ms
Search .execute time is:11717ms
Search .execute time is:20133ms
Search .execute time is:9338ms

 

最开始以为是_val_:"div(photo_size,+sum(product(photo_width,+photo_height),1))"&wt=standardfq=category:9+OR+category:1

有问题，去掉。但后来问题仍然存在：

 

 

问题定位：

 

1. q = NOT id:0  fq=add_datetime:[2012-07-03T13:46:06Z+TO+*] 始终感觉有问题，带有NOT ID的执行时间：

 

tail -f catalina.2012-07-10.log | grep "feed" |grep  "NOT"|  awk -F 'QTime=' '{print $2}' > mytest
cat mytest |sort -rn | head -10

636
12
11
11
529
11
595
441
9
1
1
1
1
1
10
442
11
597
599
10
1
508
565
635
452
10
10
519
521

不带NOT ID的执行时间：

0
7
1
1
1
0
1
0
0
0
0
47
0
1
0
0
0
0
1
1
0
7
0
0
0
0
0
0
1
1
15
18
1
1
0

2. cache 出现了OOM，和内存管理有问题。

 

 

早上起来一看JVM：

[admin@server6 ~]$ jstat -gcutil 32336 5000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  
  0.00   0.00 100.00 100.00  25.95    206    3.489 15636 12242.959 12246.448
  0.00   0.00 100.00 100.00  25.95    206    3.489 15644 12249.039 12252.528
  0.00   0.00 100.00 100.00  25.95    206    3.489 15650 12254.227 12257.716

 

初步判断和documentCache有关。

<documentCache
                      class="solr.LRUCache"
                      size="40480"
                      initialSize="512"/>