全文翻译太累了，大概的说一下文章的内容吧：
1.Oracle 11gR2的grid增加了许多特性，其中的一个是OLR，主要存放本地资源配置的信息，功能类似rac中的OCR
2. Oracle 11gR2 ASM的spfile存放在asm disks上，而没有存放在操作系统文件系统的dbs目录下
3.asm disks上的内容需要在asm实例启动并挂载了diskgroup之后才能被读取，而asm实例的启动是需要读取spfile的，这样就出现了鸡生蛋、蛋生鸡的问题
4.为了能在asm实例启动前获取spfile的内容，oracle启动asm实例是采取直接读取asm disks内容的方式获取spfile内容
5.如果asm的安装使用了asmlib，则可以直接读取asm disks的内容，因为asm disks的路径是定死的 /dev/oracleasm/disks/* ，应该是硬编码的东西
6.如果asm的安装没有使用asmlib，则oracle不知道asm disks的位置，不能直接读取，这时它会从olr中获取asm disks的信息，最终还是会读取asm disks中spfile的内容
Oracle 11gR2 have many new futures in Grid Computing area. One of them is Oracle Local Repository (OLR), this repository designed to store information and profiles for local resources, resources that dedicated to particular node. It improves the performance of accessing local resources profile information, redundancy and manageability. In Grid Infrastructure RAC configuration there is usual one global shared OCR and OLR’s on each node. In Oracle Restart environment there is only OLR repository. In 11g R2 there is also new feature, Grid Plug and Play (GPNP), as the name implies, it helps to automate and simplify some of the aspects of grid administration. GPNP maintains profile, it is XML file that stores the configuration information of some components maintained by GPNP, for example vips and interconnect information is stored here.
But in this post, I wont to discuss another new feature of 11g R2, it is the possibility to store the server parameter file (spfile) on the ASM disks, also for ASM instance itself! And there is some reasonable questions arise:
- How process can read the spfile from ASM disks, if they are not mounted yet
- How process knows where to get spfile if there is no init.ora file in $ORACLE_HOME/dbs with spfile=[path to spfile] parameter
- What about asm_diskstring parameter?
Third point is important for ASM instance because, before reading spfile from ASM disks, we need to identify them, to identify them we need spfile to get asm_diskstring parameter!
To make clear all this points I will use system calls tracing tools, to trace oracle processes. We can use tools like Dtrace, strace, truss and so on, it depends on platform, I will use strace because I am on Linux.
My environment is Oracle Grid Infrastructure 11g R2 (11.2.0.1.0) in Oracle Restart mode. I simulated ASM disks using loop devices, /dev/loop*, I have 5 disks
bash> /etc/init.d/oracleasm listdisks DISK1 DISK2 DISK3 DISK4 DISK5 SQL> conn / as sysasm Connected. SQL> col path format a15 SQL> select a.name,a.state,b.name,b.path from v$asm_diskgroup a, v$asm_disk b where a.group_number=b.group_number order by b.name; NAME STATE NAME PATH ------------------------------ ----------- ------------------------------ --------------- DGROUP1 MOUNTED DISK1 ORCL:DISK1 DGROUP1 MOUNTED DISK2 ORCL:DISK2 DGROUP2 MOUNTED DISK3 ORCL:DISK3 DGROUP2 MOUNTED DISK4 ORCL:DISK4 DGROUP2 MOUNTED DISK5 ORCL:DISK5
As seen from the listing, I have two disk groups DGROUP1 and DGROUP2. My spfile is located on DGROUP1
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ----------------------------------------------------
spfile string +DGROUP1/asm/asmparameterfile/registry.253.740659373
Lets make some tests
SQL> conn / as sysasm Connected. SQL> startup ASM instance started Total System Global Area 284565504 bytes Fixed Size 1336036 bytes Variable Size 258063644 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> SQL> show parameter power NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_power_limit integer 1 SQL> alter system set asm_power_limit=3; System altered. SQL> show parameter power NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_power_limit integer 3 SQL>--We set asm_power_limit to 3 in spfile SQL> shutdown immediate; ASM diskgroups dismounted ASM instance shutdown SQL> !crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DGROUP1.dg ora....up.type OFFLINE OFFLINE ora.DGROUP2.dg ora....up.type OFFLINE OFFLINE ora....ER.lsnr ora....er.type ONLINE ONLINE testdb03 ora.asm ora.asm.type OFFLINE OFFLINE ora.cssd ora.cssd.type ONLINE ONLINE testdb03 ora.diskmon ora....on.type ONLINE ONLINE testdb03 SQL>-- Lets start instance in nomount mode, it will not mount the diskgroups SQL> startup nomount; ASM instance started Total System Global Area 284565504 bytes Fixed Size 1336036 bytes Variable Size 258063644 bytes ASM Cache 25165824 bytes SQL> show parameter spfile; NAME TYPE VALUE ------------------------------------ ----------- ---------------------------------------------------- spfile string +DGROUP1/asm/asmparameterfile/registry.253.740659373 SQL> show parameter power NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_power_limit integer 3 SQL> select * from v$spparameter; select * from v$spparameter * ERROR at line 1: ORA-15001: diskgroup "DGROUP1" does not exist or is not mounted SQL> alter system set asm_power_limit=10; alter system set asm_power_limit=10 * ERROR at line 1: ORA-32000: write to SPFILE requested but SPFILE is not modifiable SQL> !asmcmd ASMCMD> ls ASMCMD> exit SQL> !crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DGROUP1.dg ora....up.type OFFLINE OFFLINE ora.DGROUP2.dg ora....up.type OFFLINE OFFLINE ora....ER.lsnr ora....er.type ONLINE ONLINE testdb03 ora.asm ora.asm.type ONLINE ONLINE testdb03 ora.cssd ora.cssd.type ONLINE ONLINE testdb03 ora.diskmon ora....on.type ONLINE ONLINE testdb03
So, what we see, we were able to start ASM instance without mounting the disks, our process picked up correct spfile. The command SHOW PARAMETER reads the parameter values from the memory, but if we try to read directly from spfile (select from v$spparameter) or write to it (alter system set) we will get errors, because diskgroups not mounted yet. It means that our process was read spfile directly from ASM disks. The design about from which disk to read it made based on information from disk header. I use ASMlib, and my asm_diskstring parameter is equal to default value null, ASM instance by default scans the /dev/oracleasm/disks/* on Linux, that is why process found my disks properly. But what if we use HP-UX and our disks is multipathing disks in /dev/rdisk/*, ASM will not scan them by default, we need to use asm_diskstring parameter, how our process will read asm_diskstring parameter before accessing spfile?
Lets start sqlplus, after we will connect as sysasm it will initialize server process, this process will do all job, that is why I will trace this server process
bash> sqlplus /nolog SQL> conn / as sysasm Connected. SQL> !ps -aefH . . oracle 15505 4255 0 13:26 pts/1 00:00:00 sqlplus oracle 15507 15505 0 13:26 ? 00:00:00 oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) . . SQL>
Our server process number is 15507, now I will use strace to trace it
bash> strace -o userproc.out -p 15507
start the instance with nomount option, OCSSD must be running, or instance will not start
SQL> startup nomount; Total System Global Area 284565504 bytes Fixed Size 1336036 bytes Variable Size 258063644 bytes ASM Cache 25165824 bytes SQL>
Now we can analyze the contents of the userproc.out, I will not list all contents, just useful parts of it
First row is
read(9, "001\6\3\212\6\376\377\377\377\1\376\377\377\377\376\377\377\377"..., 8208) = 49
it is our “startup nomount” command transmitted to server process from sqlplus by socket
connect(6, {sa_family=AF_FILE, path="/var/tmp/.oracle/sOCSSD_LL_testhost_"...}, 110) = 0
open("/u01/oracle/product/grid11g/auth/css/testdb03/A6857753/84b0b692", O_WRONLY|O_CREAT|O_EXCL, 0644) = 13
write(13, "\nS\336[", 4) = 4
close(13)
from this lines we can see that our process connects to socket file of OCSSD /var/tmp/.oracle/sOCSSD_LL_testhost_ to communicate with it and authenticates itself. That is why it impossible to start instance without OCSSD, if process could not connect to this socket, it would fail. Then it establishes communication with OHASD (Oracle High Availability Services Deamon) through the socket file /var/tmp/.oracle/sCRSD_UI_SOCKET, after exchange messages with OHASD it will get information about the location of spfile and asm_diskstring parameter
access("/var/tmp/.oracle/sCRSD_UI_SOCKET", F_OK) = 0 connect(14, {sa_family=AF_FILE, path="/var/tmp/.oracle/sCRSD_UI_SOCKET"...}, 110) = 0 open("/u01/oracle/product/grid11g/auth/ohasd/testdb03/A4972914/5140f6df", O_WRONLY|O_CREAT|O_EXCL, 0644) = 15
write(15, "\20\7\3519", 4) = 4
close(15)
write(14, "4\2\2\1\1\1\1\3\1T\235\376\t"..., 52) = 52
write(14, "8\3\2\1\1\1\1\4\1T\235\376\t"..., 56) = 56
read(14, "8\3\2\1\1\1\1\3\1T\235\376\t"..., 32768) = 56
write(14, "\212\1PC\v\2\2\5\1T\235\376\t"..., 394) = 394
read(14, "\366\nPC\v\2\2\4\1T\235\376\t"..., 32768) = 2806
write(14, "0\2\20\1\1T\235\376\t"..., 48) = 48
write(3, "kggpnpSIHAGetItem 1 = +dgroup1/ASM/asmparameterfile/registry.253.740659373"..., 75) = 7
write(3, "kfspFileNameGet name=+dgroup1/ASSM/asmparameterfile/registry.253.740659373"..., 78) = 78
write(3, "kggpnpSIHAGetItem 2 = ", 23) = 23
write(3, "kgfspb_shallow_discover dscstr=\"\""..., 33) = 33
As we can see our process got spfile location (+dgroup1/ASM/asmparameterfile/registry.253.740659373) and asm_diskstring=”” which is null by default from OHASD. Now, lets see where OHASD itself gets this information. To identify that I traced the OHASD process, and I found this lines
open("/etc/oracle/olr.loc", O_RDONLY) = 4 read(4, "olrconfig_loc=/u01/oracle/produc"..., 4096) = 108 read(4, "", 4096) = 0 close(4) = 0 open("/u01/oracle/product/grid11g/cdata/localhost/testhost.olr", O_RDONLY|O_SYNC|O_LARGEFILE) = 4 pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096 pread64(4, "\1\202\300I#\4\3OCR\226\361nA"..., 4096, 143360) = 4096 pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096 pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096 pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096 pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096 pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096 pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096 pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096 pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096 pread64(4, "\1\202\236\363%\4\2\3OCR\226\361nA"..., 4096, 4345856) = 4096 pread64(4, "\1\202\334\n&\4\2\3OCR\226\361nA"..., 4096, 4349952) = 4096 pread64(4, "\1\202\325\357-\4\2\3OCR\226\361nA"..., 4096, 4378624) = 4096 pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096 pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096 pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096 pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096 pread64(4, "\1\202\325\357-\4\2\3OCR\226\361nA"..., 4096, 4378624) = 4096
As one would expect OHASD reads this information from OLR, the path to it, it gets from /etc/oracle/olr.loc. I want to note that it is true for Oracle Restart mode, in Oracle RAC environment information is stored in GPNP profile and there is GPNPD process that maintains and manages this profile information. In Oracle Restart, as we can see OHASD process executes this role, and because there is no PGNP profile it stores information in OLR file. So, what next? Our process starts to scan all disks to identify ASM headers, after identifying them, it identifies which disks belongs to which diskgroups by information from header. There is many other metadata in the ASM diskheader that it reads, including pointers to spfile and votig disk file, it is kfdhdb.spfile, kfdhdb.spfflg (first block and number of blocks) and kfdhdb.vfstart, kfdhdb.vfend (begin block and end block). It is possible to read disk header using kfed utility from $ORACLE_HOME/bin directory. For example, lets read disk header of the /dev/loop1 which corresponds to ORCL:DISK1, spfile is on this disk.
shell> kfed read /dev/loop1 | more kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD --Indicates that this is ASM disk header . kfdhdb.grptyp: 2 ; 0x026: KFDGTP_NORMAL --Indicates mirroring level, in my case it is NORMAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER --Indicates that disk is the member disk of diskgroup . kfdhdb.dskname: DISK1 ; 0x028: length=5 --Disk name kfdhdb.grpname: DGROUP1 ; 0x048: length=7 --Disk group name, to which this disk belongs kfdhdb.fgname: DISK1 ; 0x068: length=5 --To which failure group this disk belongs . kfdhdb.secsize: 512 ; 0x0b8: 0x0200 --Disk sector size kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 --Disk block size kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 --Allocation Unit size, by default 1M . kfdhdb.vfstart: 0 ; 0x0ec: 0x00000000 --Begin block address of voting disk file kfdhdb.vfend: 0 ; 0x0f0: 0x00000000 --End block address of voting disk file kfdhdb.spfile: 59 ; 0x0f4: 0x0000003b --Begin block address of spfile kfdhdb.spfflg: 1 ; 0x0f8: 0x00000001 --Number of blocks containing spfile
Now, lets see what will be next steps of our process
open("/opt/oracle/extapi/32/asm/orcl/1/libasm.so", O_RDONLY) = 17
read(17, "\177ELF\1\1\1\3\3\1000\v004"..., 512) = 512
close(17) = 0
open("/dev/oracleasm/.query_version", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\1\20", 16) = 16
read(17, "MSA\2\1\20", 16) = 16
close(17) = 0
open("/dev/oracleasm/.get_iid", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\2\30", 24) = 24
read(17, "MSA\2\2\30\3", 24) = 24
close(17) = 0
open("/dev/oracleasm/.check_iid", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\3\30\3", 24) = 24
read(17, "MSA\2\3\30\3", 24) = 24
close(17) = 0
open("/dev/oracleasm/iid/0000000000000003", O_RDWR|O_CREAT|O_LARGEFILE, 0770) = 17
open("/dev/oracleasm/disks/DISK1", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200-%\310", 32) = 32
close(18) = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK2", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200+%\310", 32) = 32
close(18) = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK3", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200!%\310", 32) = 32
close(18) = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK4", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200#%\310", 32) = 32
close(18) = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK5", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200)%\310", 32) = 32
close(18) = 0 read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80 read(17, "MSA\2\7P@\364\311\20"..., 80) = 80 read(17, "MSA\2\7P@\364\311\20"..., 80) = 80 read(17, "MSA\2\7P@\364\311\20"..., 80) = 80 read(17, "MSA\2\7P@\364\311\20"..., 80) = 80 read(17, "MSA\2\7P@\364\311\20"..., 80) = 80 . . open("/u01/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0660) = 16
write(16, "Tue Jan 18 17:45:13 2011\n", 25) = 25
write(16, "Starting ORACLE instance (normal"..., 33) = 33
write(16, "\n", 1) = 1
close(16)
.
.
It opens /opt/oracle/extapi/32/asm/orcl/1/libasm.so library, reads it. Then it opens special files /dev/oracleasm/.query_version, /dev/oracleasm/.get_iid and /dev/oracleasm/.check_iid, this files is interfaces to ASM device manager. First one is used to get managers version, second one is used to get identifier of the ASM disk device manager instance (not ASM instance) and third for verifying of this instance identifier. In our case ASM device managers instance identifier is 0000000000000003. Then, our process opens /dev/oracleasm/iid/0000000000000003, in other words it establishes connection to the ASM device manager instance. Process will use this interface to read and write to ASM disks. Then it checks all disks from DISK1 to DISK5. After it gets all information about disks, i.e. groups, mirroring level, AU size, failure group and so on it starts reading ASM disk through established interface. I think in this step it reads the spfile. After it gets initialization parameters it starts to allocate memory structures and starting necessary background processes.
Now lets see what is stored in OLR regarding ora.asm resource
shell> cd $ORACLE_HOME/cdata/localhost shell> strings testhost.olr | grep dgroup1/ASM/asmparameterfile/registry | sed 's/~/\n/g' DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) DEGREE=1 DESCRIPTION=Oracle ASM resource ENABLED=1 GEN_USR_ORA_INST_NAME=+ASM LOAD=1 LOGGING_LEVEL=1 NAME=ora.asm NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373 START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr) START_TIMEOUT=900 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(ora.cssd) STOP_TIMEOUT=600 TYPE=ora.asm.type TYPE_ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r-- UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_INST_NAME=+ASM USR_ORA_OPEN_MODE=mount USR_ORA_OPI=false USR_ORA_STOP_MODE=immediate VERSION=11.2.0.1.0
We can found there information about spfile “SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373”, we also can see that ora.cssd is hard dependency resource, because OCSSD is responsible for synchronization between ASM and database instance. But what about asm_diskstring parameter, where is it?
shell> strings testhost.olr | grep ASM_DISKSTRING | sed 's/~/\n/g'
nothing, it is because my asm_diskstring parameter equals to null, default value, lets change it
SQL> alter system set asm_diskstring='ORCL:DISK1,ORCL:DISK2' scope=spfile; System altered.
check OLR file again
strings testdb03.olr | grep ASM_DISKSTRING | sed 's/~/\n/g' bASM_DISKSTRING ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME= ASM_DISKSTRING=ORCL:DISK1,ORCL:DISK2 AUTO_START=restore BASE_TYPE=ora.local_resource.type
Now we can see that information about asm_diskstring is also stored in OLR profile. And next time, at ASM instance startup it will scan only specified disk strings. If we specify disks on which spfile not present, our instance will not start
SQL> alter system set asm_diskstring='ORCL:DISK3,ORCL:DISK4,ORCL:DISK5' scope=spfile; System altered. SQL> startup nomount force; ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+DGROUP1/asm/asmparameterfile/registry.253.740659373' ORA-17503: ksfdopn:2 Failed to open file +DGROUP1/asm/asmparameterfile/registry.253.740659373 SQL>
You can also use the ocrdump utility to extract this entries on any platform in XML format:
shell> ocrdump -local -keyname SYSTEM.OHASD.RESOURCES.ora\!asm.CONFIG -xml -noheader shell> more OCRDUMPFILE SYSTEM.OHASD.RESOURCES.ora!asm.CONFIG ORATEXT ASM_DISKSTRING=ORCL:DISK3,ORCL:D ISK4,ORCL:DISK5~AUTO_START=restore~BASE_TYPE=ora.local_resource.type~CHECK_INTERVAL=1~CHECK_TIMEOUT=600~DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)~DE GREE=1~DESCRIPTION=Oracle ASM resource~ENABLED=1~GEN_USR_ORA_INST_NAME=+ASM~LOAD=1~LOGGING_LEVEL=1~NAME=ora.asm~NLS_LANG=~NOT_RESTARTING_TEMPLATE=~OFFLINE_CHECK_INTERVAL=0~PROFILE_CHANGE_TEMPLATE=~RES TART_ATTEMPTS=5~SCRIPT_TIMEOUT=60~SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373~START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)~START_TIMEOUT=900~STATE_CHANGE_TEMPLATE=~STOP_DE PENDENCIES=hard(ora.cssd)~STOP_TIMEOUT=600~TYPE=ora.asm.type~TYPE_ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r--~UPTIME_THRESHOLD=1d~USR_ORA_ENV=~USR_ORA_INST_NAME=+ASM~USR_ORA_OPEN_MODE=mount~USR_ORA_O PI=false~USR_ORA_STOP_MODE=immediate~VERSION=11.2.0.1.0~]]> PROCR_ALL_ACCESS PROCR_NONE PROCR_NONE oracle dba
So, draw conclusions, gentlemen!
643
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
