使用oradebug ipc来检查cluster interconnect

最新推荐文章于 2022-01-13 13:50:06 发布

msdnchina

最新推荐文章于 2022-01-13 13:50:06 发布

阅读量1.4k

点赞数

分类专栏： Oracle DB 文章标签： oradebug ipc interconnect

本文链接：https://blog.csdn.net/msdnchina/article/details/73621740

版权

Oracle DB 专栏收录该内容

773 篇文章 3 订阅

订阅专栏

参考文章:
How to Use Oradebug to Get Trace File Name and Location (文档 ID 310830.1)
Tuning Inter-Instance Performance in RAC and OPS (文档 ID 181489.1)
Instances Unable To Start If MTU Size Is Different for Cluster_interconnect (文档 ID 300388.1)


[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 23 11:57:51 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options


SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
IPC information written to the trace file
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@rac1 ~]$

/u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc文件的内容如下:

Trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name:	Linux
Node name:	rac1
Release:	3.10.0-327.el7.x86_64
Version:	#1 SMP Thu Oct 29 17:29:29 EDT 2015
Machine:	x86_64
Instance name: orcl1
Redo thread mounted by this instance: 1
Oracle process number: 34
Unix process pid: 17205, image: oracle@rac1 (TNS V1-V3)




*** 2017-06-23 12:03:14.891
*** SESSION ID:(1274.497) 2017-06-23 12:03:14.891
*** CLIENT ID:() 2017-06-23 12:03:14.891
*** SERVICE NAME:(SYS$USERS) 2017-06-23 12:03:14.891
*** MODULE NAME:(sqlplus@rac1 (TNS V1-V3)) 2017-06-23 12:03:14.891
*** ACTION NAME:() 2017-06-23 12:03:14.891
 
Processing Oradebug command 'setmypid'


*** 2017-06-23 12:03:14.891
Oradebug command 'setmypid' console output: <none>


*** 2017-06-23 12:03:20.882
Processing Oradebug command 'ipc'
Dump of unix-generic skgm context
areaflags            000000f7
realmflags           0000001f
mapsize              00000800
protectsize          00001000
lcmsize              00001000
seglen               00200000
largestsize  0000000fba082000
smallestsize 0000000008000000
stacklimit     0x7ffee85188d0
stackdir                   -1
mode                      640
magic                acc01ade
Handle:        0x7f71abcd40b0 `/u01/app/oracle/product/11.2.0/db_1orcl1'
Dump of unix-generic realm handle `/u01/app/oracle/product/11.2.0/db_1orcl1', flags = 00000000
 Area #0 `Fixed Size' containing Subareas 0-0
  Total size 0000000000229ae0 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      0        0   327686 0x00000060000000 0x00000060000000
                              Subarea size     Segment size
                          000000000022a000 0000000018000000
 Area #1 `Variable Size' containing Subareas 4-4
  Total size 0000000df8000000 Minimum Subarea size 08000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      1        4   360455 0x00000078000000 0x00000078000000
                              Subarea size     Segment size
                          0000000df8000000 0000000df8000000
 Area #2 `Redo Buffers' containing Subareas 2-2
  Total size 0000000007caf000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      2        2   327686 0x0000007034f000 0x0000007034f000
                              Subarea size     Segment size
                          0000000007caf000 0000000018000000
 Area #3 `Base Allocator Control' containing Subareas 3-3
  Total size 0000000000002000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      3        3   327686 0x00000077ffe000 0x00000077ffe000
                              Subarea size     Segment size
                          0000000000002000 0000000018000000
 Area #4 `Slab Allocator Control' containing Subareas 1-1
  Total size 0000000010125000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      4        1   327686 0x0000006022a000 0x0000006022a000
                              Subarea size     Segment size
                          0000000010125000 0000000018000000
 Area #5 `skgm overhead' containing Subareas 5-5
  Total size 0000000000005000 Minimum Subarea size 00000000
   Area  Subarea    Shmid      Stable Addr      Actual Addr
      5        5   393224 0x00000e70000000 0x00000e70000000
                              Subarea size     Segment size
                          0000000000005000 0000000000200000
Dump of Linux-specific skgm context
sharedmmu 00000001
shareddec        0
used region        0: start 0000000060000000 length 000001c208000000
Maximum processes:               = 1000
Number of semaphores per set:    = 125
Semaphores key overhead per set: = 4
User Semaphores per set:         = 121
Number of semaphore sets:        = 9
Semaphore identifiers:           = 9
Semaphore List=
524298
557067
589836
622605
655374
688143
720912
753681
786450
-------------- system semaphore information -------------
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 0          root       644        80         2                       
0x00000000 32769      root       644        16384      2                       
0x00000000 65538      root       644        280        2                       
0x00000000 196611     grid       640        4096       0                       
0x00000000 229380     grid       640        4096       0                       
0xa8c203cc 262149     grid       640        4096       0                       
0x00000000 327686     oracle     640        402653184  43                      
0x00000000 360455     oracle     640        59995324416 43                      
0x9f8b9fa4 393224     oracle     640        2097152    43                      
------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x000000a7 0          root       600        1         
0x1fe737a8 131073     grid       640        125       
0x1fe737a9 163842     grid       640        125       
0x1fe737aa 196611     grid       640        125       
0x1fe737ab 229380     grid       640        125       
0x1fe737ac 262149     grid       640        125       
0x1fe737ad 294918     grid       640        125       
0x1fe737ae 327687     grid       640        125       
0x1fe737af 360456     grid       640        125       
0x1fe737b0 393225     grid       640        125       
0xe7ec25d0 524298     oracle     640        125       
0xe7ec25d1 557067     oracle     640        125       
0xe7ec25d2 589836     oracle     640        125       
0xe7ec25d3 622605     oracle     640        125       
0xe7ec25d4 655374     oracle     640        125       
0xe7ec25d5 688143     oracle     640        125       
0xe7ec25d6 720912     oracle     640        125       
0xe7ec25d7 753681     oracle     640        125       
0xe7ec25d8 786450     oracle     640        125       
ksxpdmp:  facility 0 (?) (0x0, (nil)) counts 0, 0
Dumping the osd state
  Dumping the osd context (brief)
SKGXP:[7f71abce3c48.0]{ctx}: SKGXPCTX: 0x0x7f71abce3c48 ctx   
SKGXP:[7f71abce3c48.1]{ctx}: Dump start: 1498190600906194 (gtod usec)
SKGXP:[7f71abce3c48.2]{ctx}: Compatible: 0 0.0 0
SKGXP:[7f71abce3c48.3]{ctx}: SKGXP Versions: 3.3.4.2.2.c1.7(3.2.4.1)
SKGXP:[7f71abce3c48.4]{ctx}: 
SKGXP:[7f71abce3c48.5]{ctx}: WAIT HISTORY
SKGXP:[7f71abce3c48.6]{ctx}: Wait Time  Time since     Fast reaps  Wait Type  Return Code
SKGXP:[7f71abce3c48.7]{ctx}:   (ms)     prev wait(ms)  before                           
SKGXP:[7f71abce3c48.8]{ctx}: ---------  -------------- ----------- ---------  -----------
SKGXP:[7f71abce3c48.9]{ctx}: wait delta 321 sec (321964 msec) ctx ts 0 sec (1 msec) last ts 0 msec twait secs 5 (blocked 0) curts = 321964
SKGXP:[7f71abce3c48.10]{ctx}: 
SKGXP:[7f71abce3c48.11]{ctx}: EXTENDED WAIT HISTORY STATS
SKGXP:[7f71abce3c48.12]{ctx}: Time       Prev      Ncomp     Nwait     Nloop     Nsel      Select    No Svc     Events      
SKGXP:[7f71abce3c48.13]{ctx}: (-ms)      (ms)                                              (ms)      (ms)                   
SKGXP:[7f71abce3c48.14]{ctx}: ---------- --------- --------- --------- --------- --------- --------- ---------- ----------- 
SKGXP:[7f71abce3c48.15]{ctx}: current (0)
SKGXP:[7f71abce3c48.16]{ctx}:     321964         0         0         0         0         0         0          0           0 
SKGXP:[7f71abce3c48.17]{ctx}: 
SKGXP:[7f71abce3c48.18]{ctx}: user cpu time since last wait 0 sec 140728898420736 ticks
SKGXP:[7f71abce3c48.19]{ctx}: system cpu time since last wait 0 sec 140728898420736 ticks
SKGXP:[7f71abce3c48.20]{ctx}: locked 2
SKGXP:[7f71abce3c48.21]{ctx}: blocked 0
SKGXP:[7f71abce3c48.22]{ctx}: timed wait receives 0
SKGXP:[7f71abce3c48.23]{ctx}: fast reaps since last wait 0
SKGXP:[7f71abce3c48.24]{ctx}: context timestamp 0x1
SKGXP:[7f71abce3c48.25]{ctx}: flags=1 flags1=30 zcpyflg=5824 iflags=40820 aflags=80 antopts 0 trcflags=0 trclevel=3
SKGXP:[7f71abce3c48.26]{ctx}: rcvbuf=131072 sndbuf=262144
SKGXP:[7f71abce3c48.27]{ctx}: expire idle_ach=120 def_reap=60 if_detect=60 hb=180 nopath=5 vrpc=30 (on=300, off=30)
SKGXP:[7f71abce3c48.28]{ctx}: mcph alloc=0 inuse=0 inactive=0 thresh=0 hashtbl=(nil)
SKGXP:[7f71abce3c48.29]{ctx}: 
SKGXP:[7f71abce3c48.30]{ctx}: admno 0x3d9849db admport:
SKGXP:[7f71abce3c48.31]{ctx}: 	SSKGXPT 0x7f71abce51b8 flags 0x5 { READPENDING } sockno 4 IP 169.254.9.62 UDP 62213 lerr 0
SKGXP:[7f71abce3c48.32]{ctx}: 
SKGXP:[7f71abce3c48.33]{ctx}: post port: 
SKGXP:[7f71abce3c48.34]{ctx}: 	SSKGXPT 0x7f71abce5680 flags 0x0  sockno 7 IP 127.0.0.1 UDP 62213 lerr 0
SKGXP:[7f71abce3c48.35]{ctx}: 
SKGXP:[7f71abce3c48.36]{ctx}: sconno [6eb2aebf, 6eb2aebf]=0 aconno [504ea0a2, 504ea0a2]=0
SKGXP:[7f71abce3c48.37]{ctx}: 
SKGXP:[7f71abce3c48.38]{ctx}: SSKGXPCTX 0x7f71abce3c48 flags=0x0 npt=3 zcomp=0 lerr=0
SKGXP:[7f71abce3c48.39]{ctx}: 
SKGXP:[7f71abce3c48.40]{ctx}: time handle
SKGXP:[7f71abce3c48.41]{ctx}: SSKGXPTH 0x7f71abce4c68 start 1463076436732 last 1463076758696 count 0 thresh 0 tcb 1
SKGXP:[7f71abce3c48.42]{ctx}: 
SKGXP:[7f71abce3c48.43]{ctx}: memory (native) allocs 6 free 1 lost 0 out 8376 max 32768
SKGXP:[7f71abce3c48.44]{ctx}: 
SKGXP:[7f71abce3c48.45]{ctx}: ctx(10176) sctx(1960) rqh(1064) cid(24)
SKGXP:[7f71abce3c48.46]{ctx}: prt(480) cnh(1984/15088) ach(1568/14672) rpc(2448) pid(64)
SKGXP:[7f71abce3c48.47]{ctx}: rgnst(472) rgnpt(296) rgn(15784) rbd(120) bid(64) mcph(8416/21520) mupdh(1168/14272)
SKGXP:[7f71abce3c48.48]{ctx}: mcb(852008) smap(4184) nsm(2084) me(1152) mcbe(48) mcbi(0) epid(40)
SKGXP:[7f71abce3c48.49]{ctx}: mhdr(76) fhdr(28) madm(272) aggpid(32)
SKGXP:[7f71abce3c48.50]{ctx}: sta msc(448) cnh(344) prt(216) ep(48)
SKGXP:[7f71abce3c48.51]{ctx}: aggpt(208) swin(384) finfo(176) segf(200)
SKGXP:[7f71abce3c48.52]{ctx}: 
SKGXP:[7f71abce3c48.53]{ctx}: done Queue
SKGXP:[7f71abce3c48.54]{ctx}: 	no completed requests
SKGXP:[7f71abce3c48.55]{ctx}: 
SKGXP:[7f71abce3c48.56]{ctx}: connection Queue
SKGXP:[7f71abce3c48.57]{ctx}: 	no pending connect disconnect operations
SKGXP:[7f71abce3c48.58]{ctx}: 
SKGXP:[7f71abce3c48.59]{ctx}: sends waiting to be transmitted
SKGXP:[7f71abce3c48.60]{ctx}: 	no sends waiting to be transmitted
SKGXP:[7f71abce3c48.61]{ctx}: 
SKGXP:[7f71abce3c48.62]{ctx}: pending ack Queue
SKGXP:[7f71abce3c48.63]{ctx}: 	no send requests pending ack
SKGXP:[7f71abce3c48.64]{ctx}: 
SKGXP:[7f71abce3c48.65]{ctx}: Vrpc requests pending reply
SKGXP:[7f71abce3c48.66]{ctx}: 
SKGXP:[7f71abce3c48.67]{ctx}: Port queue
SKGXP:[7f71abce3c48.68]{ctx}: 
SKGXP:[7f71abce3c48.69]{ctx}: Region queue
SKGXP:[7f71abce3c48.70]{ctx}: SKGXPRGN: 0x7f71a8a3c258
SKGXP:[7f71abce3c48.71]{ctx}:    region id 0x23ec rgn idx: 0 rgn key: 1329307295
SKGXP:[7f71abce3c48.72]{ctx}:    base address 0x78000000 size: 59995324416
SKGXP:[7f71abce3c48.73]{ctx}:    flags 0x0 state: 0x2 largest buffer: 0
SKGXP:[7f71abce3c48.74]{ctx}:    num bids prepared: 0
SKGXP:[7f71abce3c48.75]{ctx}: SKGXPRGNSTATE: 0x7f71abce4fd8
SKGXP:[7f71abce3c48.76]{ctx}: 	rgns mapped: 1 num shared: 1 max: 1
SKGXP:[7f71abce3c48.77]{ctx}: 	flags: 0 bseqno: 11886 nxtrgnno: 9196 mmapcnt: 1
SKGXP:[7f71abce3c48.78]{ctx}: 	shared ports open: 1 limits (1,16)
SKGXP:[7f71abce3c48.79]{ctx}: 	shared bids prepared: 0 hiwat: 0
SKGXP:[7f71abce3c48.80]{ctx}: 	space - used: 0 max possible: 4194304 sock limit: 524288
SKGXP:[7f71abce3c48.81]{ctx}: 	rgnarr[0x7f71abced860(32)] rgnpts[0]=0x7f71abced980
SKGXP:[7f71abce3c48.82]{ctx}:   REGION ARRAY
SKGXP:[7f71abce3c48.83]{ctx}: 	rgn mapped[0] = 0x7f71a8a3c258
SKGXP:[7f71abce3c48.84]{ctx}:   SHARED PORT ARRAY
SKGXP:[7f71abce3c48.85]{ctx}:    rgn port[0]=0x7f71abced980
SKGXP:[7f71abce3c48.86]{ctx}: 	SKGXPRGNPORT: 0x7f71abced980
SKGXP:[7f71abce3c48.87]{ctx}: 	nbids: 0 space: 0 used: 0
SKGXP:[7f71abce3c48.88]{ctx}: 	SSKGXPT 0x7f71abced980 flags 0x0  sockno 10 IP 169.254.9.62 UDP 42117 lerr 0 ---------->>>此处
SKGXP:[7f71abce3c48.89]{ctx}:   SKGXPGPID Internet address 169.254.9.62 UDP port number 42117, mask 16 ---------->>>此处
SKGXP:[7f71abce3c48.90]{ctx}: 
SKGXP:[7f71abce3c48.91]{ctx}: Dumping Connection Handle Table (0x7f71abce5918)
SKGXP:[7f71abce3c48.92]{ctx}:                hdl     sconno     aconno      admno   RmtPid  state   seq#       msgs     rtrans    credits      rtt   last ack         id               ip 
SKGXP:[7f71abce3c48.93]{ctx}: 
SKGXP:[7f71abce3c48.94]{ctx}: Dumping Accept Handle Table ((nil))
SKGXP:[7f71abce3c48.95]{ctx}: 
SKGXP:[7f71abce3c48.96]{ctx}: Dump end: 1498190600906893 (gtod usec)
SKGXP:[7f71abce3c48.97]{ctx}: SKGXPCTX: 0x0x7f71abce3c48 complete
SKGXP:[7f71abce3c48.98]{ctx}: 
Done dumping the osd state
Dumping ksxp state
  ksxppg=0x7f71a8cb47a8 ksxpsg=0xe49972028 ksxpsg_a=0xe49972028ksxpssg=0xe49971d70 rm=0xe69745528
proc state: (pid: 34) [flg: 1 sg: 1]
 curts 1498190600 wtctr 0
  Dumping ksxp contexts
    Context[2] 0x7f71a8a31fb0 GES state 1
      Dumping region queue
       region count: 0
      Dumping connection queue
       connection count: 0
      Dumping done requests queue
       done requests count: 0
      Dumping mupq requests queue
       mupq requests count: 0
      Dumping zombie requests queue
       zombie requests count: 0
      Dumping delayed callback requests queue
       delayed callback requests count: 0
    Context[1] 0x7f71abce3798 gc ksxp component context state 1
      Dumping region queue
       region count: 1
      Dumping connection queue
       connection count: 0
      Dumping done requests queue
       done requests count: 0
      Dumping mupq requests queue
       mupq requests count: 0
      Dumping zombie requests queue
       zombie requests count: 0
      Dumping delayed callback requests queue
       delayed callback requests count: 0
  Dumping pending request queue
   pending request count: 0


*** 2017-06-23 12:03:20.907
Oradebug command 'ipc' console output: 
IPC information written to the trace file


*** 2017-06-23 12:03:40.517
Processing Oradebug command 'tracefile_name'


*** 2017-06-23 12:03:40.517
Oradebug command 'tracefile_name' console output: 
/u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc

查看该主机的IP地址情况:

[oracle@rac1 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet 192.168.100.193/24 brd 192.168.100.255 scope global secondary enp5s0f0:1
       valid_lft forever preferred_lft forever
    inet 192.168.100.191/24 brd 192.168.100.255 scope global secondary enp5s0f0:3
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feac:3cc8/64 scope link 
       valid_lft forever preferred_lft forever
3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1
       valid_lft forever preferred_lft forever
    inet 169.254.9.62/16 brd 169.254.255.255 scope global enp5s0f1:1 --------->>>>注意此处.
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feac:3cc9/64 scope link 
       valid_lft forever preferred_lft forever
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:91:7f11/64 scope link 
       valid_lft forever preferred_lft forever
5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:9003:91:7ed1/64 scope link 
       valid_lft forever preferred_lft forever
7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024
    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff
[oracle@rac1 ~]$