最后的话
最近很多小伙伴找我要Linux学习资料,于是我翻箱倒柜,整理了一些优质资源,涵盖视频、电子书、PPT等共享给大家!
资料预览
给大家整理的视频资料:
给大家整理的电子书资料:
如果本文对你有帮助,欢迎点赞、收藏、转发给朋友,让我有持续创作的动力!
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
mpstat命令输出每个可用处理器的标准输出活动,输出显示中cpu0是第一个处理器。还报告了所有处理器的全局的平均活动。mpstat命令可以在SMP和UP机器上使用,但在后者(UP机器上)中,只打印全局平均活动。如果未指定参数,则默认报告为CPU整体利用率报告。
备注:
UP(Uni-Processor):系统只有一个处理器单元,即单核CPU系统。
SMP(Symmetric Multi-Processors):系统有多个处理器单元。各个处理器之间共享总线,内存等等
[root@localhost ~]# mpstat
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
02:48:09 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:48:09 PM all 0.31 0.00 0.28 0.01 0.00 0.00 0.00 0.00 0.00 99.40
mpstat ...... [ interval [ count ] ]
interval参数指定每个报告之间的时间量(以秒为单位)。值为0(或根本没有参数)表示自系统启动(引导)以来将报告处理器统计信息。如果没有将interval参数设置为零,则可以将count参数与interval参数一起指定。count的值决定了间隔几秒生成的报告的数量。如果指定interval参数而不指定count参数,则mpstat命令将连续生成报表。
[root@localhost ~]# mpstat 2 5
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
02:55:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:55:12 PM all 0.50 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 99.00
02:55:14 PM all 0.25 0.00 0.63 0.00 0.00 0.00 0.00 0.00 0.00 99.12
02:55:16 PM all 0.38 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 99.12
02:55:18 PM all 0.50 0.00 0.62 0.00 0.00 0.00 0.00 0.00 0.00 98.88
02:55:20 PM all 0.38 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 99.12
Average: all 0.40 0.00 0.55 0.00 0.00 0.00 0.00 0.00 0.00 99.05
以两秒间隔显示所有处理器之间的五个全局统计数据报告。
就是读取 /proc/stat 文件中的数据:
open("/proc/stat", O_RDONLY) = 3
二、mpstat -P
-P { cpu [,...] | ON | ALL }
指示要报告统计信息的处理器编号,cpu是处理器编号,处理器0是第一个处理器。ON关键字表示要为每个在线处理器报告统计信息,而ALL关键字表示要报告所有处理器的统计信息。
输出处理器1(第二个处理器)的报告统计信息:
[root@localhost ~]# mpstat -P 1
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
03:08:34 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
03:08:34 PM 1 0.28 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 99.43
输出每个在线处理器报告统计信息:
[root@localhost ~]# mpstat -P ON
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
03:09:59 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
03:09:59 PM all 0.31 0.00 0.28 0.01 0.00 0.00 0.00 0.00 0.00 99.40
03:09:59 PM 0 0.31 0.00 0.27 0.00 0.00 0.00 0.00 0.00 0.00 99.41
03:09:59 PM 1 0.28 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 99.43
03:09:59 PM 2 0.33 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 99.38
03:09:59 PM 3 0.34 0.00 0.28 0.01 0.00 0.00 0.00 0.00 0.00 99.37
输出所有处理器的统计信息:
[root@localhost ~]# mpstat -P ALL
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
03:11:24 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
03:11:24 PM all 0.31 0.00 0.28 0.01 0.00 0.00 0.00 0.00 0.00 99.40
03:11:24 PM 0 0.31 0.00 0.27 0.00 0.00 0.00 0.00 0.00 0.00 99.41
03:11:24 PM 1 0.28 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 99.43
03:11:24 PM 2 0.33 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 99.38
03:11:24 PM 3 0.34 0.00 0.28 0.01 0.00 0.00 0.00 0.00 0.00 99.37
每个字段的含义:
CPU
Processor number. The keyword all indicates that statistics are calculated as averages among all processors.
%usr
Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice
Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%sys
Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and soft‐
ware interrupts.
%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%irq
Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.
%soft
Show the percentage of time spent by the CPU or CPUs to service software interrupts.
%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%guest
Show the percentage of time spent by the CPU or CPUs to run a virtual processor.
%gnice
Show the percentage of time spent by the CPU or CPUs to run a niced guest.
%idle
Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
具体请参考:Linux top命令的cpu使用率和内存使用率 这篇文章中关于各个字段的释义。
三、mpstat -I
-I { SUM | CPU | SCPU | ALL }
报告中断统计信息
3.1 mpstat -I SUM
使用SUM关键字,mpstat命令报告每个处理器的中断总数。将显示以下值:
CPU:处理器编号,关键字all表示统计数据是以所有处理器的平均值计算的。
intr/s:显示CPU每秒接收的中断总数。
[root@localhost ~]# mpstat -I SUM
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
03:21:23 PM CPU intr/s
03:21:23 PM all 93.37
3.2 mpstat -I CPU
3.2.1 数据来源
使用CPU关键字,将显示 CPU or CPUs 每秒接收到的每个中断(硬中断)的数量。
硬中断是硬件中断处理程序,在Linux 中称为上半部分,优先级最高,硬件中断处理程序处理过程中会屏蔽其它中断。
[root@localhost ~]# mpstat -I CPU
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
03:23:41 PM CPU 0/s 1/s 8/s 9/s 12/s 16/s 20/s 120/s 121/s 122/s 123/s 124/s 125/s 126/s 127/s NMI/s LOC/s SPU/s PMI/s IWI/s RTR/s RES/s CAL/s TLB/s TRM/s THR/s DFR/s MCE/s MCP/s ERR/s MIS/s PIN/s NPI/s PIW/s
03:23:41 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 4.03 0.00 0.00 0.00 20.01 0.00 0.00 0.19 0.00 0.21 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:23:41 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21.62 0.00 0.00 0.21 0.00 0.15 0.00 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:23:41 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22.75 0.00 0.00 0.17 0.00 0.15 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:23:41 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 22.86 0.00 0.00 0.25 0.00 0.15 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
数据来源就是读取 /proc/interrupts 文件,/proc/interrupts 提供了硬中断的运行情况:
备注:中断本质上是一种特殊的电信号,由硬件设备发向处理器。处理器接受到中断后,会马上向操作系统反映中断信号的到来,然后由操作系统负责处理这些新到来的数据。硬件设备生成中断的时候不考虑与处理器的时钟同步,即中断随时可以产生。
中断其实是一种异步的事件处理机制,可以提高系统的并发处理能力。
由于中断处理程序会打断其他进程的运行,所以,为了减少对正常进程运行调度的影响,中断处理程序就需要尽可能快地运行
open("/proc/interrupts", O_RDONLY) = 3
[root@localhost ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 55 0 0 0 IR-IO-APIC-edge timer
1: 4 0 0 0 IR-IO-APIC-edge i8042
8: 1 0 0 0 IR-IO-APIC-edge rtc0
9: 4 0 0 0 IR-IO-APIC-fasteoi acpi
12: 3 3 0 0 IR-IO-APIC-edge i8042
16: 0 0 0 0 IR-IO-APIC-fasteoi i801_smbus
20: 0 0 0 0 IR-IO-APIC-fasteoi idma64.0
120: 0 0 0 0 DMAR_MSI-edge dmar0
121: 0 0 0 0 DMAR_MSI-edge dmar1
122: 0 0 0 0 IR-PCI-MSI-edge aerdrv, PCIe PME
123: 148 16 10 2 IR-PCI-MSI-edge xhci_hcd
124: 4738 519 421 54943 IR-PCI-MSI-edge 0000:00:17.0
125: 1111752 0 0 0 IR-PCI-MSI-edge enp1s0
126: 38 1 109 9 IR-PCI-MSI-edge i915
127: 541 136 191 85 IR-PCI-MSI-edge snd_hda_intel:card0
NMI: 56 52 55 56 Non-maskable interrupts
LOC: 5504316 5950291 6263079 6292876 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 56 52 55 56 Performance monitoring interrupts
IWI: 52427 58892 47990 68017 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 56937 40801 42634 41527 Rescheduling interrupts
CAL: 1150 1147 1194 1149 Function call interrupts
TLB: 28982 39043 19609 20986 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 918 918 918 918 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event
其中的一些字段:
NMI(Non-maskable interrupts):在这种情况下,NMI会递增,因为每个定时器中断都会生成一个NMI(非屏蔽中断),NMI看门狗使用它来检测锁定。
LOC:LOC是每个CPU的内部APIC的 the local interrupt counter。
SPU:a spurious interrupt 是在APIC完全处理之前由某个IO设备引发然后降低的某个中断。因此,APIC看到这种中断,但不知道它来自哪个设备。在这种情况下,APIC将生成IRQ向量为0xff的中断。这也可能是芯片组错误造成的。
RES(Rescheduling interrupts)、CAL(Function call interrupts)、TLB(TLB shootdowns):根据OS的需要从一个CPU向另一个CPU发送重新调度、调用和TLB刷新中断。通常,内核开发人员和感兴趣的用户使用它们的统计信息来确定给定类型中断的发生。
TRM( Thermal event interrupts):当超过CPU的温度阈值时,发生热事件中断。当温度降至正常值时,也可能会产生该中断。
THR(Threshold APIC interrupts):当机器检查阈值计数器(通常计数内存或缓存的ECC纠正错误)超过可配置阈值时引发的中断。仅在某些系统上可用。
3.2.2 内核源码解析
// linux-3.10/fs/proc/interrupts.c
/\*
\* /proc/interrupts
\*/
static void \*int\_seq\_start(struct seq\_file \*f, loff\_t \*pos)
{
return (\*pos <= nr_irqs) ? pos : NULL;
}
static void \*int\_seq\_next(struct seq\_file \*f, void \*v, loff\_t \*pos)
{
(\*pos)++;
if (\*pos > nr_irqs)
return NULL;
return pos;
}
static void int\_seq\_stop(struct seq\_file \*f, void \*v)
{
/\* Nothing to do \*/
}
static const struct seq\_operations int_seq_ops = {
.start = int_seq_start,
.next = int_seq_next,
.stop = int_seq_stop,
.show = show_interrupts
};
static int interrupts\_open(struct inode \*inode, struct file \*filp)
{
return seq\_open(filp, &int_seq_ops);
}
static const struct file\_operations proc_interrupts_operations = {
.open = interrupts_open,
.read = seq_read,
.llseek = seq_lseek,
.release = seq_release,
};
static int __init proc\_interrupts\_init(void)
{
proc\_create("interrupts", 0, NULL, &proc_interrupts_operations);
return 0;
}
module\_init(proc_interrupts_init);
其中show_interrupts函数:
// linux-3.10/kernel/irq/proc.c
int show\_interrupts(struct seq\_file \*p, void \*v)
{
......
arch\_show\_interrupts(p, prec);
......
}
arch_show_interrupts是一个与架构有关的函数,对于x86架构:
// linux-3.10/arch/x86/kernel/irq.c
#define irq\_stats(x) (&per\_cpu(irq\_stat, x))
/\*
\* /proc/interrupts printing for arch specific interrupts
\*/
int arch\_show\_interrupts(struct seq\_file \*p, int prec)
{
int j;
seq\_printf(p, "%\*s: ", prec, "NMI");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->__nmi_count);
seq\_printf(p, " Non-maskable interrupts\n");
#ifdef CONFIG\_X86\_LOCAL\_APIC
seq\_printf(p, "%\*s: ", prec, "LOC");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->apic_timer_irqs);
seq\_printf(p, " Local timer interrupts\n");
seq\_printf(p, "%\*s: ", prec, "SPU");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_spurious_count);
seq\_printf(p, " Spurious interrupts\n");
seq\_printf(p, "%\*s: ", prec, "PMI");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->apic_perf_irqs);
seq\_printf(p, " Performance monitoring interrupts\n");
seq\_printf(p, "%\*s: ", prec, "IWI");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->apic_irq_work_irqs);
seq\_printf(p, " IRQ work interrupts\n");
seq\_printf(p, "%\*s: ", prec, "RTR");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->icr_read_retry_count);
seq\_printf(p, " APIC ICR read retries\n");
#endif
if (x86_platform_ipi_callback) {
seq\_printf(p, "%\*s: ", prec, "PLT");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->x86_platform_ipis);
seq\_printf(p, " Platform interrupts\n");
}
#ifdef CONFIG\_SMP
seq\_printf(p, "%\*s: ", prec, "RES");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_resched_count);
seq\_printf(p, " Rescheduling interrupts\n");
seq\_printf(p, "%\*s: ", prec, "CAL");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_call_count -
irq\_stats(j)->irq_tlb_count);
seq\_printf(p, " Function call interrupts\n");
seq\_printf(p, "%\*s: ", prec, "TLB");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_tlb_count);
seq\_printf(p, " TLB shootdowns\n");
#endif
#ifdef CONFIG\_X86\_THERMAL\_VECTOR
seq\_printf(p, "%\*s: ", prec, "TRM");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_thermal_count);
seq\_printf(p, " Thermal event interrupts\n");
#endif
#ifdef CONFIG\_X86\_MCE\_THRESHOLD
seq\_printf(p, "%\*s: ", prec, "THR");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", irq\_stats(j)->irq_threshold_count);
seq\_printf(p, " Threshold APIC interrupts\n");
#endif
#ifdef CONFIG\_X86\_MCE
seq\_printf(p, "%\*s: ", prec, "MCE");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", per\_cpu(mce_exception_count, j));
seq\_printf(p, " Machine check exceptions\n");
seq\_printf(p, "%\*s: ", prec, "MCP");
for\_each\_online\_cpu(j)
seq\_printf(p, "%10u ", per\_cpu(mce_poll_count, j));
seq\_printf(p, " Machine check polls\n");
#endif
seq\_printf(p, "%\*s: %10u\n", prec, "ERR", atomic\_read(&irq_err_count));
#if defined(CONFIG\_X86\_IO\_APIC)
seq\_printf(p, "%\*s: %10u\n", prec, "MIS", atomic\_read(&irq_mis_count));
#endif
return 0;
}
可以看到主要是从 per-cpu内存区读取相应的数据,关于x86_64 per-cpu相关知识请参考:Linux per-cpu
// linux-3.10/arch/x86/include/asm/hardirq.h
typedef struct {
unsigned int __softirq_pending;
unsigned int __nmi_count; /\* arch dependent \*/
#ifdef CONFIG\_X86\_LOCAL\_APIC
unsigned int apic_timer_irqs; /\* arch dependent \*/
unsigned int irq_spurious_count;
unsigned int icr_read_retry_count;
#endif
#ifdef CONFIG\_HAVE\_KVM
unsigned int kvm_posted_intr_ipis;
#endif
unsigned int x86_platform_ipis; /\* arch dependent \*/
unsigned int apic_perf_irqs;
unsigned int apic_irq_work_irqs;
#ifdef CONFIG\_SMP
unsigned int irq_resched_count;
unsigned int irq_call_count;
/\*
\* irq\_tlb\_count is double-counted in irq\_call\_count, so it must be
\* subtracted from irq\_call\_count when displaying irq\_call\_count
\*/
unsigned int irq_tlb_count;
#endif
#ifdef CONFIG\_X86\_THERMAL\_VECTOR
unsigned int irq_thermal_count;
#endif
#ifdef CONFIG\_X86\_MCE\_THRESHOLD
unsigned int irq_threshold_count;
#endif
} ____cacheline_aligned irq\_cpustat\_t;
DECLARE\_PER\_CPU\_SHARED\_ALIGNED(irq\_cpustat\_t, irq_stat);
// linux-3.10/include/linux/irq\_cpustat.h
/\*
\* Simple wrappers reducing source bloat. Define all irq\_stat fields
\* here, even ones that are arch dependent. That way we get common
\* definitions instead of differing sets for each arch.
\*/
#ifndef \_\_ARCH\_IRQ\_STAT
extern irq\_cpustat\_t irq_stat[]; /\* defined in asm/hardirq.h \*/
#define \_\_IRQ\_STAT(cpu, member) (irq\_stat[cpu].member)
#endif
3.3 mpstat -I SCPU
3.3.1 数据来源
使用SCPU关键字,将显示 CPU or CPUs 每秒接收到的每个软件中断的数量。
软中断是预留给系统中对时间要求较严格和重要的下半部使用的(上半部是硬件中断处理,优先级最高),软中断执行过程中会响应其它的中断。
驱动中只有块设备和网络子系统使用了软中断。
[root@localhost ~]# mpstat -I SCPU
Linux 3.10.0-957.el7.x86\_64 (localhost.localdomain) 11/28/2022 \_x86\_64\_ (4 CPU)
04:48:54 PM CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s BLOCK_IOPOLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
04:48:54 PM 0 0.00 10.62 0.17 4.26 0.02 0.00 0.02 6.63 0.00 3.92
04:48:54 PM 1 0.00 12.78 0.00 0.03 0.00 0.00 0.00 7.19 0.00 4.89
04:48:54 PM 2 0.00 12.41 0.00 0.03 0.00 0.00 0.00 7.28 0.00 4.48
04:48:54 PM 3 0.00 12.97 0.00 0.02 0.19 0.00 0.00 7.13 0.00 4.90
数据来源读取 /proc/softirqs 文件,/proc/softirqs 提供了软中断的运行情况:
最后的话
最近很多小伙伴找我要Linux学习资料,于是我翻箱倒柜,整理了一些优质资源,涵盖视频、电子书、PPT等共享给大家!
资料预览
给大家整理的视频资料:
给大家整理的电子书资料:
如果本文对你有帮助,欢迎点赞、收藏、转发给朋友,让我有持续创作的动力!
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
数据来源读取 /proc/softirqs 文件,/proc/softirqs 提供了软中断的运行情况:
最后的话
最近很多小伙伴找我要Linux学习资料,于是我翻箱倒柜,整理了一些优质资源,涵盖视频、电子书、PPT等共享给大家!
资料预览
给大家整理的视频资料:
[外链图片转存中…(img-Q7dfQlen-1715485079270)]
给大家整理的电子书资料:
[外链图片转存中…(img-6pti3VFM-1715485079270)]
如果本文对你有帮助,欢迎点赞、收藏、转发给朋友,让我有持续创作的动力!
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!