linux TTY子系统 --- bug来了,稍候再整理

 

 

 

 

                                                                                                                                          Version: 0.9

                                                                                                                                          Date: 2018-09-27

                                                                                                                                           Nemo.Han


目录

1.学习出发点及目标

1.1 学习出发点

1.2 学习目标

2 什么是TTY

2.1 串口

2.2 伪终端

2.3 控制终端

2.4 虚拟终端

2.5 其他类型

3 TTY子系统

3.1 TTY核心

3.2 tty_register_driver

3.3 tty_register_device

3.4 tty_register_device_attr

3.5 tty_port_register_device_attr_serdev

3.6 tty_cdev_add

3.7 tty_port_init

3.7 tty_open

目录

1.学习出发点及目标

1.1 学习出发点

1.2 学习目标

2 什么是TTY

2.1 串口

2.2 伪终端

2.3 控制终端

2.4 虚拟终端

2.5 其他类型

3 TTY子系统

3.1 TTY核心

3.2 tty_register_driver

3.3 tty_register_device

3.4 tty_register_device_attr

3.5 tty_port_register_device_attr_serdev

3.6 tty_cdev_add

3.7 tty_port_init

3.7 tty_open

3.8 tty_write

3.9 tty_read

3.11 tty_filp_buf_push

4 UART核心层

4.1 uart_register_driver


 

3.8 tty_write


1.学习出发点及目标

1.1 学习出发点

 

本文主要记录uart与tty的关系,学习目标如下:

  1. TTY子系统
  2. LINUX下TTY、CONSOLE、串口之间是怎样的层次关系?具体的函数接口是怎样的?
  3. UART怎么注册到TTY的
  4. TTY是怎么在内核系统中工作的
  5. 上层配置数据代码流程
  6. 上层数据下发流程
  7. 驱动数据上报流程
  8. poll机制实现
  9. 为什么Linux系统中会有那么多的TTY
  10. struct tty_struct是在什么时候分配并初始化的
  11. open_wait在哪里完成的?应该是在驱动层
  12. 同时在4.4版本中,uart_open会调用uart_startup,而4.14中没有调用,那uart_startup是在哪里调用的呢?
  13. tty_insert_file_char函数流程需要梳理,buf初始化,buf管理
  14. tty_update_time中涉及到的timer有什么用,tty核心是怎么用来管理的?

1.2 学习目标

  1. 可独立设计serial驱动,并能很好的工作
  2. 驱动中允许DMA进行数据收发
  3. 使用QT开发串口助手,并且能够很好的进行工作,并网上发布工具
  4. 输出文档,提供层次结构图、数据流程图、关键函数注释

2 什么是TTY

在Linux中,TTY也许是跟终端有关系的最为混乱的术语。TTY是TeleTYpe的一个老缩写。Teletypes,或者telety pewriters,原来指的是电传打字机,是通过串行线用打印机键盘通过阅读和发送信息的东西,和古老的电报机区别并不是很大。之后,当计算机只能以批处理方式运行时(当时穿孔卡片阅读器是唯一一种使程序载入运行的方式),电传打字机成为唯一能够被使用的“实时”输入/输出设备。最终,电传打字机被键盘和显示器终端所取代,但在终端或TTY接插的地方,操作系统仍然需要一个程序来监视串行端口。一个getty“Get TTY”的处理过程是:一个程序监视物理的TTY/终端接口。对一个虚拟网络控制台(VNC)来说,一个伪装的TTY(Pseudo-TTY,即假冒的TTY,也叫做“PTY”)是等价的终端。当你运行一个xterm(终端仿真程序)或GNOME终端程序时,PTY对虚拟的用户或者如xterm一样的伪终端来说,就像是一个TTY在运行。“Pseudo”的意思是“duplicating in a fake way”(用伪造的方法复制),它相比“virtual”或“emulated”更能真实的说明问题。而在的计算中,它却处于被放弃的阶段。

tty也是一个Unix命令,用来给出当前终端设备的名称。

终端是一种字符型设备,它有多种类型,通常使用tty来简称各种类型的终端设备。

在Linux系统的设备特殊文件目录/dev/下,终端特殊设备文件一般有以下几种:串口、伪终端、控制终端、控制台、虚拟终端、其他类型。

2.1 串口

串行端口终端(Serial Port Terminal)是使用计算机串行端口连接的终端设备。计算机把每个串行端口都看作是一个字符设备。有段时间这些串行端口设备通常被称为终端设备,因为那时它的最大用途就是用来连接终端。这些串行端口所对应的设备名称是/dev/tts/0(或/dev/ttyS0),/dev/tts/1(或/dev/ttyS1)等,设备号分别是(4,0),(4,1)等,分别对应于DOS系统下的COM1、COM2等。若要向一个端口发送数据,可以在命令行上把标准输出重定向到这些特殊文件名上即可。例如,在命令行提示符下键入:echo test > /dev/ttyS1会把单词”test”发送到连接在ttyS1(COM2)端口的设备上。

2.2 伪终端

伪终端(Pseudo Terminal)是成对的逻辑终端设备(即master和slave设备,对master的操作会反映到slave上)。

例如/dev/ptyp3和/dev/ttyp3(或者在设备文件系统中分别是/dev/pty /m3和 /dev/pty/s3)。它们与实际物理设备并不直接相关。如果一个程序把ptyp3(master设备)看作是一个串行端口设备,则它对该端口的读/ 写操作会反映在该逻辑终端设备对应的另一个ttyp3(slave设备)上面。而ttyp3则是另一个程序用于读写操作的逻辑设备。telnet主机A就是通过“伪终端”与主机A的登录程序进行通信。

2.3 控制终端

如果当前进程有控制终端(Controlling Terminal)的话,那么/dev/tty就是当前进程的控制终端的设备特殊文件。可以使用命令”ps –ax”来查看进程与哪个控制终端相连。对于你登录的shell,/dev/tty就是你使用的终端,设备号是(5,0)。使用命令”tty”可以查看它具体对应哪个实际终端设备。/dev/tty有些类似于到实际所使用终端设备的一个联接。

2.4 虚拟终端

在Xwindow模式下的伪终端.如在Kubuntu下用konsole,就是用的虚拟终端,用tty命令可看到/dev/pts/name,name为当前用户名。

2.5 其他类型

Linux系统中还针对很多不同的字符设备存在有很多其它种类的终端设备特殊文件。例如针对ISDN设备的/dev/ttyIn终端设备等。

tty设备包括虚拟控制台,串口以及伪终端设备。

/dev/tty代表当前tty设备,在当前的终端中输入 echo “hello” > /dev/tty ,都会直接显示在当前的终端中。

3 TTY子系统

tty设备有四层:tty核心、tty线路规程、uart核心层、tty驱动。我们写驱动只负责最底层的tty驱动,线路规程的设置也是在底层的tty驱动,tty核心是封装好的。用户空间主要是通过设备文件同tty核心交互,而tty核心根据用户空间的操作再选择跟tty线路规程或者tty驱动交互,例:设置硬件的ioctl指令就直接交给tty驱动处理,read和write操作交给tty线路规程处理。

 

3.1 TTY核心

所有tty类型的驱动的顶层构架,向应用层提供了统一的接口,应用层的read/write等调用首先会到达这里。此层由内核实现,代码主要分布在drivers/char目录下的n_tty.c,tty_io.c等文件中。

3.2 tty_register_driver


int tty_register_driver(struct tty_driver *driver)

{

       int error;
       int i;
       dev_t dev;
       struct device *d;
      //初始化sprd_uart_driver时,major与minor都为0
      //接下来按照字符设备的注册流程,注册一个cdev
       if (!driver->major) {
           error = alloc_chrdev_region(&dev, driver->minor_start,driver->num, driver->name);
           if (!error) {
                     driver->major = MAJOR(dev);
                     driver->minor_start = MINOR(dev);
              }
       } else {
              dev = MKDEV(driver->major, driver->minor_start);
              error = register_chrdev_region(dev, driver->num, driver->name);
       }
       if (error < 0)
              goto err;
            //在uart_register_driver函数中,normal->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV
       if (driver->flags & TTY_DRIVER_DYNAMIC_ALLOC) {
              error = tty_cdev_add(driver, dev, 0, driver->num);
       }
       mutex_lock(&tty_mutex);
       list_add(&driver->tty_drivers, &tty_drivers); //添加驱动到全局tty驱动链表中
       mutex_unlock(&tty_mutex);
            /*根据动态设备标记决定是否此时注册tty设备,就是uart_register_driver里的normal->flags,在这里uart_register_driver中有设置,它是在sprd_probe里注册的,device注册后将在sys中形成层次结构,系统启动后,udev系统(android中为vold)将在/dev/下面生成节点文件,这些节点文件操作都是按照该字符设备的tty_fops进行*/
       if (!(driver->flags & TTY_DRIVER_DYNAMIC_DEV)) {
              for (i = 0; i < driver->num; i++) {
                     d = tty_register_device(driver, i, NULL);
                     ……
              }
       }
       proc_tty_register_driver(driver);  à driver->proc_entry = ent;
       driver->flags |= TTY_DRIVER_INSTALLED;
       return 0;
            ……
}

注册tty设备:如果TTY驱动程序的标志设置了TTY_DRIVER_DYNAMIC_DEV标志位,则需要调用此函数注册单个TTY设备;如果未设置此标志位,tty驱动程序不应当调用此函数。

3.3 tty_register_device

struct device *tty_register_device(struct tty_driver *driver, unsigned index,struct device *device)
{
	return tty_register_device_attr(driver, index, device, NULL, NULL);
}

tty_register_device的实际执行函数为tty_regitser_device_attr。

3.4 tty_register_device_attr

struct device *tty_register_device_attr(struct tty_driver *driver, unsigned index, struct device *device, void *drvdata, const struct attribute_group **attr_grp)
{
       char name[64];
       dev_t devt = MKDEV(driver->major, driver->minor_start) + index;
       struct device *dev = NULL;
       int retval = -ENODEV;
       bool cdev = false;
       if (index >= driver->num) {
              printk(KERN_ERR "Attempt to register invalid tty line number "" (%d).\n", index);
              return ERR_PTR(-EINVAL);
       }
       if (driver->type == TTY_DRIVER_TYPE_PTY)
              pty_line_name(driver, index, name);
       else
              tty_line_name(driver, index, name);
            //在uart_register_driver中,flag字段设置为TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV,此处会调用tty_cdev_add
       if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
              retval = tty_cdev_add(driver, devt, index, 1);
              if (retval)
                     goto error;
              cdev = true;
       }
       dev = kzalloc(sizeof(*dev), GFP_KERNEL);
       if (!dev) {
              retval = -ENOMEM;
              goto error;
       }
       dev->devt = devt;
       dev->class = tty_class;
       dev->parent = device;
       dev->release = tty_device_create_release;
       dev_set_name(dev, "%s", name);
       dev->groups = attr_grp;
       dev_set_drvdata(dev, drvdata);
       retval = device_register(dev);
       if (retval)
              goto err_put;
       if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
              /*
               * Free any saved termios data so that the termios state is
               * reset when reusing a minor number.
               */
              tp = driver->termios[index];
              if (tp) {
                     driver->termios[index] = NULL;
                     kfree(tp);
              }
              retval = tty_cdev_add(driver, devt, index, 1);
              if (retval)
                     goto err_del;
       }
       dev_set_uevent_suppress(dev, 0);
       kobject_uevent(&dev->kobj, KOBJ_ADD);
       return dev;
err_del:
       device_del(dev);
err_put:
       put_device(dev);
       return ERR_PTR(retval);

}

 

3.5 tty_port_register_device_attr_serdev

此函数为4.14中新增加的函数,在4.4中uart_add_one_port直接调用了tty_register_device_attr。

struct device *tty_port_register_device_attr_serdev(struct tty_port *port,struct tty_driver *driver, unsigned index,struct device *device, void *drvdata,const struct attribute_group **attr_grp)
{
       struct device *dev;
       tty_port_link_device(port, driver, index);
       dev = serdev_tty_port_register(port, device, driver, index);
       if (PTR_ERR(dev) != -ENODEV) {
              /* Skip creating cdev if we registered a serdev device */
              return dev;
       }
       return tty_register_device_attr(driver, index, device, drvdata,attr_grp);
}

3.6 tty_cdev_add

static int tty_cdev_add(struct tty_driver *driver, dev_t dev, unsigned int index, unsigned int count)
{
       int err;
       /* init here, since reused cdevs cause crashes */
       driver->cdevs[index] = cdev_alloc();
       if (!driver->cdevs[index])
              return -ENOMEM;
       //关联设备操作函数
       driver->cdevs[index]->ops = &tty_fops;
       driver->cdevs[index]->owner = driver->owner;
       //向内核添加字符设备,之后用户空间通过/dev/ttySn节点open时,会调用tty_fops成员函数
       err = cdev_add(driver->cdevs[index], dev, count);
       if (err)
              kobject_put(&driver->cdevs[index]->kobj);
       return err;

}

 

3.7 tty_port_init

初始化port相关的数据成员,主要是buf相关。

void tty_port_init(struct tty_port *port)
{
    memset(port, 0, sizeof(*port));
    tty_buffer_init(port);
    struct tty_bufhead *buf = &port->buf;
    tty_buffer_reset(&buf->sentinel, 0);
    buf->head = &buf->sentinel;
    buf->tail = &buf->sentinel;
    init_llist_head(&buf->free);
    atomic_set(&buf->mem_used, 0);
    atomic_set(&buf->priority, 0);
    //驱动上报数据时,将数据推送到线路规程层通过flush_to_ldisc实现
    INIT_WORK(&buf->work, flush_to_ldisc);
    buf->mem_limit = TTYB_DEFAULT_MEM_LIMIT;
    init_waitqueue_head(&port->open_wait);
    init_waitqueue_head(&port->delta_msr_wait);
    mutex_init(&port->mutex);
    mutex_init(&port->buf_mutex);
    spin_lock_init(&port->lock);
    port->close_delay = (50 * HZ) / 100;
    port->closing_wait = (3000 * HZ) / 100;
}

 

3.7 tty_open

之前有提到,在tty_cdev_add函数中,将设备文件与tty_fops进行了关联。在用户通过设备节点打开tty设备时,会调用到tty_open函数。

static int tty_open(struct inode *inode, struct file *filp)

{
       struct tty_struct *tty;
       int noctty, retval;
       dev_t device = inode->i_rdev;
       unsigned saved_flags = filp->f_flags;
       nonseekable_open(inode, filp);
retry_open:
       retval = tty_alloc_file(filp);
       if (retval)
              return -ENOMEM;
        //tty的获取部分未理清,tty在哪里申请的内存,怎么和tty_driver关联的?
        //在uart_register_driver时,会创建tty_struct,之后与tty_driver,tty_ldisc等结构体进行关联,具体可以参考tty_init_dev函数的具体实现
       tty = tty_open_current_tty(device, filp);
       if (!tty)
              tty = tty_open_by_driver(device, inode, filp);
       if (IS_ERR(tty)) {
              tty_free_file(filp);
              retval = PTR_ERR(tty);
              if (retval != -EAGAIN || signal_pending(current))
                     return retval;
              schedule();
              goto retry_open;
       }
       tty_add_file(tty, filp);
       check_tty_count(tty, __func__);
       tty_debug_hangup(tty, "opening (count=%d)\n", tty->count);
            //调用到uart_ops中的open成员
       if (tty->ops->open)
              retval = tty->ops->open(tty, filp);
       else
              retval = -ENODEV;
       filp->f_flags = saved_flags;
       if (retval) {
              tty_debug_hangup(tty, "open error %d, releasing\n", retval);
              tty_unlock(tty); /* need to call tty_release without BTM */
              tty_release(inode, filp);
              if (retval != -ERESTARTSYS)
                     return retval;
              if (signal_pending(current))
                     return retval;
              schedule();
              /*
               * Need to reset f_op in case a hangup happened.
               */
              if (tty_hung_up_p(filp))
                     filp->f_op = &tty_fops;
              goto retry_open;
       }
       clear_bit(TTY_HUPPED, &tty->flags);
       noctty = (filp->f_flags & O_NOCTTY) ||
               (IS_ENABLED(CONFIG_VT) && device == MKDEV(TTY_MAJOR, 0)) ||
               device == MKDEV(TTYAUX_MAJOR, 1) ||
               (tty->driver->type == TTY_DRIVER_TYPE_PTY &&
               tty->driver->subtype == PTY_TYPE_MASTER);
       if (!noctty)
              tty_open_proc_set_tty(filp, tty);
       tty_unlock(tty);
       return 0;
}

 

3.8 tty_write

static ssize_t tty_write(struct file *file, const char __user *buf,size_t count, loff_t *ppos)
{
    struct tty_struct *tty = file_tty(file);
    if (!ld->ops->write) ret = -EIO;
    else ret = do_tty_write(ld->ops->write, tty, file, buf, count);
    if (tty->write_cnt < chunk)
    {
        buf_chunk = kmalloc(chunk, GFP_KERNEL);
        kfree(tty->write_buf);
        tty->write_cnt = chunk;
        tty->write_buf = buf_chunk;
        //这里调用到ld->ops->wirte函数,即为线路规程中的n_tty_write。
        ret = write(tty, file, tty->write_buf, size);
    }
}

 

3.9 tty_read

用户通过read系统调用执行到tty_read,之后调用到线路规程层的n_tty_read函数。

static ssize_t tty_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
{
    struct inode *inode = file_inode(file);
    struct tty_struct *tty = file_tty(file);
    //调用线程规程层的read函数,即n_tty_read
    if (ld->ops->read) i = ld->ops->read(tty, file, buf, count);
}

3.10 tty_insert_filp_char

当前的tty_buffer空间不够时调用tty_insert_flip_string_flags,在这个函数里会去查找下一个tty_buffer,并将数据放到下一个tty_buffer的char_buf_ptr里。这里char_buf_ptr的数据是如何放到线路规程的read_buf中的呢?是在tty open操作的时候,tty_init_dev -> initialize_tty_struct -> initialize_tty_struct -> tty_buffer_init

int tty_insert_flip_char(struct tty_port *port, unsigned char ch, char flag)
{
       struct tty_buffer *tb = port->buf.tail;
       int change;
       change = (tb->flags & TTYB_NORMAL) && (flag != TTY_NORMAL);
       if (!change && tb->used < tb->size) {
              if (~tb->flags & TTYB_NORMAL)
                     *flag_buf_ptr(tb, tb->used) = flag;
              *char_buf_ptr(tb, tb->used++) = ch;
              return 1;
       }
       return __tty_insert_flip_char(port, ch, flag);
}

 

3.11 tty_filp_buf_push

将终端缓冲区推送到线路规程层。可以在IRQ/原子上下文中调用。如果队列忙,工作队列将会暂停并稍候进行重试。

void tty_flip_buffer_push(struct tty_port *port)
{
      tty_schedule_flip(port);
           struct tty_bufhead *buf = &port->buf;
           smp_store_release(&buf->tail->commit, buf->tail->used);
           //此处调用到tty_port_init时初关联的flush_to_ldisc
           queue_work(system_unbound_wq, &buf->work);
}

tty_schedule_flip:
{
    struct tty_bufhead *buf = &port->buf;
    smp_store_release(&buf->tail->commit, buf->tail->used);
    //此处调用到tty_port_init时初关联的flush_to_ldisc
    queue_work(system_unbound_wq, &buf->work);
}

 

3.12 flush_to_ldisc

从软中断中调用此例程,将数据从缓冲链表中推送到线路规程层。对于每个tty实例,receive_buf方法都是单线程的。驱动层通过调用tty_filp_buf_push唤醒buf->work,会调用到此函数。

static void flush_to_ldisc(struct work_struct *work)
{
    struct tty_port *port = container_of(work, struct tty_port, buf.work);
    struct tty_bufhead *buf = &port->buf;
    while(1)
    {
        struct tty_buffer *head = buf->head;
        struct tty_buffer *next;
        next = smp_load_acquire(&head->next);
        count = smp_load_acquire(&head->commit) - head->read;
        if (!count) {
            if (next == NULL)
                break;
            buf->head = next;
            tty_buffer_free(port, head);
            continue;
        }
        count = receive_buf(port, head, count);
        //receive_buf调用了client_ops中的成员函数,即为defatlu_client_ops的成员函数
        return port->client_ops->receive_buf(port, p, f, count);
        if (!count) break;
    head->read += count;
}

 

3.13 tty_port_default_receive_buf

static int tty_port_default_receive_buf(struct tty_port *port, const unsigned char *p, const unsigned char *f, size_t count)
{
	int ret;
	struct tty_struct *tty;
	struct tty_ldisc *disc;

	tty = READ_ONCE(port->itty);
	if (!tty)
		return 0;

	disc = tty_ldisc_ref(tty);
	if (!disc)
		return 0;

	ret = tty_ldisc_receive_buf(disc, p, (char *)f, count);

	tty_ldisc_deref(disc);

	return ret;
}

int tty_ldisc_receive_buf(struct tty_ldisc *ld, const unsigned char *p,  char *f, int count)
{
	if (ld->ops->receive_buf2)
		count = ld->ops->receive_buf2(ld->tty, p, f, count);
	else {
		count = min_t(int, count, ld->tty->receive_room);
		if (count && ld->ops->receive_buf)
            //由tty核心层调用到线路规程层,对应到n_tty_ops的相关成员函数
			ld->ops->receive_buf(ld->tty, p, f, count);
	}
	return count;
}

4 UART核心层

4.1 uart_register_driver

使用UART核心层注册驱动程序。依次向TTY核心层注册,并初始化核心驱动程序pre-port状态;

int uart_register_driver(struct uart_driver *drv)
{
	struct tty_driver *normal;
	int i, retval;

	BUG_ON(drv->state);

	/*
	 * Maybe we should be using a slab cache for this, especially if
	 * we have a large number of ports to handle.
	 */
    //内存申请
	drv->state = kzalloc(sizeof(struct uart_state) * drv->nr, GFP_KERNEL);
	if (!drv->state)
		goto out;
    //分配tty_driver内存
	normal = alloc_tty_driver(drv->nr);
	if (!normal)
		goto out_kfree;

	drv->tty_driver = normal;

	normal->driver_name	= drv->driver_name;
	normal->name		= drv->dev_name;
	normal->major		= drv->major;
	normal->minor_start	= drv->minor;
	normal->type		= TTY_DRIVER_TYPE_SERIAL;
	normal->subtype		= SERIAL_TYPE_NORMAL;
	normal->init_termios	= tty_std_termios;
	normal->init_termios.c_cflag = B9600 | CS8 | CREAD | HUPCL | CLOCAL;
	normal->init_termios.c_ispeed = normal->init_termios.c_ospeed = 9600;
    //标志设置,后面会用到
	normal->flags		= TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV; 
	normal->driver_state    = drv;
    //tty_driver->ops = uart_ops,关联操作函数
	tty_set_operations(normal, &uart_ops);

	/*
	 * Initialise the UART state(s).
	 */
	for (i = 0; i < drv->nr; i++) {
		struct uart_state *state = drv->state + i;
		struct tty_port *port = &state->port;
        //初始化port->buf,及port->buf.work,在uart收到数据上报时,会使用port->buf以及work
		tty_port_init(port);
		port->ops = &uart_port_ops; //uart_driver->state->port->ops = uart_port_ops;
	}
    //注册到TTY核心层
	retval = tty_register_driver(normal);
	if (retval >= 0)
		return retval;

	for (i = 0; i < drv->nr; i++)
		tty_port_destroy(&drv->state[i].port);
	put_tty_driver(normal);
out_kfree:
	kfree(drv->state);
out:
	return -ENOMEM;
}

4.2 uart_add_one_port

attach驱动程序定义的端口结构。这允许驱动程序使用核心驱动程序注册自己的uart_port结构,主要目的是允许低级别的uart驱动程序扩展uart_port,而不是拥有更多级别的结构。

int uart_add_one_port(struct uart_driver *drv, struct uart_port *uport)
{
	struct uart_state *state;
	struct tty_port *port;
	int ret = 0;
	struct device *tty_dev;
	int num_groups;

	BUG_ON(in_interrupt());

	if (uport->line >= drv->nr)
		return -EINVAL;

	state = drv->state + uport->line;
	port = &state->port;

	mutex_lock(&port_mutex);
	mutex_lock(&port->mutex);
	if (state->uart_port) {
		ret = -EINVAL;
		goto out;
	}

	/* Link the port to the driver state table and vice versa */
	atomic_set(&state->refcount, 1);
	init_waitqueue_head(&state->remove_wait);
	state->uart_port = uport;
	uport->state = state;

	state->pm_state = UART_PM_STATE_UNDEFINED;
	uport->cons = drv->cons;
	uport->minor = drv->tty_driver->minor_start + uport->line;
	uport->name = kasprintf(GFP_KERNEL, "%s%d", drv->dev_name,
				drv->tty_driver->name_base + uport->line);
	if (!uport->name) {
		ret = -ENOMEM;
		goto out;
	}

	/*
	 * If this port is a console, then the spinlock is already
	 * initialised.
	 */
	if (!(uart_console(uport) && (uport->cons->flags & CON_ENABLED))) {
		spin_lock_init(&uport->lock);
		lockdep_set_class(&uport->lock, &port_lock_key);
	}
	if (uport->cons && uport->dev)
		of_console_check(uport->dev->of_node, uport->cons->name, uport->line);

	uart_configure_port(drv, state, uport);

	port->console = uart_console(uport);

	num_groups = 2;
	if (uport->attr_group)
		num_groups++;

	uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups),
				    GFP_KERNEL);
	if (!uport->tty_groups) {
		ret = -ENOMEM;
		goto out;
	}
	uport->tty_groups[0] = &tty_dev_attr_group;
	if (uport->attr_group)
		uport->tty_groups[1] = uport->attr_group;

	/*
	 * Register the port whether it's detected or not.  This allows
	 * setserial to be used to alter this port's parameters.
	 */
	tty_dev = tty_port_register_device_attr_dev(port, drv->tty_driver,
			uport->line, uport->dev, port, uport->tty_groups);
	if (likely(!IS_ERR(tty_dev))) {
		device_set_wakeup_capable(tty_dev, 1);
	} else {
		dev_err(uport->dev, "Cannot register tty device on line %d\n",
		       uport->line);
	}

	/*
	 * Ensure UPF_DEAD is not set.
	 */
	uport->flags &= ~UPF_DEAD;

 out:
	mutex_unlock(&port->mutex);
	mutex_unlock(&port_mutex);

	return ret;
}
    1. uart_config_port

static void uart_configure_port(struct uart_driver *drv, struct uart_state *state,

                  struct uart_port *port)

{

       unsigned int flags;

 

       /*

        * If there isn't a port here, don't do anything further.

        */

       if (!port->iobase && !port->mapbase && !port->membase)

              return;

 

       /*

        * Now do the auto configuration stuff.  Note that config_port

        * is expected to claim the resources and map the port for us.

        */

       flags = 0;

       if (port->flags & UPF_AUTO_IRQ)

              flags |= UART_CONFIG_IRQ;

       if (port->flags & UPF_BOOT_AUTOCONF) {

              if (!(port->flags & UPF_FIXED_TYPE)) {

                     port->type = PORT_UNKNOWN;

                     flags |= UART_CONFIG_TYPE;

              }

              port->ops->config_port(port, flags); //调用serial_sprd_ops中的sprd_config_port函数

       }

 

       if (port->type != PORT_UNKNOWN) {

              unsigned long flags;

                        //给出调试信息

              uart_report_port(drv, port);

 

              /* Power up port for set_mctrl() */

              uart_change_pm(state, UART_PM_STATE_ON);

 

              /*

               * Ensure that the modem control lines are de-activated.

               * keep the DTR setting that is set in uart_set_options()

               * We probably don't need a spinlock around this, but

               */

              spin_lock_irqsave(&port->lock, flags);

              port->ops->set_mctrl(port, port->mctrl & TIOCM_DTR);

              spin_unlock_irqrestore(&port->lock, flags);

 

              /*

               * If this driver supports console, and it hasn't been

               * successfully registered yet, try to re-register it.

               * It may be that the port was not available.

               */

                         //sprd_console的定义中,flasgCON_PRINTBUFFER,此处会注册sprd_console,这里会调用到printk.c中的函数,暂不分析

              if (port->cons && !(port->cons->flags & CON_ENABLED))

                     register_console(port->cons);

 

              /*

               * Power down all ports by default, except the

               * console if we have one.

               */

              if (!uart_console(port))

                     uart_change_pm(state, UART_PM_STATE_OFF);

       }

}

    1. uart_open

调用在tty_open流程中,会调用到ops->open函数,此处ops赋值是在前面的uart_register_driver函数中,所以进入uart_ops结构体的open函数,这里就是从tty核心转到serial核心,往下走了一层。uart_open函数会等待open_wait完成。同时在4.4版本中,uart_open会调用uart_startup,而4.14中没有调用,那uart_startup是在哪里调用的呢?

static int uart_open(struct tty_struct *tty, struct file *filp)

struct uart_driver *drv = tty->driver->driver_state;

int retval, line = tty->index;

struct uart_state *state = drv->state + line;

tty->driver_data = state;

retval = tty_port_open(&state->port, tty, filp);

    ++port->count;

    tty_port_tty_set(port, tty)

          port->tty = tty_kref_get(tty);  :此处会导致port->itty与port->tty指向同一个tty_struct,为什么

    //是否会调用到activate呢?port->ops中实现了activate成员函数,推测是会调用的,需要分析

    if (port->ops->activate) { int retval = port->ops->activate(port, tty);}

return tty_port_block_til_ready(port, tty, filp);//此函数暂不做展开

    1. uart_port_activate

 

int uart_port_activate(struct tty_port *port, struct tty_struct *tty)

     struct uart_state *state = container_of(port, struct uart_state, port);

     uport = uart_port_check(state);

     return uart_startup(tty, state, 0);

    1. uart_startup

 

int uart_startup(struct tty_struct *tty, struct uart_state *state,  int init_hw)

     struct tty_port *port = &state->port;

     retval = uart_port_startup(tty, state, init_hw);

               uart_change_pm(state, UART_PM_STATE_ON);

               retval = uport->ops->startup(uport);//调用驱动层的startupsprd_startup

               uart_change_speed(tty, state, NULL);   //会调用到uport->ops->set_termios函数

 

    1. uart_write

此函数将数据放到到缓冲区中,之后调用驱动层实现的uart_ops结构体:serial_sprd_ops中的成员函数。

static int uart_write(struct tty_struct *tty,
const unsigned char *buf, int count)
struct uart_state *state = tty->driver_data;
circ = &state->xmit;
port = uart_port_lock(state, flags); //port = state->uart_port
memcpy(circ->buf + circ->head, buf, c);
circ->head = (circ->head + c) & (UART_XMIT_SIZE - 1);
__uart_start(tty);
    struct uart_state *state = tty->driver_data;
    struct uart_port *port = state->uart_port;
    //此处调用到驱动层的serial_sprd_ops中的sprd_start_tx函数
    port->ops->start_tx(port);

 

    1. uart_insert_char

void uart_insert_char(struct uart_port *port, unsigned int status,

               unsigned int overrun, unsigned int ch, unsigned int flag)

{

       struct tty_port *tport = &port->state->port;

       if ((status & port->ignore_status_mask & ~overrun) == 0)

              if (tty_insert_flip_char(tport, ch, flag) == 0)

                     ++port->icount.buf_overrun;

       if (status & ~port->ignore_status_mask & overrun)

              if (tty_insert_flip_char(tport, 0, TTY_OVERRUN) == 0)

                     ++port->icount.buf_overrun;

}

 

  1. 线路规程

ine discipline(LDISC) 线路规程,是linux和类unix系统终端子系统的一个软件驱动层。line discipline把TTY核心层和tty driver关联在一起,策略的分离使得TTY核心层和tty driver不需要关注数据语法处理,tty driver可以被相同的硬件复用,而只需更改line discipline。

linux终端设备缺省的线路规程是N_TTY,它从tty driver和应用程序接收数据,按照终端设置处理数据。对于输入数据,它处理特殊的中断字符(比如Control-C),删除字符(backspace, delete)等等;对于输出数据,它用CR/LF序列替换LF字符。当uart port用做普通串口时,使用N_TTY线路规程。当uart port设备用做serial modem 的internet拨号连接时,使用PPP线路规程处理数据;ppp LDISC把从uart core来的串口数据组装为PPP输入packet,然后分发给网络协议栈;ppp LDISC把从网络协议栈发送来的数据拆包发送给uart port。

 

    1. tty _register_ldisc

向内核注册一个新的线路规程,默认的线路规程是在初始化阶段完成注册的,索引为0。默认的tty_ldiscs数据长度为30,在tty.h中定义了各种设备的索引值,目前内核使用了27个。

Int tty_register_ldisc(int disc, struct tty_ldisc_ops *new_ldisc)

      tty_ldiscs[disc] = new_ldisc;

      new_ldisc->num = disc;

      new_ldisc->refcount = 0;

 

    1. tty_ldisc_init

线程规程初始化的相关函数是在tty_open时才会调用,在注册流程中并未发现申请相关结构体及初始化的地方。此函数主要完成新tty的ldisc的内存申请及设置:为新申请的tty_struct设置线程规程对象,调用此函数时,tty_struct结构还未完全设置。初始化时,传递的N_TTY,在tty.h中定义为0,最终会从全局的tty_ldiscs中取出对应的tty_ldisc_ops进行初始化。

int tty_ldisc_init(struct tty_struct *tty)

      struct tty_ldisc *ld = tty_ldisc_get(tty, N_TTY);

             ldops = get_ldops(disc);

      tty->ldisc = ld;

    1. tty_ldisc_get

 

static struct tty_ldisc *tty_ldisc_get(struct tty_struct *tty, int disc)

     ldops = get_ldops(disc);

     ld = kmalloc(sizeof(struct tty_ldisc), GFP_KERNEL | __GFP_NOFAIL);

     ld->ops = ldops;

     ld->tty = tty;

    1. get_ldops

从全局数组tty_ldiscs中,根据索引取出tty_ldisc_ops

struct tty_ldisc_ops *get_ldops(int disc)

         ldops = tty_ldiscs[disc];

    1. n_tty_write

终端设备的写功能函数。对数据进行处理,之后调用到uart_ops中的uart_write函数

static ssize_t n_tty_write(struct tty_struct *tty, struct file *file, const unsigned char *buf, size_t nr)

      DEFINE_WAIT_FUNC(wait, woken_wake_function);

      down_read(&tty->termios_rwsem);

      process_echoes(tty);

      add_wait_queue(&tty->write_wait, &wait);

      //此处两种写法有什么区别,需要进行分析,不过最终都会调用到tty->ops->write,即为uart_ops中的uart_write

      if (O_OPOST(tty)){……}

      else{ c = tty->ops->write(tty, b, nr); }

    1. n_tty_receive_buf

n_tty_receice_buf和n_tty_receive_buf2都是通过调用n_tty_receive_buf_common实现。n_tty_receice_buf会检查room,而n_tty_receice_buf2不会检查room。

int n_tty_receive_buf_common(struct tty_struct *tty, const unsigned char *cp, char *fp, int count, int flow)

struct n_tty_data *ldata = tty->disc_data;

__receive_buf(tty, cp, fp, n);

        //receice_buf最终调用到put_tty_queue,从read_buf中读取数据

             put_tty_queue

                   *read_buf_addr(ldata, ldata->read_head) = c;

                           return &ldata->read_buf[i & (N_TTY_BUF_SIZE - 1)];

             ldata->read_head++;

        wake_up_interruptible_poll(&tty->read_wait, POLLIN);

    tty->receive_room = room;

 

    1. n_tty_read

ssize_t n_tty_read(struct tty_struct *tty, struct file *file, unsigned char __user *buf, size_t nr)

struct n_tty_data *ldata = tty->disc_data;

unsigned char __user *b = buf;

DEFINE_WAIT_FUNC(wait, woken_wake_function);

c = job_control(tty, file);

packet = tty->packet;

tail = ldata->read_tail;

add_wait_queue(&tty->read_wait, &wait);

uncopied = copy_from_read_buf(tty, &b, &nr);

uncopied += copy_from_read_buf(tty, &b, &nr);

           //将数据从ldata->read_buf中读出来传递到用户层

const unsigned char *from = read_buf_addr(ldata, tail);

          return &ldata->read_buf[i & (N_TTY_BUF_SIZE - 1)];

retval = copy_to_user(*b, from, n);

 

    1. tty_ldisc_setup

在初次打开tty/pty时调用,用来设置线程规程并绑定到tty。

int tty_ldisc_setup(struct tty_struct *tty, struct tty_struct *o_tty)

      int retval = tty_ldisc_open(tty, tty->ldisc);

                if (ld->ops->open) {

                    //调用到线路规程层中n_tty_open

                    ret = ld->ops->open(tty);

}

 

    1. n_tty_open

打开一个ldisc。在线路规程与终端设备匹配时调用。允许睡眠。

static int n_tty_open(struct tty_struct *tty)

       struct n_tty_data *ldata;

       ldata = vmalloc(sizeof(*ldata));

       tty->disc_data = ldata;

reset_buffer_flags(tty->disc_data);

       n_tty_set_termios(tty, NULL);

 

  1. 驱动层

本文以展讯平台串口驱动为例做说明。

    1. sprd_probe

个人习惯从probe函数进行分析。此处为设备驱动匹配的第一步,在probe函数中使用到的结构体如下:

 

static int sprd_probe(struct platform_device *pdev)

{

       struct uart_port *up;

       ……

       //找到空闲的结构体指针

       for (index = 0; index < ARRAY_SIZE(sprd_port); index++)

              if (sprd_porst[index] == NULL)

                     break;

 

       //申请内存空间

       sprd_port[index] = devm_kzalloc(&pdev->dev,

              sizeof(*sprd_port[index]), GFP_KERNEL);

       ……

       //初始化uart_port

       up = &sprd_port[index]->port;

       up->dev = &pdev->dev;

       up->line = index;

       up->type = PORT_SPRD;

       up->iotype = UPIO_MEM;

       up->uartclk = SPRD_DEF_RATE;

       up->fifosize = SPRD_FIFO_SIZE;

       up->ops = &serial_sprd_ops;

       up->flags = UPF_BOOT_AUTOCONF;

 

       ……

       up->mapbase = res->start;

       up->membase = devm_ioremap_resource(&pdev->dev, res);

       ……

       //probe时,当sprd_ports_num为零时,需要重新注册sprd_uart_driver

       //如果有设备remove,在remove时,如果sprd_ports_num为零,则卸载sprd_uart_driver

       if (!sprd_ports_num) {

              //注册sprd_uart_driver驱动

              ret = uart_register_driver(&sprd_uart_driver);

             uart_register_driver调用到tty_register_driver,会在proc下创建节点,之后返回。按我的理解,tty_register_driver是注册了一个tty的驱动,这个驱动有了逻辑能力,但是这个时候这个驱动还没有对应任何设备,所以后续还要添加对应的端口(也就是芯片的物理串口),并创建/dev/下的设备节点,上层用tty_driver驱动的逻辑来操作对应的端口

              ……

       }

       sprd_ports_num++;

       ret = uart_add_one_port(&sprd_uart_driver, up);

       ……

       platform_set_drvdata(pdev, up);

       return ret;

}

 

通过代码可以看到,sprd_probe函数完成以下任务:

  1. 申请uart_port空间,初始化uart_port,将uart_port->ops与驱动空间的处理函数serial_sprd_ops进行关联;
  2. 调用uart_register_driver注册驱动
  3. 调用uart_add_one_port添加port
    1. sprd_handle_irq

数据的实际发送是通过设置对应的TX/RX中断enable来启动。中断在sprd_startup函数中进行注册ret = devm_request_irq(port->dev, port->irq, sprd_handle_irq, IRQF_SHARED, sp->name, port);中断处理中,通过对中断位的状态进行收发处理。

static irqreturn_t sprd_handle_irq(int irq, void *dev_id)

struct uart_port *port = dev_id;

ims = serial_in(port, SPRD_IMSR);

if (ims & SPRD_IMSR_TIMEOUT)

serial_out(port, SPRD_ICLR, SPRD_ICLR_TIMEOUT);

    if (ims & (SPRD_IMSR_RX_FIFO_FULL | SPRD_IMSR_BREAK_DETECT | SPRD_IMSR_TIMEOUT))

            sprd_rx(port);

if (ims & SPRD_IMSR_TX_FIFO_EMPTY)

sprd_tx(port);

 

 

    1. sprd_tx

 

实际数据发送函数

void sprd_tx(struct uart_port *port)

{

    struct circ_buf *xmit = &port->state->xmit;

    int count;

 

    if (port->x_char) {

       serial_out(port, SPRD_TXD, port->x_char);

       port->icount.tx++;

       port->x_char = 0;

       return;

    }

 

    if (uart_circ_empty(xmit) || uart_tx_stopped(port)) {

       sprd_stop_tx(port);

       return;

    }

 

    count = THLD_TX_EMPTY;

    do {

       serial_out(port, SPRD_TXD, xmit->buf[xmit->tail]);

       xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);

       port->icount.tx++;

       if (uart_circ_empty(xmit))

           break;

    } while (--count > 0);

 

    if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)

       uart_write_wakeup(port);

 

    if (uart_circ_empty(xmit))

       sprd_stop_tx(port);

}

 

    1. sprd_rx

根据硬件状态,接着uart_insert_char来处理该字符,把这个数据放到uart层。

void sprd_rx(struct uart_port *port)

{

       struct tty_port *tty = &port->state->port;

       unsigned int ch, flag, lsr, max_count = SPRD_TIMEOUT;

 

       while ((serial_in(port, SPRD_STS1) & 0x00ff) && max_count--) {

              lsr = serial_in(port, SPRD_LSR);

              ch = serial_in(port, SPRD_RXD);

              flag = TTY_NORMAL;

              port->icount.rx++;

 

              if (lsr & (SPRD_LSR_BI | SPRD_LSR_PE |

                     SPRD_LSR_FE | SPRD_LSR_OE))

                     if (handle_lsr_errors(port, &lsr, &flag))

                            continue;

              if (uart_handle_sysrq_char(port, ch))

                     continue;

                         //将数据传输到uart

              uart_insert_char(port, lsr, SPRD_LSR_OE, ch, flag);

       }

            //将数据传输到线路规程

       tty_flip_buffer_push(tty);

}

 

    1. sprd_start_tx

函数中并未进行实际数据发送,而是查看TX_EMPTY中断是否使能,如果未使能,enable。之后在uart中断处理函数中,完成实际数据的发送。

static void sprd_start_tx(struct uart_port *port)

     ien = serial_in(port, SPRD_IEN);

if (!(ien & SPRD_IEN_TX_EMPTY)) {

ien |= SPRD_IEN_TX_EMPTY;

serial_out(port, SPRD_IEN, ien);

}

 

  1. 流程分析
    1. register流程

static int sprd_probe(struct platform_device *pdev)

ret = uart_register_driver(&sprd_uart_driver);

     tty_set_operations(normal, &uart_ops);

     tty_port_init(port);

port->ops = &uart_port_ops;

retval = tty_register_driver(normal)

    error = alloc_chrdev_region(&dev, driver->minor_start, driver->num, driver->name);

    error = tty_cdev_add(driver, dev, 0, driver->num);

        driver->cdevs[index]->ops = &tty_fops;

        err = cdev_add(driver->cdevs[index], dev, count);

    list_add(&driver->tty_drivers, &tty_drivers);

    d = tty_register_device(driver, i, NULL);

         return tty_register_device_attr(driver, index, device, NULL, NULL);

             dev_t devt = MKDEV(driver->major, driver->minor_start) + index;
                          tty_line_name(driver, index, name);
                          dev = kzalloc(sizeof(*dev), GFP_KERNEL);
                          dev_set_drvdata(dev, drvdata);
                          retval = device_register(dev);
                          retval = tty_cdev_add(driver, devt, index, 1);

    proc_tty_register_driver(driver);

ent = proc_create_data(driver->driver_name, 0, proc_tty_driver,

driver->ops->proc_fops, driver);

driver->proc_entry = ent;

ret = uart_add_one_port(&sprd_uart_driver, up);

     state = drv->state + uport->line;

port = &state->port;

uart_configure_port(drv, state, uport);

uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups), GFP_KERNEL);

uport->tty_groups[0] = &tty_dev_attr_group;

tty_dev = tty_port_register_device_attr_serdev(port, drv->tty_driver,

                    uport->line, uport->dev, port, uport->tty_groups);

    tty_port_link_device(port, driver, index);

         driver->ports[index] = port;

    dev = serdev_tty_port_register(port, device, driver, index);

   //两次调用,两次有什么差别?

return tty_register_device_attr(driver, index, device, drvdata, attr_grp); 

 

    1. open流程

alloc_chrdev_region是动态分配了主从设备号,接着cdev_init,它file_operations结构提和它关联了起来,以后我们open /dev/ttyS节点时候会调用他的tty_fops的open成员函数。

static int tty_open(struct inode *inode, struct file *filp)

retval = tty_alloc_file(filp);

tty = tty_open_current_tty(device, filp); 

if (!tty) tty = tty_open_by_driver(device, inode, filp);

        tty = tty_driver_lookup_tty(driver, filp, index);

        if (tty) { ……}

        else { tty = tty_init_dev(driver, index); }  //此处申请struct tty_strcut,并关联注册时涉及到的结构体,并调用uart_startup函数

                tty = alloc_tty_struct(driver, idx);

                        tty->magic = TTY_MAGIC;

                      //TTY核心层转移到线路规程层

                     if (tty_ldisc_init(tty)){……}

INIT_WORK(&tty->hangup_work, do_tty_hangup);

                        INIT_WORK(&tty->SAK_work, do_SAK_work);

                        tty->driver = driver;

                        tty->ops = driver->ops;

                        tty->index = idx;

                        tty_line_name(driver, idx, tty->name);

                        tty->dev = tty_get_device(tty);

                                dev_t devt = tty_devnum(tty);

                                return class_find_device(tty_class, NULL, &devt, dev_match_devt);

                retval = tty_driver_install_tty(driver, tty);

                       //uart_ops没有实现install成员函数,故调用tty_standard_install

                       return driver->ops->install ? driver->ops->install(driver, tty) : tty_standard_install(driver, tty);

                                 driver->ttys[tty->index] = tty;

                if (!tty->port)  tty->port = driver->ports[idx];

                retval = tty_ldisc_setup(tty, tty->link);  ---- 会调用到ld->ops->open,即调用线路规程中的n_tty_open

tty_add_file(tty, filp);

      struct tty_file_private *priv = file->private_data;

      priv->tty = tty;

      priv->file = file;

      list_add(&priv->list, &tty->tty_files);

if (tty->ops->open) retval = tty->ops->open(tty, filp);  à调用uart_ops中的uart_open

    1. write流程

用户层的write函数入口为tty_write

static ssize_t tty_write(struct file *file, const char __user *buf,

                                                        size_t count, loff_t *ppos)

struct tty_struct *tty = file_tty(file);

if (!ld->ops->write) ret = -EIO;

else ret = do_tty_write(ld->ops->write, tty, file, buf, count);

      if (tty->write_cnt < chunk)

      buf_chunk = kmalloc(chunk, GFP_KERNEL);

      kfree(tty->write_buf);

      tty->write_cnt = chunk;

      tty->write_buf = buf_chunk;

      //这里调用到ld->ops->wirte函数,即为线路规程中的n_tty_write

      ret = write(tty, file, tty->write_buf, size);

 

    1. read流程

用户层read函数入口为tty_read

  1. 其他杂项
    1. 控制台终端(/dev/ttyn, /dev/console)
在Linux系统中,电脑显示器通常被称为控制台终端
(Console)。他仿真了类型为Linux的一种终端(TERM=Linux),并且有一些设备特别文档和之相关联:tty0、tty1、tty2等。当您在控制台上登录时,使用的是tty1。使用Alt+[F1―F6]组合键时,我们就能够转换到tty2、tty3等上面去。tty1tty6等称为虚拟终端,而tty0则是当前所使用虚拟终端的一个别名,系统所产生的信息会发送到该终端上(这时也叫控制台终端)。因此不管当前正在使用哪个虚拟终端,系统信息都会发送到控制台终端上。您能够登录到不同的虚拟终端上去,因而能够让系统同时有几个不同的会话期存在。只有系统或终极用户root能够向/dev/tty0进行写操作。
    1. /dev/console是什么
/dev/console即控制台,是和操作系统交互的设备,系统将一些信息直接输出到控制台上。现在只有在单用户模式下,才允许用户登录控制台。
    1. 数据关系图

 

 

  1. 参考资料

https://www.xuebuyuan.com/1722895.html

http://developer.51cto.com/art/201209/357501.htm

https://www.cnblogs.com/easynote/p/3409449.html

https://blog.csdn.net/lizuobin2/article/details/51773305

 

 

AMessage ALooper AHandler 介绍https://blog.51cto.com/cthhqu/1283673

  • 0
    点赞
  • 14
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值