在ARM上调试xen、linux的时候printk和early_printk都是非常重要的方式。即使有DSTREAM,还是一定会用到屏幕或串口输出。但ARM平台上不像X86那样有bios里的基本的VGA支持,LCD一般都启动的比较靠后了。所以通过串口输出很重要。
在用xen里作为dom0跑linux的时候, 会遇到linux串口的输出不见了。原因是xen起来后自己要用,就对dom0屏蔽了串口。Linux要想通过串口输出,可以指定console=hvc0。
Linux里的early_printk是在内核刚加载的时候就基本上可以用了,一旦early_printk可以用,至少也可以松口气,说明后面可以调了。最怕的就是一上来屏幕一黑什么都没有。Early_printk是的配置是和平台相关的,不同的板子都不一样。首先要linux里面支持,通过指定串口控制芯片的MMIO基址,用直接写寄存器的方式输出。当然正常串口的驱动最后也会是这样,early_printk比较直接吧,因为linux刚加载的时候驱动框架什么的都还没有起来,比如在内核自解压缩的时候。
以xen 4.3为例,平台是arndale5250-K,芯片是三星的Exynos5250,两个Cortex A15。
xen\xen\drivers\char\exynos4210-uart.c
/* TODO: Parse UARTconfig from the command line */
static int __init exynos4210_uart_init(structdt_device_node *dev,
const void *data)
{
const char *config =data;
struct exynos4210_uart *uart;
int res;
u64 addr,size;
if ( strcmp(config,"") )
{
early_printk("WARNING: UART configuration is notsupported\n");
}
uart =&exynos4210_com;
/*uart->clock_hz = 0x16e3600; */
uart->baud =BAUD_AUTO;
uart->data_bits = 8;
uart->parity =PARITY_NONE;
uart->stop_bits = 1;
res = dt_device_get_address(dev,0, &addr, &size);
if ( res )
{
early_printk("exynos4210: Unable to retrieve the base"
"address of the UART\n");
returnres;
}
uart->regs =ioremap_nocache(addr,size);
if ( !uart->regs )
{
early_printk("exynos4210: Unable to map the UART memory\n");
return-ENOMEM;
}
res = dt_device_get_irq(dev,0, &uart->irq);
if ( res )
{
early_printk("exynos4210: Unable to retrieve the IRQ\n");
return res;
}
uart->vuart.base_addr=addr;
uart->vuart.size =size;
uart->vuart.data_off=UTXH;
uart->vuart.status_off=UTRSTAT;
uart->vuart.status =UTRSTAT_TXE |UTRSTAT_TXFE;
/* Register withgeneric serial driver. */
serial_register_uart(SERHND_DTUART, &exynos4210_uart_driver,uart);
dt_device_set_used_by(dev,DOMID_XEN);
return 0;
}
/* TODO: Parse UARTconfig from the command line */
static int __init exynos4210_uart_init(structdt_device_node *dev,
const void *data)
{
const char *config =data;
struct exynos4210_uart *uart;
int res;
u64 addr,size;
if ( strcmp(config,"") )
{
early_printk("WARNING: UART configuration is notsupported\n");
}
uart =&exynos4210_com;
/*uart->clock_hz = 0x16e3600; */
uart->baud =BAUD_AUTO;
uart->data_bits = 8;
uart->parity =PARITY_NONE;
uart->stop_bits = 1;
res = dt_device_get_address(dev,0, &addr, &size);
if ( res )
{
early_printk("exynos4210: Unable to retrieve the base"
"address of the UART\n");
returnres;
}
uart->regs =ioremap_nocache(addr,size);
if ( !uart->regs )
{
early_printk("exynos4210: Unable to map the UART memory\n");
return-ENOMEM;
}
res = dt_device_get_irq(dev,0, &uart->irq);
if ( res )
{
early_printk("exynos4210: Unable to retrieve the IRQ\n");
return res;
}
uart->vuart.base_addr=addr;
uart->vuart.size =size;
uart->vuart.data_off=UTXH;
uart->vuart.status_off=UTRSTAT;
uart->vuart.status =UTRSTAT_TXE |UTRSTAT_TXFE;
/* Register withgeneric serial driver. */
serial_register_uart(SERHND_DTUART, &exynos4210_uart_driver,uart);
dt_device_set_used_by(dev,DOMID_XEN);
return 0;
}
Xen里平台相关的串口驱动的Init函数。
res = dt_device_get_address(dev,0, &addr, &size);
串口设备的MMIO基址和大小通过FDT文件获得。这里得到的是物理地址。
uart->regs = ioremap_nocache(addr,size);
把串口的物理地址映射到xen的地址空间。在xen环境下就可以通过uart->regs操作串口了。
uart->vuart.base_addr=addr;
uart->vuart.size =size;
uart->vuart.data_off=UTXH;
uart->vuart.status_off=UTRSTAT;
uart->vuart.status =UTRSTAT_TXE|UTRSTAT_TXFE;
这里的vuart就是xen在模拟串口设备时要用到的参数。
/* Register withgeneric serial driver. */
serial_register_uart(SERHND_DTUART, &exynos4210_uart_driver,uart);
这句是xen注册串口驱动的。下面仔细看看它。
xen\xen\drivers\char\serial.c
void __initserial_register_uart(intidx,struct uart_driver*driver,
void *uart)
{
/* StoreUART-specific info. */
com[idx].driver =driver;
com[idx].uart =uart;
}
函数本身非常简单,不过在我看到这个函数之前,一直没搞懂com是怎样用的,一会看vuart的代码。
xen\xen\drivers\char\serial.c
static struct serial_portcom[SERHND_IDX+ 1] = {
[0 ... SERHND_IDX]= {
.rx_lock= SPIN_LOCK_UNLOCKED,
.tx_lock= SPIN_LOCK_UNLOCKED
}
};
xen\xen\include\xen\serial.h
/* 'Serial handles'are composed from the following fields. */
#define SERHND_IDX (3<<0)/* COM1, COM2, DBGP, DTUART? */
# define SERHND_COM1 (0<<0)
# define SERHND_COM2 (1<<0)
# define SERHND_DBGP (2<<0)
# define SERHND_DTUART (0<<0)/* Steal SERHND_COM1 value */
#define SERHND_HI (1<<2)/* Mux/demux each transferredchar by MSB. */
#define SERHND_LO (1<<3)/* Ditto, except that the MSB iscleared. */
#define SERHND_COOKED (1<<4)/* Newline/carriage-returntranslation? */
Xen一共有3个串口,SERHND_DTUART就看做com1吧,还不知道为什么不直接用com1表示。
现在知道vuart里的数据,如base和size这些是从哪来的了。确实是平台相关的,是具体的串口驱动提供的。不过因为用了FDT,所以最终base是在FDT里描述的那个。
回过头去看我最开始看的vuart.c
xen\xen\arch\arm\vuart.c
/*
* xen/arch/arm/vuart.c
*
* Virtual UART Emulator.
*
* This emulator uses the information fromdtuart. This is not intended to be
* a full emulation of an UART device. Ratherit is intended to provide a
* sufficient veneer of one that early code(such as Linux's boot time
* decompressor) which hardcodes outputdirectly to such a device are able to
* make progress.
*
* The minimal register set to emulate an UARTare:
* -Single byte transmit register
* -Single status register
*
* /!\ This device is not intended to beenumerable or exposed to the OS
* (e.g. via Device Tree).
*
* Julien Grall <julien.grall@linaro.org>
* Ian Campbell <ian.campbell@citrix.com>
* Copyright (c) 2012 Citrix Systems.
*
* This program is free software; you canredistribute it and/or modify
* it under the terms of the GNU General PublicLicense as published by
* the Free Software Foundation; either version2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope thatit will be useful,
* but WITHOUT ANY WARRANTY; without even theimplied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULARPURPOSE. See the
* GNU General Public License for more details.
*/
int domain_vuart_init(structdomain *d)
{
ASSERT(!d->domain_id);
d->arch.vuart.info =serial_vuart_info(SERHND_DTUART);
if ( !d->arch.vuart.info )
return0;
spin_lock_init(&d->arch.vuart.lock);
d->arch.vuart.idx = 0;
d->arch.vuart.buf =xzalloc_array(char,VUART_BUF_SIZE);
if ( !d->arch.vuart.buf )
return-ENOMEM;
return 0;
}
arch_domain_create()调用domain_vuart_init()
xen\xen\arch\arm\domain.c
/*
* Virtual UART is only used by linux earlyprintk and decompress code.
* Only use it for dom0 because the linuxkernel may not support
* multi-platform.
*/
if ( (d->domain_id== 0) && (rc =domain_vuart_init(d)))
goto fail;
domain_vuart_init初始化xen虚拟的串口,其实最重要的是把虚拟串口信息的放到domain里。
xen\xen\drivers\char\serial.c
const struct vuart_info*serial_vuart_info(intidx)
{
if ( (idx >= 0) && (idx<ARRAY_SIZE(com))&&
com[idx].driver&&com[idx].driver->vuart_info)
return com[idx].driver->vuart_info(&com[idx]);
return NULL;
}
这回再看就很清楚了,com里保存的是注册过的串口驱动。com[idx].driver->vuart_info(&com[idx])也很简单。
xen\xen\drivers\char\exynos4210-uart.c
static struct uart_driver__read_mostly exynos4210_uart_driver= {
.init_preirq = exynos4210_uart_init_preirq,
.init_postirq= exynos4210_uart_init_postirq,
.endboot = NULL,
.suspend = exynos4210_uart_suspend,
.resume = exynos4210_uart_resume,
.tx_ready = exynos4210_uart_tx_ready,
.putc = exynos4210_uart_putc,
.getc = exynos4210_uart_getc,
.irq = exynos4210_uart_irq,
.dt_irq_get = exynos4210_uart_dt_irq,
.vuart_info = exynos4210_vuart_info,
};
static const struct vuart_info *exynos4210_vuart_info(structserial_port*port)
{
struct exynos4210_uart *uart=port->uart;
return&uart->vuart;
}
顺便提下
xen\xen\include\xen\serial.h
struct serial_port {
/* Uart-driverparameters. */
struct uart_driver *driver;
void *uart;
enum serial_port_statestate;
/* Transmit databuffer (interrupt-driven uart). */
char *txbuf;
unsigned int txbufp,txbufc;
bool_t tx_quench;
int tx_log_everything;
/* Forcesynchronous transmit. */
int sync;
/* Receivercallback functions (asynchronous receivers). */
serial_rx_fn rx_lo,rx_hi, rx;
/* Receive databuffer (polling receivers). */
char rxbuf[serial_rxbufsz];
unsigned int rxbufp,rxbufc;
/* Serial I/O isconcurrency-safe. */
spinlock_t rx_lock,tx_lock;
};
Com的结构,注册串口驱动时,就填driver和uart,driver里是各种操作的函数指针,uart里是这个com的参数。
好了,就算是xen里已经初始化了串口驱动,domain里已经有了vuart的信息,作为dom0的linux怎样把early_printk的内容输出到串口里呢?在linux里要不要配置early_printk和板子相关的参数呢?参数又是什么?
当然要在linux里配early_printk的参数啊,否则early_printk都没办法用。要是用printk,也只能通过console=hvc0了。奇怪我怎么会问这个问题。参数,就是配置时的串口MMIO的物理地址。这样就是说当linux里调用early_printk时,early_printk会写物理地址。所以接下来的问题就是xen有没有把这段物理地址映射到dom0中?Linux在写这段物理地址的时候会发生什么?
这时候到时想到了在前面的函数exynos4210_uart_init里见到过这一句
dt_device_set_used_by(dev,DOMID_XEN);
意思是挺明显,说这个设备是让xen用的。但里面的实现无非是
xen\xen\include\xen\device_tree.h
static inline void dt_device_set_used_by(structdt_device_node *device,
domid_tused_by)
{
/* TODO: childrenmust inherit to the used_by thing */
device->used_by =used_by;
}
Used_by只是一个标志位。搜了下用到它的地方。
xen\xen\arch\arm\domain_build.c
static int handle_node(struct domain *d, structkernel_info *kinfo,
conststructdt_device_node*node)
{
static conststructdt_device_matchskip_matches[]__initconst =
{
DT_MATCH_COMPATIBLE("xen,xen"),
DT_MATCH_COMPATIBLE("xen,multiboot-module"),
DT_MATCH_COMPATIBLE("arm,psci"),
DT_MATCH_PATH("/cpus"),
DT_MATCH_TYPE("memory"),
{ /* sentinel*/ },
};
staticconststructdt_device_matchgic_matches[]__initconst =
{
DT_MATCH_GIC,
{ /* sentinel*/ },
};
static conststructdt_device_matchtimer_matches[]__initconst =
{
DT_MATCH_TIMER,
{ /* sentinel*/ },
};
const structdt_device_node*child;
int res;
const char *name;
const char *path;
path = dt_node_full_name(node);
DPRINT("handle %s\n",path);
/* Skip thesesnodes and the sub-nodes */
if ( dt_match_node(skip_matches,node) )
{
DPRINT(" Skip it(matched)\n");
return0;
}
if ( platform_device_is_blacklisted(node) )
{
DPRINT(" Skip it(blacklisted)\n");
return0;
}
/* Replace thesenodes with our own. Note that the original may be
* used_by DOMID_XEN so this check comesfirst. */
if ( dt_match_node(gic_matches,node) )
return make_gic_node(d,kinfo->fdt,node);
if ( dt_match_node(timer_matches,node) )
return make_timer_node(d,kinfo->fdt,node);
/* Skip nodesused by Xen */
if ( dt_device_used_by(node)==DOMID_XEN )
{
DPRINT(" Skip it(used by Xen)\n");
return0;
}
/*
* Some device doesn't need to be mapped inXen:
* -Memory: the guest will see a different view of memory. It will
* be allocated later.
* -Disabled device: Linux is able to cope with status="disabled"
* property. Therefore these device doesn't need to be mapped. This
* solution can be use later for pass through.
*/
if ( !dt_device_type_is_equal(node,"memory") &&
dt_device_is_available(node) )
{
res= map_device(d,node);
if ( res )
returnres;
}
/*
* The property "name" is used tohave a different name on older FDT
* version. We want to keep the nameretrieved during the tree
* structure creation, that is store in thenode path.
*/
name = strrchr(path,'/');
name = name ?name + 1:path;
res = fdt_begin_node(kinfo->fdt,name);
if ( res )
return res;
res = write_properties(d,kinfo,node);
if ( res )
return res;
for ( child =node->child;child !=NULL;child=child->sibling)
{
res= handle_node(d,kinfo,child);
if ( res )
returnres;
}
if ( node ==dt_host)
{
res= make_hypervisor_node(d,kinfo->fdt,node);
if ( res )
returnres;
res= make_psci_node(kinfo->fdt,node);
if ( res )
returnres;
res= make_cpus_node(d,kinfo->fdt,node);
if ( res )
returnres;
res= make_memory_node(d,kinfo->fdt,node,kinfo);
if ( res )
returnres;
}
res = fdt_end_node(kinfo->fdt);
return res;
}
再看这个函数的调用过程
construct_dom0()->prepare_dtb()->handle_node()
可以看出除了一些特殊的设备,gic、timer什么的,还有在黑名单里的设备不会被映射。其余的设备,如果是被标记上used_by DOMID_XEN,就不会调用map_device(),因此不会被映射到dom0里。
xen\xen\arch\arm\domain_build.c
/* Map the device inthe domain */
static int map_device(struct domain *d, conststructdt_device_node*dev)
{
unsigned intnirq;
unsigned intnaddr;
unsigned inti;
int res;
struct dt_irqirq;
struct dt_raw_irqrirq;
u64 addr,size;
nirq = dt_number_of_irq(dev);
naddr = dt_number_of_address(dev);
DPRINT("%s nirq = %d naddr = %u\n",dt_node_full_name(dev),nirq,naddr);
/* Map IRQs */
for ( i = 0;i <nirq; i++ )
{
res= dt_device_get_raw_irq(dev,i, &rirq);
if ( res )
{
printk(XENLOG_ERR"Unableto retrieve irq %u for %s\n",
i,dt_node_full_name(dev));
returnres;
}
/*
* Don't map IRQ that have no physicalmeaning
* ie: IRQ whose controller is not theGIC
*/
if ( rirq.controller!=dt_interrupt_controller )
{
DPRINT("irq %u not connected to primary controller."
"Connectedto %s\n", i, dt_node_full_name(rirq.controller));
continue;
}
res= dt_irq_translate(&rirq, &irq);
if ( res )
{
printk(XENLOG_ERR"Unableto translate irq %u for %s\n",
i,dt_node_full_name(dev));
returnres;
}
DPRINT("irq %u = %u type = 0x%x\n",i,irq.irq,irq.type);
/* Don'tcheck return because the IRQ can be use by multiple device */
gic_route_irq_to_guest(d, &irq,dt_node_name(dev));
}
/* Map theaddress ranges */
for ( i = 0;i <naddr; i++ )
{
res= dt_device_get_address(dev,i, &addr, &size);
if ( res )
{
printk(XENLOG_ERR"Unableto retrieve address %u for %s\n",
i,dt_node_full_name(dev));
returnres;
}
DPRINT("addr %u = 0x%"PRIx64" - 0x%"PRIx64"\n",
i,addr,addr+size - 1);
res= map_mmio_regions(d,addr &PAGE_MASK,
PAGE_ALIGN(addr+size) - 1,
addr & PAGE_MASK);
if ( res )
{
printk(XENLOG_ERR"Unableto map 0x%"PRIx64
"- 0x%"PRIx64" in dom0\n",
addr& PAGE_MASK, PAGE_ALIGN(addr +size) -1);
returnres;
}
}
return 0;
}
Map_device里对MMIO会调用map_mmio_regions()。
这样看来,当串口驱动注册时,标明了com1是used_by DOMID_XEN,也就是说这个串口在dom0中是不存在的,它的MMIO物理地址在dom0的地址空间里是没有被xen映射出来的。并且在dom0 linux看到的FDT中,也不存在描述串口的这个node。
如果不标记used_by DOMID_XEN的话,应该linux直接就可以输出到串口,相当于xen和linux共用这个设备。我想串口这个设备比较简单,这样用也不会有什么问题。是不是所说的passthrough就是把某一设备直接映射到一个domain里,其它domian和xen都不可见呢?这个以后再看。
当linux里的early_printk写物理地址时,一定会引发写异常。这个异常就会由xen处理了。
接下来就看xen怎样处理这个异常。
xen\xen\arch\arm\vuart.c
static int vuart_mmio_check(struct vcpu *v, paddr_taddr)
{
const structvuart_info*info =v->domain->arch.vuart.info;
return (domain_has_vuart(v->domain) &&addr>=info->base_addr&&
addr<= (info->base_addr+info->size));
}
static int vuart_mmio_read(struct vcpu *v, mmio_info_t*info)
{
struct domain *d =v->domain;
struct hsr_dabtdabt =info->dabt;
struct cpu_user_regs *regs=guest_cpu_user_regs();
register_t*r =select_user_reg(regs,dabt.reg);
paddr_t offset =info->gpa -d->arch.vuart.info->base_addr;
/* By defaultzeroed the register */
*r = 0;
if ( offset ==d->arch.vuart.info->status_off)
/* Allholding registers empty, ready to send etc */
*r =d->arch.vuart.info->status;
return 1;
}
static int vuart_mmio_write(struct vcpu *v, mmio_info_t*info)
{
struct domain *d =v->domain;
struct hsr_dabtdabt =info->dabt;
struct cpu_user_regs *regs=guest_cpu_user_regs();
register_t*r =select_user_reg(regs,dabt.reg);
paddr_t offset =info->gpa -d->arch.vuart.info->base_addr;
if ( offset ==d->arch.vuart.info->data_off)
/* ignore anystatus bits */
vuart_print_char(v, *r &0xFF);
return 1;
}
const struct mmio_handlervuart_mmio_handler = {
.check_handler= vuart_mmio_check,
.read_handler = vuart_mmio_read,
.write_handler= vuart_mmio_write,
};
xen\xen\arch\arm\io.c
static const struct mmio_handler *constmmio_handlers[] =
{
&vgic_distr_mmio_handler,
&vuart_mmio_handler,
};
#define MMIO_HANDLER_NRARRAY_SIZE(mmio_handlers)
int handle_mmio(mmio_info_t*info)
{
struct vcpu *v =current;
int i;
for ( i = 0;i <MMIO_HANDLER_NR;i++)
if ( mmio_handlers[i]->check_handler(v,info->gpa))
returninfo->dabt.write ?
mmio_handlers[i]->write_handler(v,info) :
mmio_handlers[i]->read_handler(v,info);
return0;
}
Xen注册的mmio_handler只有两个,一个是vgic一个是vuart,不知道其它的设备是怎样模拟的,比如IDE什么的。这个问题以后在看。
从代码中可以看到通过check_handler判断异常的mmio是不是在自己的处理范围。如果是就分别用write_handler和read_handler做处理,也就是模拟。我想通过mmio和中断这两种操作的模拟,就能模拟所有硬件了吧,vuart只用到了mmio。
最后验证下谁在调用handle_mmio()。
do_trap_hypervisor()->do_trap_data_abort_guest()->handle_mmio()
xen\xen\arch\arm\arm32\entry.S
#defineDEFINE_TRAP_ENTRY(trap) \
ALIGN; \
trap_##trap: \
SAVE_ALL; \
cpsie i; /* local_irq_enable */ \
adr lr, return_from_trap; \
mov r0, sp; \
mov r11, sp; \
bic sp, #7; /* Align the stack pointer(noop on guest trap) */ \
b do_trap_##trap
#defineDEFINE_TRAP_ENTRY_NOIRQ(trap) \
ALIGN; \
trap_##trap: \
SAVE_ALL; \
adr lr, return_from_trap; \
mov r0, sp; \
mov r11, sp; \
bic sp, #7; /* Align the stack pointer(noop on guest trap) */ \
b do_trap_##trap
.align 5
GLOBAL(hyp_traps_vector)
.word 0 /* 0x00 - Reset */
b trap_undefined_instruction /* 0x04 - Undefined Instruction */
b trap_supervisor_call /* 0x08 - Supervisor Call */
b trap_prefetch_abort /* 0x0c - Prefetch Abort */
b trap_data_abort /* 0x10 - Data Abort */
b trap_hypervisor /* 0x14 - Hypervisor */
b trap_irq /* 0x18 - IRQ */
b trap_fiq /* 0x1c - FIQ */
DEFINE_TRAP_ENTRY(undefined_instruction)
DEFINE_TRAP_ENTRY(supervisor_call)
DEFINE_TRAP_ENTRY(prefetch_abort)
DEFINE_TRAP_ENTRY(data_abort)
DEFINE_TRAP_ENTRY(hypervisor)
DEFINE_TRAP_ENTRY_NOIRQ(irq)
DEFINE_TRAP_ENTRY_NOIRQ(fiq)
最终在hyp_traps_vector里的trap_hypervisor调用 do_trap_hypervisor()。
总结下,对于之前的疑问,hypervisor为什么会截获到vuart的mmio物理内存读写操作,是因为xen没有映射对应的物理内存给dom0,dom0在fdt里也没有看到这个device的描述,所以也就不会建立起va到pa的映射,当然early_printk是直接用物理内存访问的,所以在读写的时候,是读写了一段不存在的物理内存,导致hyperviosr截获这个事件。
这也就是为什么我在linux开early_printk的时候,串口输出的消息前都加了个”DOM0:”。