linux+内核中开启nfs,NFS Client in Linux Kernel - Open

1. Callstack for Opening for a file

open() => nfs4_file_open() => nfs4_atomic_open() => nfs4_do_open() => _nfs4_do_open()

_nfs4_do_open分析:

nfs4_get_state_owner()获得open owner对象

设置claim，他有几种可能CLAIM_NULL(文件用文件名描述)，CLAIM_FH(文件用current filehandle描述)， CLAIM_PREVIOUS(用于reclaim open)

nfs4_opendata_alloc()分配nfs4_opendata

_nfs4_open_and_get_state()发送OPEN NFS命令

4.1 _nfs4_proc_open, 发送OPEN NFS命令

4.2 nfs4_opendata_to_nfs4_state() =>_nfs4_opendata_to_nfs4_state(), 创建open state,并将OPEN NFS命令得到的open state id，跟新这里。

open state存在ctx->state，其中ctx是nfs_open_context，它存储在filp->private_data。对文件open以后，就可以通过filp->private_data得到ctx，并得到ctx->state->open_stateid，通过这个stateid向NFS Server发送READ/WRITE/LOCK操作。

2. NFS协议中的Open owner

NFS协议规定发送OPEN时候要发送一个open owner，标识这个open的owner。在Server看来，clientid可以区分是来自哪个client的，open owner可以区分不同进程的open请求。

摘自[nfs4.1协议]https://tools.ietf.org/html/rfc5661

NFS协议中Open owner和Lock owner都用下面这个结构体表达。

struct state_owner4 {

clientid4 clientid;

opaque owner;

};

clientid,Server分配的，唯一标识一个和server建立session的client

字符串+字符串的长度,唯一标识client中的一个进程

client标识+进程标识，唯一确定一个lock的owner。

### 2.1 Linux Kernel中对Open owner的设置

static inline void encode_openhdr(struct xdr_stream *xdr, const struct nfs_openargs *arg)

{

...

*p++ = cpu_to_be32(24);

p = xdr_encode_opaque_fixed(p, "open id:", 8);

*p++ = cpu_to_be32(arg->server->s_dev);

*p++ = cpu_to_be32(arg->id.uniquifier);

xdr_encode_hyper(p, arg->id.create_time);

}

Lock owner大小24字节:

8字节的固定字符，"lock id:"

4字节的s_dev，表示是本地哪个文件系统。

4字节的id。由ida_get_new函数创建，在本地文件系统中唯一。

8字节的create_time

以上三点，保证了在一个client内Lock owner是唯一的。

再加上clientid，可以保证所有和Server建立session连接的client来自的lock owner是唯一确定。

3. Linux Kernel对一个打开实体Owner的描述

struct nfs4_state_owner {

struct nfs_server *so_server;

struct list_head so_lru;

unsigned long so_expires;

struct rb_node so_server_node;

struct rpc_cred *so_cred; //以这个为key,插入到红黑树里

spinlock_t so_lock;

atomic_t so_count;

unsigned long so_flags;

struct list_head so_states;

struct nfs_seqid_counter so_seqid;

seqcount_t so_reclaim_seqcount;

struct mutex so_delegreturn_mutex;

};

3.1 创建nfs4_state_owner

nfs4_get_state_owner()

首先通过so_cred为key，在红黑树里搜索

如果没有搜到，创建一个

4. Linux Kernel对一个打开实体的描述

这个数据结构专门描述open state，一个实体会被多个Owner打开，所以struct nfs4_state和struct nfs4_state_owner是一对多的关系

struct nfs4_state {

struct list_head open_states; //对于同一个inode,所有open state存在inode->open_states

struct list_head inode_states; //对于同一个owner,所有的open state存在owner->so_states

struct list_head lock_states; //这个open state下所有的lock state

struct nfs4_state_owner *owner; /* Pointer to the open owner */

struct inode *inode; /* Pointer to the inode */

unsigned long flags; /* Do we hold any locks? */

spinlock_t state_lock; /* Protects the lock_states list */

seqlock_t seqlock; /* Protects the stateid/open_stateid */

nfs4_stateid stateid; /* Current stateid: may be delegation */

nfs4_stateid open_stateid; /* OPEN stateid */

/* The following 3 fields are protected by owner->so_lock */

unsigned int n_rdonly; /* Number of read-only references */

unsigned int n_wronly; /* Number of write-only references */

unsigned int n_rdwr; /* Number of read/write references */

fmode_t state; /* State on the server (R,W, or RW) */

atomic_t count;

};

4.1 创建nfs4_state

nfs4_get_open_state

通过owner和inode的组合，试图寻找一下

如果找不到创建一个和inode关联

4. open stateid for Read/Write/Lock/Setattr/Close

这些操作都是依靠open stateid来向Server交互的。

参见下列函数的实现

encode_read()

encode_write()

encode_setattr()

encode_close()

encode_lock()

encode_locku()

static void encode_read(struct xdr_stream *xdr, const struct nfs_pgio_args *args,

struct compound_hdr *hdr)

{

__be32 *p;

encode_op_hdr(xdr, OP_READ, decode_read_maxsz, hdr);

encode_nfs4_stateid(xdr, &args->stateid); //NFS Server需要通过stateid找到file owner

p = reserve_space(xdr, 12);

p = xdr_encode_hyper(p, args->offset);

*p = cpu_to_be32(args->count);

}

调用栈

//open for file

nfs4_file_open <= sys_open

open_context // nfs4_atomic_open

nfs4_do_open

_nfs4_do_open

nfs4_recover_expired_lease

//如果当前Dentry访问过,则claim = NFS4_OPEN_CLAIM_FH

nfs4_opendata_alloc

_nfs4_open_and_get_state

_nfs4_proc_open

nfs4_run_open_task //send OPEN request to server

nfs4_open_prepare

//如果claim == NFS4_OPEN_CLAIM_FH，将OPEN修改成OPEN_NOATTR

//open for dir

nfs_opendir <= sys_open

if (filp->f_path.dentry == filp->f_path.mnt->mnt_root)

__nfs_revalidate_inode

getattr

//read dir

nfs_readdir <= sys_getdents

readdir_search_pagecache

find_cache_page

get_cache_page

=> nfs_readdir_filler

nfs_readdir_xdr_to_array

nfs_readdir_xdr_filler

readdir //nfs4_proc_readdir

_nfs4_proc_readdir //send READDIR

nfs_readdir_search_array