Linux的O_DIRECT选项

在man 2 open的时候发现一个O_DIRECT选项，使用O_DIRECT选项后，可以不使用缓存直接写入。在海量数据写入的时候，不使用缓存貌似更快呢！于是也尝试写了一个用O_DIRECT选项的文件写入。完成O_DIRECT选项写入的代码还真不容易，使用new或者malloc分配的内存是无法在O_DIRECT选项下工作的，必须使用posix_memalign(或valloc, memalign，这两个函数已经被标记为废弃)。
      我的猜想（唉，没学过内核，只能是猜想）是：要使用高性能的调用，则在用户空间分配的内存的格式必须与内核空间所使用的内存页一致，这样内核才能高速地处理数据。posix_memalign是基于页的方式来分配内存的，且分配的大小必须是页大小的整数倍。

     以下是实现了O_DIRECT选项的文件写入代码：
//写入200MB数据花费的时间是3100994微秒，3.1秒，更慢了，而且慢太多！！！
/*
测试大数据写入的性能 test_io_4.cpp 使用posix函数，且使用O_DIRECT选项
*/

#include <stdio.h>
#include <time.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>

static int operator-(struct timeval& lsh, struct timeval& rsh)
{
    if (lsh.tv_sec==rsh.tv_sec)
    {
        return lsh.tv_usec - rsh.tv_usec;
    }
    else
    {
        return (lsh.tv_sec-rsh.tv_sec)*1000000 + (lsh.tv_usec - rsh.tv_usec);
    }
}

void test()
{
    struct timeval start;
    struct timeval end;
    const int DATA_LEN = 1024*1024*200; //200MB
    char* pData = NULL;
    printf("page size=%d\n", getpagesize());
    int nTemp = posix_memalign((void**)&pData, getpagesize(), DATA_LEN);
    if (0!=nTemp)
    {
        perror("posix_memalign error");
        return;
    }
    //pData[DATA_LEN-1] = '\0';
    gettimeofday(&start, NULL);
    int fd = open("write_direct.dat", O_RDWR | O_CREAT | O_DIRECT);
    if (fd<0)
    {
        perror("open error:");
        return;
    }
    int nLen = write(fd, pData, DATA_LEN);
    if (nLen<DATA_LEN)
    {
        perror("write error:");
        return;
    }
    close(fd);
    fd = -1;

    gettimeofday(&end, NULL);
    free(pData);
    pData = NULL;
    //显示占用时间
    struct tm stTime;
    localtime_r(&start.tv_sec, &stTime);
    char strTemp[40];
    strftime(strTemp, sizeof(strTemp)-1, "%Y-%m-%d %H:%M:%S", &stTime);
    printf("start=%s.%07d\n", strTemp, start.tv_usec);
    //
    localtime_r(&end.tv_sec, &stTime);
    strftime(strTemp, sizeof(strTemp)-1, "%Y-%m-%d %H:%M:%S", &stTime);
    printf("end =%s.%07d\n", strTemp, end.tv_usec);
    printf("spend=%d 微秒\n", end-start);
}

int main()
{
    test();
    return 1;
}

    使用了O_DIRECT选项反而是文件写入更慢了，百思不得其解，终于在网上找到这篇文章：
《Linus对O_DIRECT非常不感冒啊，呵呵》
http://www.linux-ren.org/modules/newbb/viewtopic.php?topic_id=2722

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

A thread on the lkml began with a query about using O_DIRECT when opening a file. An early white paper written by Andrea Arcangeli [interview] to describe the O_DIRECT patch before it was merged into the 2.4 kernel explains, "with O_DIRECT the kernel will do DMA directly from/to the physical memory pointed [to] by the userspace buffer passed as [a] parameter to the read/write syscalls. So there will be no CPU and memory bandwidth spent in the copies between userspace memory and kernel cache, and there will be no CPU time spent in kernel in the management of the cache (like cache lookups, per-page locks etc..)." Linux creator Linus Torvalds was quick to reply that despite all the claims there is no good reason for mounting files with O_DIRECT, suggesting that interfaces like madvise() and posix_fadvise() should be used instead.

From: Aubrey [email blocked]
To: "Hua Zhong" [email blocked]O_
Subject: O_DIRECT question
Date: Thu, 11 Jan 2007 10:57:06 +0800

Hi all,

Opening file with O_DIRECT flag can do the un-buffered read/write access.
So if I need un-buffered access, I have to change all of my
applications to add this flag. What's more, Some scripts like "cp
oldfile newfile" still use pagecache and buffer.
Now, my question is, is there a existing way to mount a filesystem
with O_DIRECT flag? so that I don't need to change anything in my
system. If there is no option so far, What is the right way to achieve
my purpose?

Thanks a lot.
-Aubrey



From: Linus Torvalds [email blocked]
Subject: Re: O_DIRECT question
Date: Wed, 10 Jan 2007 19:05:30 -0800 (PST)



On Thu, 11 Jan 2007, Aubrey wrote:
>
> Now, my question is, is there a existing way to mount a filesystem
> with O_DIRECT flag? so that I don't need to change anything in my
> system. If there is no option so far, What is the right way to achieve
> my purpose?

The right way to do it is to just not use O_DIRECT.

The whole notion of "direct IO" is totally braindamaged. Just say no.

This is your brain: O
This is your brain on O_DIRECT: .

Any questions?

I should have fought back harder. There really is no valid reason for EVER
using O_DIRECT. You need a buffer whatever IO you do, and it might as well
be the page cache. There are better ways to control the page cache than
play games and think that a page cache isn't necessary.

So don't use O_DIRECT. Use things like madvise() and posix_fadvise()
instead.

Linus



From: Linus Torvalds [email blocked]
Subject: Re: O_DIRECT question
Date: Wed, 10 Jan 2007 19:15:48 -0800 (PST)



On Wed, 10 Jan 2007, Linus Torvalds wrote:
>
> So don't use O_DIRECT. Use things like madvise() and posix_fadvise()
> instead.

Side note: the only reason O_DIRECT exists is because database people are
too used to it, because other OS's haven't had enough taste to tell them
to do it right, so they've historically hacked their OS to get out of the
way.

As a result, our madvise and/or posix_fadvise interfaces may not be all
that strong, because people sadly don't use them that much. It's a sad
example of a totally broken interface (O_DIRECT) resulting in better
interfaces not getting used, and then not getting as much development
effort put into them.

So O_DIRECT not only is a total disaster from a design standpoint (just
look at all the crap it results in), it also indirectly has hurt better
interfaces. For example, POSIX_FADV_NOREUSE (which _could_ be a useful and
clean interface to make sure we don't pollute memory unnecessarily with
cached pages after they are all done) ends up being a no-op ;/

Sad. And it's one of those self-fulfilling prophecies. Still, I hope some
day we can just rip the damn disaster out.

Linus

---

From: Andrew Morton [email blocked]
Subject: Re: O_DIRECT question
Date: Wed, 10 Jan 2007 20:51:57 -0800

On Thu, 11 Jan 2007 10:57:06 +0800
Aubrey [email blocked] wrote:

> Hi all,
>
> Opening file with O_DIRECT flag can do the un-buffered read/write access.
> So if I need un-buffered access, I have to change all of my
> applications to add this flag. What's more, Some scripts like "cp
> oldfile newfile" still use pagecache and buffer.
> Now, my question is, is there a existing way to mount a filesystem
> with O_DIRECT flag? so that I don't need to change anything in my
> system. If there is no option so far, What is the right way to achieve
> my purpose?

Not possible, basically.

O_DIRECT reads and writes must be aligned to the device's block size
(usually 512 bytes) in memory addresses, file offsets and read/write request
sizes. Very few applications will bother to do that and will hence fail if
their files are automagically opened with O_DIRECT.
----------------------------------------------------------------------------------

    呵呵，注意这一句：The whole notion of "direct IO" is totally braindamaged. (使用"direct IO"的想法简直是脑子坏掉了！) 恩，看来O_DIRECT选项是早就不推荐使用的了。

NE:

使用O_DIRECT的话，就必须以页为单位进行I/O，这是没办法的事，因为设备本身就是块设备。你可以加一层中间代码，自己计算对齐后的文件偏移量，用posix_memalign生成对齐的buffer，进行I/O以后，再把buffer里面的内容copy到调用者的buffer里面去。