【外部排序】快排思想完成外部排序

置顶 pass night

已于 2022-10-04 21:28:07 修改

阅读量1.9k

点赞数 11

分类专栏：小型项目文章标签：算法排序算法大数据 c++

于 2022-07-21 21:09:56 首次发布

本文链接：https://blog.csdn.net/apple_50661801/article/details/125921080

版权

小型项目专栏收录该内容

4 篇文章 4 订阅

订阅专栏

问题描述

利用快排思想，实现外排。

算法思想

在这里插入图片描述

将磁盘划分为四个区域：middle, small, large, input
当input为空时，从磁盘中读取数据
将游标存放在middle中，middle采用双端堆进行存储，可以以 $\log(n)$ 的效率获取最大最小值。 在本实验中，因为数据可能重复的缘故，所以使用了stl中自带的优先队列进行实现，而非使用更高效的平衡二叉树实现
先从磁盘中读取数据，填充Middle
不断从input中读取数据，与游标进行比较，较大值放入Large，较小值放入Small；当输出缓存满时写入磁盘两侧
递归磁盘较大组合较小组

算法实现

缓存类

struct Cache
{
    int *cache;
    int capacity;
    // the start position of the cache in disk
    // when the cache is empty, startPosition = endPosition = 0
    int diskStartPosition;
    int diskEndPosition;
    // when Cache is not filled, endPosition - startPosition != capacity
    int startPosition;
    int endPosition;
    int size;
};

声明缓存类管理缓存，同时保存部分相关数据；但因为数组位置信息保存在数组类当中，所以不提供读写操作

Middle Cache的实现

双端堆的实现

template <class T>
class DoubleEndPriorityQueue
{
    friend class DoubleEndPriorityQueueCache;
    //升序队列
    priority_queue<int, vector<int>, greater<int>> *acc;
    //降序队列
    priority_queue<int, vector<int>, less<int>> *desc;

public:
    int size()
    {
        return acc->size();
    }

    bool isEmpty()
    {
        return desc->empty();
    }

    void insert(int x)
    {
        acc->push(x);
        desc->push(x);
    }

    int getMin()
    {
        return acc->top();
    }

    int getMax()
    {
        return desc->top();
    }

    void deleteMin()
    {
        int min = acc->top();
        bool flag = false;
        acc->pop();
        priority_queue<int, vector<int>, less<int>> *temp = new priority_queue<int, vector<int>, less<int>>();
        while (!desc->empty())
        {
            if (desc->top() != min)
            {
                temp->push(desc->top());
                desc->pop();
            }
            else
            {
                if (!flag)
                {
                    flag = true;
                    desc->pop();
                }
                else
                {
                    temp->push(desc->top());
                    desc->pop();
                }
            }
        }
        delete desc;
        desc = temp;
    }

    void deleteMax()
    {
        int i = desc->top();
        bool flag = false;
        desc->pop();
        priority_queue<int, vector<int>, greater<int>> *temp = new priority_queue<int, vector<int>, greater<int>>();
        while (!acc->empty())
        {
            if (acc->top() != i)
            {
                temp->push(acc->top());
                acc->pop();
            }
            else
            {
                if (!flag)
                {
                    flag = true;
                    acc->pop();
                }
                else
                {
                    temp->push(acc->top());
                    acc->pop();
                }
            }
        }
        delete acc;
        acc = temp;
    }

    DoubleEndPriorityQueue()
    {
        acc = new priority_queue<int, vector<int>, greater<int>>();
        desc = new priority_queue<int, vector<int>, less<int>>();
    }
    DoubleEndPriorityQueue(const DoubleEndPriorityQueue<T> &right)
    {
        acc = right.acc;
        desc = right.desc;
    }
};

为了节约时间，并没有自己手动实现一个双端堆
因为系统包括网上大多数红黑树及AVL树的实现都不允许数据重复，所以实验中使用了stl中的堆进行组合

数组类的实现

class ExternalArray
{
    friend class ExternalArrayTest;

private:
    int length = 0;
    Cache inputCache;
    Cache smallCache;
    Cache largeCache;
    // the cache range is [startPosition, endPosition) and start from 0
    //  when smallCache is swapped into disk, the replaced data in disk is store in here
    //  and when tempCache is not dealt with, the input will input from here first
    Cache tempCache;

    DoubleEndPriorityQueueCache middleCache;

    fstream fio;
    string filePath;
    size_t diskIOCount = 0;
};

数组类包含对应的各个缓存
使用fstream进行文件操作
使用diskIOCount计算磁盘IO数量

算法思路

在这里插入图片描述

将分段最左边读入游标缓存：将分段最左侧读入游标缓存，若分段小于游标缓存，则直接写回。否则进行下一步。
创建游标（cur），遍历分段：游标不断右移，进行分片，直到游标到达end
Small cache 写满：游标不断右移，所以不会出现覆盖的情况，若写满则直接写回
Large cache写满：
1. 写回
2. 看情况保存被覆盖的磁盘数据（当游标到写回位置的距离小于large cache 的容量）移动end位置
完成遍历后：
1. 将temp cache 中的所有数据处理
2. 将三个cache中的所有数据写回
3. 递归余下两个分段

缓存操作函数

写缓存：

    // write to disk
    void writeCache(Cache &cache, int diskStartPosition, int startPosition, int endPosition, bool sequencial = true)
    {
        cache.diskStartPosition = diskStartPosition;
        fio.seekp(diskStartPosition * 13, ios::beg);
        for (int i = startPosition; i < cache.size; i++)
        {
            fio.write(formateInt(cache[i]).append(" ").c_str(), 13);
        }
        cache.diskEndPosition = diskStartPosition + cache.size;
        fio.seekp(-1, ios::end);
        fio << NULL;
        fio.clear();
        if (!sequencial)
        {
            cache.diskEndPosition = -1;
            cache.diskStartPosition = -1;
        }
        diskIOCount++;
    }
    void writeCache(DoubleEndPriorityQueueCache &cache, int diskStartPosition)
    {
        cache.diskStartPosition = diskStartPosition;
        int count = 0;
        fio.seekp(diskStartPosition * 13, ios::beg);
        while (!cache.empty())
        {

            fio.write(formateInt(cache.popMin()).append(" ").c_str(), 13);
            count++;
        }
        cache.diskEndPosition = diskStartPosition + count;
        fio.seekp(-1, ios::end);
        fio << NULL;
        fio.clear();
        diskIOCount++;
    }

读缓存：

    // basic read method
    // read from disk
    // if the disk is empty or shorter than cache, then fill with zero
    // the append parameter is used to determine whether the cache is appended to the old data
    // when read from disk, the status flag(those position) is automatically set
    // return count number of data
    int readCache(Cache &cache, int start, int end, int diskStartPosition, bool append = false, bool sequencial = true)
    {
        fio.seekg(diskStartPosition * 13, ios::beg);
        int i = start, count = 0;
        if (!append)
            cache.size = 0;
        for (; i < end && fio.peek() != EOF; i++, count++)
        {
            fio >> cache[i];
            cache.size++;
        }
        if (append)
        {
            // cache.cacheStartPosition is equal to old one
            cache.endPosition += count;
            // cache.diskStartPosition is equal to old one
            cache.diskEndPosition += count;
        }
        else
        {
            cache.startPosition = start;
            cache.endPosition = start + count;
            cache.diskStartPosition = diskStartPosition;
            cache.diskEndPosition = diskStartPosition + count;
        }
        for (; i < end; i++)
            cache[i] = 0;
        if (!sequencial)
        {
            cache.diskEndPosition = -1;
            cache.diskStartPosition = -1;
        }

        fio.clear();
        diskIOCount++;
        return count;
    }
    void readCache(DoubleEndPriorityQueueCache &cache, int diskStartPosition, int length)
    {
        cache.diskStartPosition = diskStartPosition;
        fio.seekg(diskStartPosition * 13, ios::beg);
        int t, i;
        for (i = 0; i < length && fio.peek() != EOF; i++)
        {
            fio >> t;
            cache.insert(t);
        }
        cache.diskEndPosition = diskStartPosition + i;
        fio.clear();
        diskIOCount++;
    }

当tempCache非空时，优先读取tempCache中的数据：

    // start from pos, and fill input cache
    //  is the tempCache is not empty(or valid), then the input will be read from tempCache first
    void readInputCache(int pos, int length)
    {
        if (length > inputCache.capacity)
        {
            cout << "invalid parameter" << endl;
            exit(0);
        }

        if (tempCache.size == 0)
            readCache(inputCache, 0, length, pos, false);
        else
        {
            int i = 0;
            inputCache.size = 0;
            inputCache.startPosition = 0;
            inputCache.endPosition = 0;
            for (; i < length && tempCache.size > 0; i++, tempCache.size--)
            {
                inputCache[i] = tempCache[i];
                tempCache.endPosition--;
                inputCache.size++;
                inputCache.endPosition++;
            }
            if (inputCache.size < length)
                readCache(inputCache, i, length, pos, true, false);
            inputCache.diskStartPosition = -1;
            inputCache.diskEndPosition = -1;

            tempCache.startPosition = -1;
            tempCache.endPosition = -1;
        }
    }
    void readTempCache(int pos, int length)
    {
        readCache(tempCache, tempCache.endPosition, tempCache.endPosition + length, pos, true, false);
    }

封装后的操作：

    // add a new element to the end of the output cache, if output cache then write to disk
    // if head == true, then write from head, else write from tail
    // in fact head == true means the element is added from the small cache
    // if head == false, then the element is added from the large cache
    bool add(Cache &cache, int x, bool head, int &diskStartPosition, int &currentDiskEndPosition)
    {
        bool willMiss = false;
        if (cache.size < cache.capacity)
        {
            cache[cache.endPosition] = x;
            cache.endPosition++;
            cache.size++;
        }
        else
        {
            if (head)
            {
                writeCache(cache, diskStartPosition, 0, cache.capacity, false);
                diskStartPosition += cache.capacity;
            }
            else
            {

                if (diskStartPosition - cache.capacity > currentDiskEndPosition)
                {
                    readTempCache(diskStartPosition - cache.capacity, cache.capacity);
                    writeCache(cache, diskStartPosition - cache.capacity, 0, cache.capacity, false);
                    diskStartPosition -= cache.capacity;
                    willMiss = true;
                }
                else
                {
                    readTempCache(currentDiskEndPosition + 1, diskStartPosition - currentDiskEndPosition - 1);
                    writeCache(cache, diskStartPosition - cache.capacity, 0, cache.capacity, false);
                    diskStartPosition -= cache.capacity;
                }
            }
            cache.startPosition = 0;
            cache.endPosition = 1;
            cache[0] = x;
            cache.size = 1;
        }
        return willMiss;
    }

快排实现

在有了上述代理之后，便可以利用普通快排的思想进行排序
其中注意排序后要清空缓存

    // the array is start from start and end to end in disk
    void quickSort(int start, int end)
    {
        // point to current position in disk
        int cur = start;
        // initialization
        readCache(middleCache, start, min(middleCache.capacity, end - start));
        cur += middleCache.size();

        // point to the location in disk where the small cache should be written
        int small = start;
        // point to the location in disk where the large cache should be written
        int large = end;

        int willMiss = 0;
        //开始处理
        while (cur < large || willMiss > 0)
        {
            if (willMiss > 0)
            {
                cur -= min(inputCache.size, tempCache.size);
                willMiss--;
            }
            readInputCache(cur, min(large - cur, inputCache.capacity));
            for (int i = inputCache.startPosition; i < inputCache.endPosition; i++)
            {
                int record = inputCache[i];
                if (record <= middleCache.getMin())
                    add(smallCache, record, true, small, cur);
                else if (record >= middleCache.getMax())
                {
                    if (add(largeCache, record, false, large, cur))
                        willMiss++;
                }
                else
                {
                    add(smallCache, middleCache.popMin(), true, small, cur);
                    middleCache.insert(record);
                }

                cur++;
                if (cur >= large)
                    break;
            }
        }
        for (int i = 0; tempCache.size > 0; i++, tempCache.size--)
        {
            int record = tempCache[i];

            if (record <= middleCache.getMin())
                add(smallCache, record, true, small, cur);
            else if (record >= middleCache.getMax())
                add(largeCache, record, false, large, cur);
            else
            {
                add(smallCache, middleCache.popMin(), true, small, cur);
                middleCache.insert(record);
            }
            cur++;
            tempCache.endPosition--;
        }

        writeCache(smallCache, small, 0, smallCache.size, false);
        writeCache(middleCache, small + smallCache.size);
        writeCache(largeCache, large - largeCache.size, 0, largeCache.size, false);
        smallCache.size = 0;
        smallCache.startPosition = 0;
        smallCache.endPosition = 0;

        largeCache.size = 0;
        largeCache.startPosition = 0;
        largeCache.endPosition = 0;

        int a = middleCache.diskStartPosition;
        if (a - start > 1)
            quickSort(start, a);
        int b = middleCache.diskEndPosition;
        if (end - b > 1)
            quickSort(b, end);
    }

实验结果与分析

测试类

class ExternalArrayTest
{
public:
    // test double end priority queue
    void test0()
    {
        srand(time(NULL));
        ExternalArray a(10, 10, 10, 2000);
        for (int i = 0; i < 2000; i++)
        {
            a.middleCache.insert(rand() % 10000);
        }
        a.writeCache(a.middleCache, 0);
        cout << "isAccendent = " << a.isAccendent() << endl;
    }
    // test constructor
    void test1()
    {
        ExternalArray a(10);
        a.addRandomNumber(1000);
    }
    // test readCache()
    void test2()
    {
        ExternalArray a(10);
        a.addRandomNumber(1000);
        a.readInputCache(0, a.inputCache.capacity);
        for (int i = 0; i < 10; i++)
        {
            cout << a.inputCache[i] << " ";
        }
        a.readInputCache(990, a.inputCache.capacity);
        cout << endl;
        for (int i = 0; i < 10; i++)
        {
            cout << a.inputCache[i] << " ";
        }
    }
    void test3()
    {
        ExternalArray a(10, 10, 10, 20);
        a.addRandomNumber(5);
        a.readCache(a.middleCache, 0, 10);
        a.middleCache.print();
        a.readCache(a.inputCache, 0, 10, 0, false);
        a.inputCache.print();
        a.addRandomNumber(95);
        a.readCache(a.middleCache, 0, 10);
        a.middleCache.print();
        a.readCache(a.inputCache, 0, 10, 0, false);
        a.inputCache.print();
    }
    // test writeCache()
    void test4()
    {
        ExternalArray a(10, 10, 10, 20);
        a.addRandomNumber(100);
        a.middleCache.insert(0);
        a.middleCache.insert(1);
        a.middleCache.insert(2);
        a.writeCache(a.middleCache, 0);
        a.inputCache[0] = 9;
        a.inputCache[1] = 99;
        a.inputCache[2] = 999;
        a.writeCache(a.inputCache, 3, 0, 3);
    }
    void test5()
    {
        ExternalArray a(10, 10, 10, 20);
        a.addRandomNumber(100);
        a.readCache(a.middleCache, 0, 10);
        a.middleCache.print();
        a.readCache(a.inputCache, 0, 10, 0, false);
        a.inputCache.print();
    }
    //足够大的内存
    int test6()
    {
        ExternalArray a(10, 10, 10, 10000);
        a.addRandomNumber(10000);
        a.quickSort();
        return a.isAccendent();
    }
    int test7()
    {
        ExternalArray a(10, 10, 10, 80);
        a.addRandomNumber(100);
        a.quickSort();
        return a.isAccendent();
    }
    //恰好够大的内存
    int test8()
    {
        ExternalArray a(10, 10, 10, 80);
        a.addRandomNumber(100);
        a.quickSort();
        return a.isAccendent();
    }
    //不够大的内存
    int test9()
    {
        ExternalArray a(100, 100, 100, 300);
        a.addRandomNumber(1000);
        a.quickSort();
        return a.isAccendent();
    }

    //大量数据
    int test10()
    {
        ExternalArray a(2500, 2500, 2500, 30000);
        a.addRandomNumber(100000);
        a.quickSort();
        return a.isAccendent();
    }
    //非常大量数据
    int test11()
    {
        ExternalArray a(2500, 2500, 2500, 30000);
        a.addRandomNumber(1000000);
        a.quickSort();
        return a.isAccendent();
    }
    void runTest()
    {
        if (test6() == -1)
            cout << "test6 pass" << endl;
        if (test7() == -1)
            cout << "test7 pass" << endl;
        if (test8() == -1)
            cout << "test8 pass" << endl;
        if (test9() == -1)
            cout << "test9 pass" << endl;
        if (test10() == -1)
            cout << "test10 pass" << endl;
    }
    void experience_diskIO()
    {
        for (int length = 1000; length < 3000; length += 500)
        {
            for (int inputCacheSize = 100; inputCacheSize < 400; inputCacheSize += 100)
            {
                for (int middleCacheSize = 300; middleCacheSize < 600; middleCacheSize += 100)
                {
                    for (int largeCacheSize = 500; largeCacheSize < 1000; largeCacheSize += 100)
                    {
                        ExternalArray a(inputCacheSize, inputCacheSize, inputCacheSize, middleCacheSize);
                        a.addRandomNumber(length);
                        a.quickSort();
                        if (a.isAccendent() == -1)
                        {
                            cout << "length = " << length << " input and output cache size = " << inputCacheSize << "; middleCacheSize = " << middleCacheSize << " diskIOCount = " << a.diskIOCount << endl;
                        }
                    }
                }
            }
        }
    }
    void experience_time()
    {
        for (int length = 1000; length < 3000; length += 500)
        {
            for (int inputCacheSize = 100; inputCacheSize < 400; inputCacheSize += 100)
            {
                for (int middleCacheSize = 300; middleCacheSize < 600; middleCacheSize += 100)
                {
                    for (int largeCacheSize = 500; largeCacheSize < 1000; largeCacheSize += 100)
                    {
                        ExternalArray a(inputCacheSize, inputCacheSize, inputCacheSize, middleCacheSize);
                        a.addRandomNumber(length);
                        clock_t start = clock();
                        a.quickSort();
                        clock_t end = clock();
                        cout << "length = " << length << " input and output cache size = " << inputCacheSize << "; middleCacheSize = " << middleCacheSize << " time = " << (double)(end - start) / CLOCKS_PER_SEC << endl;
                    }
                }
            }
        }
    }
};

实验结果

正确性分析

在这里插入图片描述

实验结果

在这里插入图片描述

实验分析

在本次实验中，为了尽量地分析缓存大小对排序效率的影响，所有的缓存都可以进行独立配置。其大小如下

inputCache: inputCacheCapacity
smallCache: smallCacheCapacity
largeCache: largeCacheCapacity
tempCache: smallCacheCapacity + largeCacheCapacity

以上缓存是直接使用的内存空间，同时还有递归带来的栈空间消耗O(length)

所以总的内存消耗为：inputCacheCapacity+2(smallCacheCapacity+largeCacheCapacity) + O(length)

而运行时间瓶颈不再CPU上，而在磁盘io上，因此通过计算磁盘io的访问次数来大致预测时间复杂度。磁盘io次数分析如下

当middleCache可以容纳所有的数据时：1
当所有输出cache和middleCache的总和能容纳所有数据时：3
当cache无法容纳所有数据（）：

平均情况：共发生了log(n)次递归，每段递归段的大小为length/log(n)

对于长度为length的数组字段：在平均情况下，数组大小成均匀分布，写入smallCache和largeCache的概率相等,而每一次递归固定读取一次middleCache。

此时若outputCache大小若大于inputCache，当outputCache未填满时，inputCache从磁盘中读取数据，次数为 $\frac{2 \times inputCacheCapacity}{smallCacheCapacity+largeCacheCapacity}$ ,接下来，平均下来每次outputCache写入时都会将文件临时存到内存，此时无需inputCache从磁盘中读取文件，访问总次数为： $2\frac{length-(smallCacheCapacity+largeCacheCapacity)-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity}$ 所以总次数为：

$\frac{2 \times inputCacheCapacity}{smallCacheCapacity+largeCacheCapacity}\\ +2\frac{length-(smallCacheCapacity+largeCacheCapacity)-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity}$

若outputCache小于inputCache,尽管outputCache在写入的时候会将磁盘中同位置的文件读到缓存，但是大小并不足以inputCache读取，inputCache依然要读取缓存，此时磁盘io数为：
$2\frac{length-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity}\\ +\frac{length-middleCacheCapacity}{inputCacheCapacity}$
此时发生发生的总磁盘iO数为
$(\frac{2 \times inputCacheCapacity}{smallCacheCapacity+largeCacheCapacity}\\ +2\frac{\frac{length}{\log(length)}-(smallCacheCapacity+largeCacheCapacity)-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity})\\ \times \log(length)$
或
$(2\frac{\frac{length}{\log(length)}-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity}\\ +\frac{\frac{length}{\log(length)}-middleCacheCapacity}{inputCacheCapacity})\\ \times \log(length)$

而对于最坏情况，每一次递归都将所有的数据进行一次移动，此时发生lenght次递归，每次递归长度为length

而磁盘io次数分析同上（尽管是这样分析，但实际情况在这种极端情况下，很有可能总是往一个缓存中写入），公式如下：

$(\frac{2 \times inputCacheCapacity}{smallCacheCapacity+largeCacheCapacity}\\ +2\frac{length-(smallCacheCapacity+largeCacheCapacity)-middleCacheCapacity}{smallCacheCapacity+largeCacheCapacity})\\ \times length$