DCMTK超大体积图像爆内存

报dcmtk错误MemoryExhaust或系统弹出out of memory

两种场景：
1、超大尺寸、多帧图像序列，读取、写入时会分配大量内存，有可能受限于硬件达不到
2、32位程序，windows里限制了用户空间最大2GB内存（改boot可以3GB），轻轻松松超过限制

dicom标准里有考虑这种情况，见PS3.3 7.5 Organizing Large Sets of Information
其思路就是拆分成多个SOP来传输/保存图像

但如果必须在一个SOP内存一张超大尺寸的图呢？

落到代码层，dcmtk我们一般通过findAndGetUint8Array、putAndInsertUint8Array读写pixeldata，这俩都要求一次性读写全部数据，必定爆内存

dmctk论坛有人提问但基本上是单机贴吧，没人解答：PixelData is out of memory

#1
Mon, 2013-11-11, 17:47
Hi everyone.
I ran into memory problem when copying data into DcmPixelData.

My codes is like:

int nBufWidth(240), nBufHeight(240), nBufBPP(3), nNumOfFrmInX(62), nNumOfFrmInY(58);
long lLenOfLevel(nBufWidth * nBufHeight * nBufBPP * nNumOfFrmInX * nNumOfFrmInY);
char* pBufImage = new char[lLenOfLevel];

DcmPixelData* pElement = new DcmPixelData(
DcmTag(0x7fe0, 0x0010), lLenOfLevel
);
if (pElement)
{
result = pElement->putUint8Array((const Uint8*)(pBuffer), lLenOfLevel);
result = pDcmItem->insert(pElement, bReplaced?OFTrue:OFFalse);
}

The resut of putUint8Array is always out of memory.

May anyone help me to fix this problem, please?

Thanks.

#2
Use DcmPixelData:createUint8Array() or DcmPixelData::createValueFromTempFile() when adding large pixel data arrays.

#3
Hi Riesmeier,
Thanks for your help.

DcmPixelData:createUint8Array() does work when it is called in an application.
My case is that I use Dcmtks in my dynamic linked library. There might be limitations of memory consuming, and DcmPixelData:createUint8Array() always return MemoryExhaust.

Wishes,

#4
By the way, is there any mention in DICOM standard to c-store a extra-large multi-frame dataset ?

For example, I may send one frame a time to DCM SCU, and DCM SCU will assembly frames to one piece.
Each dataset has a random SOP instance UID for itself, and a ref sop instance UID pointing to one requested by DCM SCU.

Thanks,

#5
Search for “concatenation” in part 3 (and related parts) of the DICOM standard.

#6
Hi Riesmeier,

Thanks for your help.

Wishes,

#7
Hi every one,

I have to go back to my problem of handling large raw elements, such as pixel data.

I noticed that there is a method DcmPixelData::getUncompressedFrame() to retrieve a specific frame without loading the complete object into memory.
So, I am wondering if there is similar way to add new frame to the end of the pixel data without loading the whole object.

Thanks,

#8
Hi,
Let me make the case more clear.

There is already a multi-frame image file at disk. I need to add new frame to the end of its Pixel Data.
Is it possible to do it without loading all frames into memory?

Thanks,

#9
I defined two functions:

int addNewFrame( DcmItem* pDcmRoot, Uint8 *pBufferChar, unsigned long numBytes )
{
DcmPixelData *pPixelData = NULL;
cond = pDcmRoot->findAndGetElement( Dcm_PixelData, pPixelData, false, false );
if (cond.bad())
{
pPixelData = new DcmPixelData(DCM_PixelData);
cond = pDcmRoot->insert(pPixelData);
}
else
{
pPixelData = (DcmPixelData *) pElement;
[b]pPixelData->loadAllDataIntoMemory();[/b]
}

    DcmPixelSequence *pDcmPixelSeq = NULL;
    pPixelData->getEncapsulatedRepresentation(xferSyntax, rep, pDcmPixelSeq);
    ///. assume pDcmPixelSeq is always be there.

   DcmPixelItem *pDcmPixelItem = new DcmPixelItem(DcmTag(DCM_Item, EVR_OB));
    cond = pDcmPixelItem->putUint8Array(pBufferChar, numBytes);
    cond = pDcmPixelSeq->insert( pDcmPixelItem );
    ... ...

}

void mainfunction()
{
int nWidth(512);
unsigned long nLenOfBuf(nWidthnWidth)
char pBuffer = new char[nLenOfBuf];

... ...
for (long i=0; i<100; ++i)
{
    DcmDataSet dcmDS;

    nError = dcmDS.load("c:\\temp\\temp20131122.15301894.1.dcm");
    
    for (long j=0; j<100; ++j)
    {
        nError = dcmDS.addNewFrame(
            &dcmDS, pBuffer, nLenOfBuf
            );
    }

    ///. EXS_JPEGProcess1TransferSyntax = 4,
    dcmDS.save("c:\\temp\\temp20131122.15301894.1.dcm", 4);
}
... ...
delete[]  pBuffer;

}
When “pPixelData->loadAllDataIntoMemory()” is removed, PixelData could not be saved correctly.
With “pPixelData->loadAllDataIntoMemory()”, the loop would become slow, and at the end stopped working when counting to tens of thousands.

#10
I also find an old post saying that
"A Pixel Sequence (a “pseudo” sequence, by the way) is only used in DICOM, if compressed (also called “encapsulated”) data is stored. So if you assemble an uncompressed multi frame image object in memory, you don’t need one. You just have put all frames into one big blob and write it into the Pixel Data attribute. If you then start conversion, DCMTK’s JPEG codec will use a Pixel Sequence for the JPEG compressed frames. As a user you don’t have to take care of that yourself. "

Then when there is limitation of memory for a uncompressed multi frame image object, how to write frames into PixelData?

#11
Now I moved to createValueFromTempFile(…).

I noticed that DcmObject::length has the type of Uint32. So when the size of temp file is bigger that 2 GB, only part of the temp file will be copied into PixelData element.

The biggest DICOM file I generated has the size of 1.37 GB.

Regards,

#12
Thu, 2013-11-28, 19:40
Hmm, it seems that you are writing with yourself? Before starting to implement anything you should probably read the relevant sections of the DICOM standard… e.g. the maximum length of a DICOM element value (including uncompressed Pixel Data and a single compressed Pixel Item) is 4.2 GB, because the length field has 32 bit (unsigned integer).

有一个解决思路是修改dcmtk的saveFile源码，引入逐行写入的机制