2年前的一个文件入库处理设计方法,记录下。
当时的入库使用的是sqlldr方法,用一个程序去专门生成控制文件,效率不错
A Data Store Process
oOriginal data is transmitted from other process constantly
oOne row of the data is such as 20110622~13100~3001~comment~
oThe 3rd field 3001 is a key field, with this identifier we can know which table should this data row be stored.
oThe function classifies the data rows by the key field of each row at first
oTo deal with each group that has been classified from the mass data flow, a buffer in the memory has been set to store it
oWhen the buffer is full, flush the data into the disk file, which is a temporary file
oWhen the temporary file achieve the appointed size, rename it so that the data store process can deal with it safely
oAt last, data store process put the data of the files into Oracle database.
oThe data store process can call the sqlldr command that supported by Oracle or call the oci functions to do it.