读文件数据入库设计

2年前的一个文件入库处理设计方法，记录下。

当时的入库使用的是sqlldr方法，用一个程序去专门生成控制文件，效率不错

 

A Data Store Process

oOriginal data is transmitted from other process constantly

oOne row of the data is such as 20110622~13100~3001~comment~

oThe 3rd field 3001 is a key field, with this  identifier we can know which table should this data row be  stored.

oThe function classifies the data rows by the key field of each row at first

 

 

oTo deal with each group that has been classified from the mass data flow, a buffer in the memory has been set to store it

oWhen the buffer  is full, flush the data into the disk file, which is a temporary file

 

 

 

oWhen the temporary file achieve the appointed size, rename it so that the data store  process can deal with it safely

oAt last, data store process put the data of the files into Oracle database.

oThe data store process can call the sqlldr command that supported by Oracle or call the oci functions to do it.