PHP读取大文本文件并处理数据的思路

//处理文件
    $file = fopen($filename, "r") or exit("Unable to open file!");
    $total_line = 0;
    while(!feof($file)){
        $f = fgets($file);
        $total_line++; //总行数
    } 
    
    $for_count = intval($total_line / 1000);
    $txt=file($filename);
　　//$txt[0] 表示第一行， $txt[1] 表示第二行， ...
   for($m=0;$m<=$for_count;$m++){
        $start = $m * 1000;
        $end = $start + 1000;
        if($end > $total_line){
            $end = $total_line;
        }
        //echo $start.'---'.$end."\n";
        $sql_value = '';
        for($i=$start;$i<$end;$i++){
            if($txt[$i]){
                $data = explode(",", $txt[$i]);
                $data[2] = "'".$data[2]."'";  
                $data[28] = "'".str_replace("\n", "", $data[28])."'";
                
                //转换得到sql语句的值
                $sql_value .= "(".implode(",", $data)."),";
                unset($data);
            }
        }
        $sql_value = substr($sql_value, 0, -1);
        $add_sql = "insert into tbname ({$fields}) values ".$sql_value;
        
        $add_res = mysqli_query($conn,$add_sql);
        if ($add_res) {
            echo 'OK';
        }else{
            echo 'Error';
            file_put_contents($file_err_name.$m, $add_sql);
        }

    }

 

【附】不分条一次性处理整个文本：

 1 //处理文件
 2     $file = fopen($filename, "r") or exit("Unable to open file!");
 3     $sql_value = '';
 4     
 5     while(!feof($file)){
 6         $f = fgets($file);
 7         if($f){
 8             $data = explode(",", $f);
 9             $data[2] = "'".$data[2]."'";           
19             $data[28] = "'".str_replace("\n", "", $data[28])."'";             
21             //转换得到sql语句的值
22             $sql_value .= "(".implode(",", $data)."),";
23         }
24         
25         unset($data);
26     }
27     
28     //添加入库
29     $sql_value = substr($sql_value, 0, -1);
30     $add_sql = "insert into tbname ({$fields}) values ".$sql_value;
31     
32     $add_res = mysqli_query($conn,$add_sql);
33     if ($add_res) {
34         echo 'OK'; 
35     }else{
36         echo 'Error';
37         file_put_contents($file_err_name, $add_sql);
38     }
39     unset($sql_value);
40     
41     fclose($file);

 

转载于:https://www.cnblogs.com/rxbook/p/6532655.html