关于多线程io能否提高程序效率

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/maopeng0535/article/details/47334441

项目中,需要对30个文件进行读取并处理数据。

想到能不能通过多线程,每个线程处理一部分文件来提高程序效率,实测发现效率反而下降了。

通过查阅资料,发现单个硬盘只能响应一个io,cpu是多核单硬盘不是。

多线程切换时,硬盘去不同的地方读东西会增加耗时。

解决方法:

1、多个硬盘存取

2、换用固态硬盘提升硬盘本身的性能

3、将文件都读到内存,然后多线程处理数据


展开阅读全文

关于 如何提高程序效率

01-19

最近阅读了拜读了《让你的软件飞起来》,感觉受益匪浅,但在实际实践时,发现了一些问题,特反馈于此,与大家一块讨论。。。 rnrn 该文主要提到如下几种提高程序执行速度的方法:rn 1. 可能的话,尽量少用 浮点运算,特别是对于一些不支持浮点运算的CPU;rn 2. 可能的话,尽量以位移运算代替除法运算;rn 3. 可能的话,尽量以查表法代替乘除运算;rn 4. 可能的话,采用宏、inline等方式来减少程序的调度;rn 理论上来说,这些方法都没有问题,但实际编码演练后,发现了一些问题,请大家指教:rnrn一、实验过程表述,源码见附件 rn 1. 实验环境 rn OS:linux 2.6, rn CPU:S3C2440 rn RAM:32M rn 编译器:arm-linux rn 示例:将rgb图像转化为黑白图像,主要涉及以下式子:Y = 0.299 * R + 0.587 * G + 0.114 * Brnrn 执行程序时,用 time 指令来查看执行时间。 rn 2. 实验数据(以您的作品中的例子来讲) rn 1) 第一次试跑(直接计算上面的式子,采用了浮点运算): rn real 0m 10.17s rn user 0m 0.45s rn sys 0m 9.73s rnrn 2) 去掉浮点运算(y = (299/1000*r) + (587/1000*g) + (114/1000*b)): rn real 0m 0.11s rn user 0m 0.10s rn sys 0m 0.01s rnrn 3) 将除法运算修改为 位移 运算 (y = ((1225*r + 2404*g + 467*b)>>12);)rn real 0m 0.16s rn user 0m 0.15s rn sys 0m 0.01s rnrn 4) 改用 查表法 rn 本方案的原理:rn D = 0.299 * R; E = 0.587 * G; F = 0.114 * Brn rgb图像中r、g、b三个分量的取值均在 0-255 之间(unsined char型),故先将D、E、F可能的取值存在D[256]、E[256]、F[256]三个变量中,则上面的式子转换为:Y = D[i] + E[i] + F[i],这样就只剩下加法运算了rn real 0m 0.13s rn user 0m 0.11s rn sys 0m 0.03s rnrn 5) 查表法 + 表项改为char型 + inline (主要是在子函数前加上 inline )rn real 0m 0.13s rn user 0m 0.12s rn sys 0m 0.00s rnrn二、疑问: rn 1. 将除法运算修改为 位移 运算后,时间反而加长 rn 2. 与整数除法相比,查表法 和 位移运算并未见优势rnrn 论坛

求指导,程序效率提高

07-09

以下这段代码如何修改rn可以使效率大幅提升,忘指导rn运用这段代码生成订单700行速度大约在20秒左右rn求指导,如何修改可以使运行更快rnrn[code=vb]Private Sub 生成订单()rn Dim 排序号 As Integerrn Dim 地域 As Stringrn Dim 负责人 As Stringrn Dim 酒店 As Stringrn Dim 订单编号 As Stringrn Frame3.Visible = Truern Call 更新排序号rnrn Set excel1 = New Excel.Applicationrn excel1.Visible = Falsern excel1.Workbooks.Addrn With excel1.Sheets("sheet1")rn .Cells.Font.Size = 10rn Adodc1.Recordset.MoveFirstrn .Cells(1, 3) = Trim(Adodc1.Recordset.Fields("日期"))rn .Cells(1, 3).Font.Size = 16rn .Cells(1, 3).Font.Name = "黑体"rn .Cells(1, 3).NumberFormatLocal = "[$-F800]dddd, mmmm dd, yyyy"rn .Range(excel1.Sheets("sheet1").Cells(1, 3), excel1.Sheets("sheet1").Cells(1, 6)).HorizontalAlignment = xlCenterrn .Range(excel1.Sheets("sheet1").Cells(1, 3), excel1.Sheets("sheet1").Cells(1, 6)).VerticalAlignment = xlCenterrn .Range(excel1.Sheets("sheet1").Cells(1, 3), excel1.Sheets("sheet1").Cells(1, 6)).Mergern .Columns("a:a").ColumnWidth = 37.5rn .Columns("b:b").ColumnWidth = 5.5rn .Columns("c:c").ColumnWidth = 4.5rn .Columns("d:d").ColumnWidth = 37.5rn .Columns("e:e").ColumnWidth = 5.5rn .Columns("f:f").ColumnWidth = 4.5rn With .Columns("A:F")rn .Borders(xlDiagonalDown).LineStyle = xlNonern .Borders(xlDiagonalUp).LineStyle = xlNonern .Borders(xlEdgeLeft).Weight = xlHairlinern .Borders(xlEdgeTop).Weight = xlHairlinern .Borders(xlEdgeBottom).Weight = xlHairlinern .Borders(xlEdgeRight).Weight = xlHairlinern .Borders(xlInsideVertical).Weight = xlHairlinern .Borders(xlInsideHorizontal).Weight = xlHairlinern End Withrn .Cells(1, 1) = "**公司客户订单"rn .Rows(1).RowHeight = 20rn With .Range(.Cells(1, 1), .Cells(1, 2))rn .HorizontalAlignment = xlRightrn .Font.Name = "黑体"rn .Font.Size = 16rn .Mergern End Withrn With .PageSetuprn .LeftMargin = Application.InchesToPoints(0.236220472440945)rn .RightMargin = Application.InchesToPoints(0.236220472440945)rn .TopMargin = Application.InchesToPoints(0.393700787401575)rn .BottomMargin = Application.InchesToPoints(0.393700787401575)rn .HeaderMargin = Application.InchesToPoints(0.31496062992126)rn .FooterMargin = Application.InchesToPoints(0.31496062992126)rn .PrintTitleRows = "$1:$2"rn End Withrnrn .Cells(2, 1) = "货物名称"rn .Cells(2, 2) = "订货量"rn .Cells(2, 3) = "出库量"rn .Cells(2, 4) = "货物名称"rn .Cells(2, 5) = "订货量"rn .Cells(2, 6) = "出库量"rn a = 3rn ProgressBar1.Max = Adodc1.Recordset.RecordCountrn ProgressBar1.Value = 0rn For i = 1 To Adodc1.Recordset.RecordCountrn If 地域 <> Trim(Adodc1.Recordset.Fields("地域")) Or i = 1 Thenrn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("地域"))rn .Cells(a, 1).Font.Name = "宋体"rn .Cells(a, 1).Font.Size = 11rn .Cells(a, 1).Font.FontStyle = "加粗"rn a = a + 1rn End Ifrn If 负责人 <> Trim(Adodc1.Recordset.Fields("负责人车辆号")) Or i = 1 Thenrn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("负责人车辆号"))rn .Cells(a, 1) = "负责人:" & .Cells(a, 1)rn .Cells(a, 1).Font.Name = "宋体"rn .Cells(a, 1).Font.Size = 11rn .Cells(a, 1).Font.FontStyle = "加粗"rn .Cells(a, 1).Interior.Pattern = xlSolidrn .Cells(a, 1).Interior.Color = 255rn a = a + 1rn End Ifrn If 酒店 <> Trim(Adodc1.Recordset.Fields("酒店名称")) Or i = 1 Thenrn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("酒店名称"))rn .Cells(a, 1).HorizontalAlignment = xlCenterrn .Cells(a, 1).VerticalAlignment = xlCenterrn .Cells(a, 1).Font.Name = "宋体"rn .Cells(a, 1).Font.Size = 11rn .Cells(a, 1).Font.FontStyle = "加粗"rn a = a + 1rn End Ifrn If 订单编号 <> Trim(Adodc1.Recordset.Fields("订单编号")) Or i = 1 Thenrn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("厨房")) & "(" & Trim(Adodc1.Recordset.Fields("订单编号")) & ")"rn .Cells(a, 1).Font.Name = "宋体"rn .Cells(a, 1).Font.Size = 9rn .Cells(a, 1).Font.FontStyle = "加粗"rn a = a + 1rn End Ifrn If Trim(Adodc1.Recordset.Fields("备注")) <> "" Thenrn '有备注的情况rn If LenB(Trim(Adodc1.Recordset.Fields("酒店货物名称")) & " ¥" & Trim(Adodc1.Recordset.Fields("酒店单价")) & " 备注:" & Trim(Adodc1.Recordset.Fields("备注"))) > 61 Thenrn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("酒店货物名称")) & " ¥" & Trim(Adodc1.Recordset.Fields("酒店单价"))rn .Cells(a + 1, 1) = "备注:" & Trim(Adodc1.Recordset.Fields("备注"))rn .Cells(a, 2) = Trim(Adodc1.Recordset.Fields("订货量"))rn '预留填写出库数量rn .Cells(a, 3) = ""rn a = a + 1rn Elsern .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("酒店货物名称")) & " ¥" & Trim(Adodc1.Recordset.Fields("酒店单价")) & " 备注:" & Trim(Adodc1.Recordset.Fields("备注"))rn .Cells(a, 2) = Trim(Adodc1.Recordset.Fields("订货量"))rn '预留填写出库数量rn .Cells(a, 3) = ""rn End Ifrn Elsern '没有备注的情况下rn .Cells(a, 1) = Trim(Adodc1.Recordset.Fields("酒店货物名称")) & " ¥" & Trim(Adodc1.Recordset.Fields("酒店单价"))rn .Cells(a, 2) = Trim(Adodc1.Recordset.Fields("订货量"))rn '预留填写出库数量rn .Cells(a, 3) = ""rnrn End Ifrn a = a + 1rn 地域 = Trim(Adodc1.Recordset.Fields("地域"))rn 负责人 = Trim(Adodc1.Recordset.Fields("负责人车辆号"))rn 酒店 = Trim(Adodc1.Recordset.Fields("酒店名称"))rn 订单编号 = Trim(Adodc1.Recordset.Fields("订单编号"))rn Adodc1.Recordset.MoveNextrn Label2.Caption = Int(ProgressBar1.Value * 100 / ProgressBar1.Max) & "%"rn DoEventsrn ProgressBar1.Value = ProgressBar1.Value + 1rn Nextrn End Withrn '========================================================================================rn Frame3.Visible = Falsern excel1.Visible = TruernrnEnd Sub[/code] 论坛

关于提高socket效率,select+多进程/多线程 付源码

11-06

hi,rn 小弟刚玩socket,日前完成了一个单个线程+select 的程序,大致功能就是可以将一个服务器A发送过来的内容转发到另一个服务器B同时将另一台服务器B发送过来的内容转发到这个服务器A。测试发现:rn1.单向传输1G文件,速率有8MB(百兆网口),但是双向同时传输文件,速率只有4MBrn2.希望能将程序改成多进程的或者多线程的来提高速率(即吞吐量)rnrn源码如下,小弟尝试在connect之后fork了2个子程序来实现(即一个进程只负责A-B,另一个只负责B-A)的操作,但是发现传文件的时候传不完就程序就断了,下面是我单个线程+select 的程序,有没高手指点一下如何修改???rnrn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rn#include rnrn#define SERVPORT 8090rn#define MAXDATASIZE 1024rnrnrnrnint do_con_n(char *address, int port)rnrn int s; rn struct sockaddr_in sin;rn fd_set set;rn struct timeval timeo = 15, 0; //time ou structrn socklen_t len = sizeof(timeo);rn int retval;rn int error;rn int code;rn rn s=socket(AF_INET, SOCK_STREAM, 0); rn // x=fcntl(s,F_GETFL,0); // Get socket flags rn fcntl(s, F_SETFL, fcntl(s, F_GETFL) | O_NONBLOCK); //设置为非阻塞模式 rn memset(&sin, 0, sizeof(struct sockaddr_in)); rn sin.sin_family=AF_INET; rn sin.sin_port=htons(port); rn //sin.sin_addr.s_addr=name_resolve(address); rn sin.sin_addr.s_addr = inet_addr(address);rn if(!sin.sin_addr.s_addr) return(-1); rn //printf("ip: %s\n",inet_ntoa(sin.sin_addr)); rn if(connect(s, (struct sockaddr *)&sin, sizeof(sin))!=0)rn rn if (errno != EINPROGRESS) rn rn printf("connect:normal network unreach!!");rn return -1;rn rn rn FD_ZERO(&set);/*将set清零使集合中不含任何fd*/ rn FD_SET(s,&set); /*将一个给定的文件描述符加入集合之中*/ rn retval = select(s + 1, NULL, &set, NULL, &timeo);rn if (retval == -1) rn rn printf("select");rn return -1;rn rn else if(retval == 0) rn rn printf("timeout\n"); //这样的select等于是变成了再timeout时间内是阻塞模式,超过timeout就直接返回rn printf("%d\n", time(NULL));rn return 0;rn rn elsern rn //printf("connected--->:[%d]\n",retval);rn if (FD_ISSET(s, &set)) rn rn error=0;rn len = sizeof(error); rn code = getsockopt(s, SOL_SOCKET, SO_ERROR, &error, &len); rn /* 如果发生错误,Solaris实现的getsockopt返回-1, rn * 把pending error设置给errno. Berkeley实现的 rn * getsockopt返回0, pending error返回给error. rn * 我们需要处理这两种情况 */ rn if (code < 0 || error) rn rn close(s); rn if (error) rn errno = error;rn printf("[getsockopt1]ERROR: code= %d ,errno = %d, strerror = %s \n" rn , code,errno, strerror(errno)); rn perror("getsockopt1:");rn return -1; rn rn printf("\nconnect to [%s] ok!\n",inet_ntoa(sin.sin_addr));rn rn else rn rn fprintf(stderr, "select error: sockfd not set"); rn exit(0); rn rn rn rn rn return(s); rnrnrnint SeanSend(int fd, void *buffer, int length) rn rn int bytes_left; rn int written_bytes; rn char *ptr=NULL; rn ptr=(char *)buffer; rn bytes_left=length; rn rn while( bytes_left>0) rn rn /* 开始写*/rn written_bytes=write(fd, ptr, bytes_left);rn //printf("\nin seasend:written_bytes=[%d]\n",written_bytes);rn if(written_bytes<=0) /* 出错了*/ rn rn if(errno==EINTR) /* 中断错误 继续写?*/ rn rn printf("[%s]:error errno==EINTR continue\n",__FUNCTION__); rn continue; rn rn rn else if(errno==EAGAIN) /* EAGAIN : Resource temporarily unavailable*/ rn rn //printf("[SeanSend]error errno==EAGAIN continue\n");rn usleep(1000);//等待防止cpu %100,希望发送缓冲区能得到释放 rn continue; rn rn rn else /* 其他错误 */ rn rn printf("[%s]:ERROR: errno = %d, strerror = %s \n",__FUNCTION__ rn , errno, strerror(errno)); rn return(-1); rn rn rn bytes_left-=written_bytes; rn ptr+=written_bytes;/* 从剩下的地方继续写?? */ rn rn return length; rn rnrnint main(int argc, char **argv)rnrn int sockfd, recvbytes;rn int sockfd1;rn int ret;rn char rcv_buf[MAXDATASIZE];rn char snd_buf[MAXDATASIZE];rn char buf[MAXDATASIZE];rn struct hostent *host;rn struct sockaddr_in serv_addr;rn struct sockaddr_in serv_addr1;rn fd_set rdfds;rn fd_set rdfds1;rn int srvport,srvport1;rn struct timeval timeo1 = 3, 0; //time1 ou structrn int rcvlen=0;rn int wrtlen=0;rn int rcvlen1=0;rn int wrtlen1=0;rn struct linger1rn rn int l_onoff;rn int l_linger;rn so_linger;rn rn so_linger.l_onoff=1;rn so_linger.l_linger=10;rn rn //setsockopt(sockfd,SOL_SOCKET, SO_LINGER, (const char *) &so_linger, sizeof(so_linger));rnrnrn if(argc != 7)rn rn fprintf(stderr, "Error!.\n");rn exit(1);rn rnrn sockfd=do_con_n(argv[2],atoi(argv[3]));rn sockfd1=do_con_n(argv[4],atoi(argv[5]));rn rn if( (-1==sockfd) || (-1==sockfd1) )rn rn printf("\nconnect error\n");rn return -1;rn rn elsern rn //printf("\nargv[6]=[%s],len=[%d]\n",argv[6],strlen(argv[6]));rn SeanSend(sockfd1,argv[6],strlen(argv[6]));rn rn rn setsockopt(sockfd,SOL_SOCKET, SO_LINGER, (const char *) &so_linger, sizeof(so_linger));rn setsockopt(sockfd1,SOL_SOCKET, SO_LINGER, (const char *) &so_linger, sizeof(so_linger));rnrn while(1)rn rn FD_ZERO(&rdfds);rn FD_SET(sockfd, &rdfds);rn FD_SET(sockfd1, &rdfds);rn rn //printf("\nsock=[%d],sock1=[%d],res=[%d]\n",sockfd,sockfd1,(sockfd>sockfd1?sockfd:sockfd1)+1);rn rn rn ret = select( (sockfd>sockfd1?sockfd:sockfd1)+1, &rdfds, NULL, NULL, NULL );rn if(ret<0)rn rn if(errno == EINTR)rn continue;rn perror("select:"); rn exit(1); rn rn //注意,当前的处理改用不加超时的select,cpu占用率更加低,少于%1rn else if(ret==0)rn rn //printf("\nret==0,timeout\n");rn //select在第一次超时之后会将timeo1的值修改为0,所以在下面的判断要做一个usleep,不然在timeout状态时cpu占用高达12%rn usleep(1000); //这样cpu处于timeout的时候 占用率会低rn continue;rn rn elsern rn /*if(FD_ISSET(0, &rdfds))rn rn memset(snd_buf, 0, sizeof(snd_buf));rn read(0, snd_buf, 256);rn snd_buf[strlen(snd_buf)] = '\0';rn write(sockfd, snd_buf, strlen(snd_buf)+1);rn rn */rn if(FD_ISSET(sockfd, &rdfds))rn rn //printf("go\n");rn memset(rcv_buf, 0x00, sizeof(rcv_buf));rn if((rcvlen=read(sockfd, rcv_buf, MAXDATASIZE))>0)rn rn printf("rcv:[%s],rcvlen=[%d]\n",rcv_buf,rcvlen);rn //write(sockfd, snd_buf, strlen(snd_buf)+1);rn /*if(write(sockfd1, rcv_buf, sizeof(rcv_buf))<0)rn rn perror("write:");rn */rn wrtlen=SeanSend(sockfd1,rcv_buf,rcvlen);rn //printf("\nwrtlen=[%d]\n",wrtlen);rn rn else if(0==rcvlen)rn rn //printf("\nclose wrtlen=[%d]\n",wrtlen);rn rn //printf("\nsockfed close:ip[%s],port[%d]\n",argv[1],atoi(argv[2]));rn //close(sockfd);rn break;rn rn elsern rn printf("sockfed ERROR: errno = %d, strerror = %s \n" rn , errno, strerror(errno)); rn rn rn rn rn if(FD_ISSET(sockfd1, &rdfds))rn rn //printf("go11\n");rn memset(rcv_buf, 0x00, sizeof(rcv_buf));rn if((rcvlen1=read(sockfd1, rcv_buf, MAXDATASIZE))>0)rn rn printf("rcv1:[%s],rcvlen=[%d]\n",rcv_buf,rcvlen1);rn /*if(write(sockfd, rcv_buf, rcvlen1)<0)rn rn perror("write:"); //缓冲区很快就满了,会出现部分写操作失败。。。rn rn */rn wrtlen1=SeanSend(sockfd,rcv_buf,rcvlen1);rn //SeanSend(sockfd,"send end",8);rn rn //printf("\nwrtlen1=[%d]\n",wrtlen1);rn rn else if(0==rcvlen1)rn rn //printf("\nclose wrtlen1=[%d]\n",wrtlen1);rn //SeanSend(sockfd1,"nihaoma",8);rn //printf("\nsockfed1 close:ip[%s],port[%d]\n",argv[1],atoi(argv[3]));rn //close(sockfd1);rn break;rn rn elsern rn printf("sockfed1 ERROR: errno = %d, strerror = %s \n" rn , errno, strerror(errno)); rn rn rn rn //continue;rn rn rn //sleep(1);rn close(sockfd);//TCP 单向关闭rn close(sockfd1);rn return 0;rnrn 论坛

如何提高程序效率?越来越慢

03-16

要实现的功能:从一个数据库里取数据,然后插入另一个数据库里。在main方法调用后可以运行,但运行速度随时间增长变慢,最初迁移一条主表数据(包括其下的明细表)需要5秒,到后边需要50秒还多,不知道原因在哪?rnrn为实现功能用到了2个for循环,第一个for先遍历原始主表数据,取其中一条,再根据这一条数据的SONO号获取原始明细表数据,第二个for遍历明细表并插入到我方明细表中。插完一条主表和其下的明细表,再插另一条主表以及明细,如此循环。其中用到了多个list和String[].rnrnpublic class Transfer rn public void AlldataChange() throws Exception rn Dbcon db = new Dbcon();//连接ebio数据库rn Dbconn db1 = new Dbconn();//连接long数据库rn Statement stmt = db.getConnection().createStatement();rn Statement stmt1 =db1.getConnection().createStatement();rn String sql = "select* from ...";// 查询对方主表数据rn ResultSet rs = stmt.executeQuery(sql);rn List l1 = new ArrayList();rn List l2 = new ArrayList();rn while (rs.next()) rn String[] s = new String[21];//定义数组s存放QAD主表数据rn s[0] = rs.getString("sono");rn .....rn .....rn s[20] = rs.getString("drafter");rn l1.add(s);//把从QAD按条件查询的主表数据封装到集合l1里面。rn rn for(Iterator it=l1.iterator();it.hasNext();) // 遍历主表的每一条数据rn String[] s1 = (String[])it.next();//定义数组s1获取l1集合中的值rn String sono = s1[0];rn ResultSet mxrs = stmt.executeQuery("select ... from ...");// 根据主表数据的合同号码,查询对方明细数据rn while (mxrs.next()) rn String[] s2 = new String[6];//定义数组s2存放QAD明细表的数据rn s2[0] = mxrs.getString("sono");rn s2[1] = mxrs.getString("line");rn s2[2] = mxrs.getString("itemno");rn s2[3] = mxrs.getString("qty");rn s2[4] = mxrs.getString("soprice");rn s2[5] = mxrs.getString("soamt");rn l2.add(s2);//把从QAD按条件查询的明细表数据封装到集合l2里面。rn rn // 查询己方主表是否存在此数据,一共17个字段rn ResultSet rs1 = stmt1.executeQuery("select ...");rn try rn if (rs1!=null) // 如果我方主表存在相同数据rn stmt1.executeUpdate("delete from ..");// 删除主表数据rn stmt1.executeUpdate("delete from..");// 删除明细表数据rn // 插入主表数据rn stmt1.executeUpdate("insert into ...");rn // 插入明细表数据rn for(Iterator it1=l2.iterator();it1.hasNext();) rn String[] s3 = (String[])it1.next();//定义数组s3获取l2集合的值 stmt1.executeUpdate("insert into ..."); elseSystem.err.println("合同号为" + s1[0] + "的数据迁移完毕");rn catch (Exception e) e.printStackTrace(); rn rn System.out.println("外销合同的所有数据导入完毕");rn System.exit(0);rn rnrnrnrnrn 论坛

没有更多推荐了,返回首页