Talk Is Cheap,Show Me The Code-GoLang动态监听chan

---

Talk Is Cheap ,Show Me The Code

• 链接在这,使用方式在README上
• 支持任意数据类型的chan 监听,自动回滚,升级
• 适用于长链接的任何场景

通常而言,监听chan的做法为(忽略退出条件,以及close情况:))

生产消费模型:
func TestForSelect(t *testing.T) {
	c := make(chan int)
	go func() {
		for {
			select {
			case v := <-c:
				fmt.Println(v)
			}
		}
	}()
	go func() {
		for {
			time.Sleep(time.Second)
			c <- 1
		}
	}()
	select {}
}

当有另外一个chan b的时候,当然可以照木照样的抄一遍,也可以如下,合并到一起:

c := make(chan int)
	c2 := make(chan int)
	go func() {
		for {
			select {
			case v := <-c:
				fmt.Println(v)
			case v2 := <-c2:
				fmt.Println(v2)
			}
		}
	}()
	go func() {
		for {
			time.Sleep(time.Second)
			c <- 1
			c2 <- 2
		}
	}()
	select {}

问题

• 管理问题:在编码阶段,我们可以是确定有哪些是固定的internalChan,可以合并到一起,也可以单独一个goroutine ,但是随着程序的运行必然会有golang 提倡的通信机制而自定义chan,那么,这么多的chan ,能否都统一归并到一起,尤其对于生命周期等同于整个程序的动态chan的时候
• 性能问题:性能问题基本涉及到的就是空间和时间问题,
  • 当用空间换时间: 既一个chan 一个goroutine的时候,golang 的MPG模型,终归会还是回到os Thread上,并且虽然goroutine是用户态概念(2k),但是,这并不能代表,可以随意的创建 ,当 chan 数量上升时,并不能保证程序有最好的吞吐量
  • 当时间换空间: 既n个chan 合并到一个goroutine的时候,当数量上升时,吞吐量不一定就会比goroutine低(原因: 因为MPG模型,真正的调用还是得靠kernal Thread)

性能问题

• goroutine模式

  • 既一个chan 一个goroutine

  • func goroutine(chans ...<-chan int) <-chan int {
    	r := make(chan int, 1)
    	wg := sync.WaitGroup{}
    	wg.Add(len(chans))
    	go func() {
    		for i := 0; i < len(chans); i++ {
    			go func(index int) {
    				defer wg.Done()
    				for v := range chans[index] {
    					r <- v
    				}
    			}(i)
    		}
    		wg.Wait()
    		close(r)
    	}()
    	return r
    }
    

• goroutineMerge模式

  • 既将routine进行合并输出

  • func mergeN(chans ...<-chan int) <-chan int {
    	r := make(chan int, 1)
    	go func() {
    		wg := sync.WaitGroup{}
    		wg.Add(len(chans))
    		for _, c := range chans {
    			go func(c <-chan int) {
    				for v := range c {
    					r <- v
    				}
    				wg.Done()
    			}(c)
    		}
    		wg.Wait()
    		close(r)
    	}()
    
    	return r
    }
    

  • 

• 反射reflect 模式

  • 既通过golang 底层的反射进行数据监听

  • func mergeReflect(chans ...<-chan int) <-chan int {
    	out := make(chan int)
    	go func() {
    		defer close(out)
    		var cases []reflect.SelectCase
    		for _, c := range chans {
    			cases = append(cases, reflect.SelectCase{
    				Dir:  reflect.SelectRecv,
    				Chan: reflect.ValueOf(c),
    			})
    		}
    
    		for len(cases) > 0 {
    			i, v, ok := reflect.Select(cases)
    			if !ok {
    				cases = append(cases[:i], cases[i+1:]...)
    				continue
    			}
    			out <- v.Interface().(int)
    		}
    	}()
    	return out
    }
    

• selectn 模式

  • 既for select 模式,不同的是,内部为多个chan,篇幅太长,请看下面的github链接

• Talk Is Cheap,Show Me The Code

• 附上benchmark的结果:

BenchmarkMerge/goroutines/1-8         	   86295	     13567 ns/op	     128 B/op	       2 allocs/op
BenchmarkMerge/goroutines/2-8         	   72342	     16708 ns/op	     129 B/op	       2 allocs/op
BenchmarkMerge/goroutines/4-8         	   44443	     26431 ns/op	     130 B/op	       2 allocs/op
BenchmarkMerge/goroutines/8-8         	   22413	     53346 ns/op	     133 B/op	       2 allocs/op
BenchmarkMerge/goroutines/16-8        	   11371	    106253 ns/op	     137 B/op	       2 allocs/op
BenchmarkMerge/goroutines/32-8        	    5449	    191332 ns/op	     150 B/op	       2 allocs/op
BenchmarkMerge/goroutines/64-8        	    3517	    363736 ns/op	     173 B/op	       2 allocs/op
BenchmarkMerge/goroutines/128-8       	    1489	    700969 ns/op	     270 B/op	       3 allocs/op
BenchmarkMerge/goroutines/256-8       	     865	   1386511 ns/op	     574 B/op	       6 allocs/op
BenchmarkMerge/goroutines/512-8       	     441	   2698255 ns/op	    1703 B/op	      18 allocs/op
BenchmarkMerge/goroutines/1024-8      	     210	   5688581 ns/op	    7626 B/op	      80 allocs/op
BenchmarkMerge/goroutines/2048-8      	      90	  12348973 ns/op	   36526 B/op	     378 allocs/op
BenchmarkMerge/goroutines/4096-8      	      44	  24618458 ns/op	   75359 B/op	     782 allocs/op
BenchmarkMerge/goroutineMerge/1-8     	  109533	     12741 ns/op	     128 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/2-8     	   73844	     17012 ns/op	     129 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/4-8     	   44526	     27054 ns/op	     130 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/8-8     	   22270	     53264 ns/op	     133 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/16-8    	   10000	    105712 ns/op	     139 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/32-8    	    5514	    199118 ns/op	     148 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/64-8    	    3096	    371435 ns/op	     158 B/op	       2 allocs/op
BenchmarkMerge/goroutineMerge/128-8   	    1798	    673136 ns/op	     245 B/op	       3 allocs/op
BenchmarkMerge/goroutineMerge/256-8   	     888	   1336941 ns/op	     521 B/op	       6 allocs/op
BenchmarkMerge/goroutineMerge/512-8   	     440	   2759375 ns/op	    1380 B/op	      15 allocs/op
BenchmarkMerge/goroutineMerge/1024-8  	     210	   5810853 ns/op	    5734 B/op	      60 allocs/op
BenchmarkMerge/goroutineMerge/2048-8  	      84	  12583786 ns/op	   19938 B/op	     208 allocs/op
BenchmarkMerge/goroutineMerge/4096-8  	      48	  25377268 ns/op	   76672 B/op	     799 allocs/op
BenchmarkMerge/reflection/1-8         	   62776	     31075 ns/op	     613 B/op	      46 allocs/op
BenchmarkMerge/reflection/2-8         	   29730	     37526 ns/op	    1778 B/op	     110 allocs/op
BenchmarkMerge/reflection/4-8         	   17108	     73674 ns/op	    6210 B/op	     293 allocs/op
BenchmarkMerge/reflection/8-8         	    5742	    192402 ns/op	   41349 B/op	     951 allocs/op
BenchmarkMerge/reflection/16-8        	    2185	    502843 ns/op	  163921 B/op	    3084 allocs/op
BenchmarkMerge/reflection/32-8        	     628	   1721796 ns/op	  653066 B/op	   10842 allocs/op
BenchmarkMerge/reflection/64-8        	     207	   5746011 ns/op	 2611283 B/op	   40417 allocs/op
BenchmarkMerge/reflection/128-8       	      45	  22606500 ns/op	10467497 B/op	  155855 allocs/op
BenchmarkMerge/reflection/256-8       	      12	  89188162 ns/op	41944716 B/op	  613093 allocs/op
BenchmarkMerge/reflection/512-8       	       3	 363011605 ns/op	167168538 B/op	 2421282 allocs/op
BenchmarkMerge/reflection/1024-8      	       1	1541255668 ns/op	668969352 B/op	 9673817 allocs/op
BenchmarkMerge/reflection/2048-8      	       1	6367310291 ns/op	2679574976 B/op	38507950 allocs/op
BenchmarkMerge/reflection/4096-8      	       1	27469170016 ns/op	10684780424 B/op	154077029 allocs/op
BenchmarkMerge/selectn/1-8            	  102465	     15167 ns/op	     120 B/op	       3 allocs/op
BenchmarkMerge/selectn/2-8            	   44065	     27441 ns/op	     112 B/op	       2 allocs/op
BenchmarkMerge/selectn/4-8            	   16956	     71150 ns/op	     113 B/op	       2 allocs/op
BenchmarkMerge/selectn/8-8            	   10000	    193766 ns/op	     115 B/op	       2 allocs/op
BenchmarkMerge/selectn/16-8           	    3488	    542824 ns/op	     119 B/op	       2 allocs/op
BenchmarkMerge/selectn/32-8           	    2762	    433152 ns/op	     118 B/op	       2 allocs/op
BenchmarkMerge/selectn/64-8           	     975	   1368257 ns/op	     123 B/op	       2 allocs/op
BenchmarkMerge/selectn/128-8          	     654	   1747111 ns/op	     130 B/op	       2 allocs/op
BenchmarkMerge/selectn/256-8          	     433	   2708657 ns/op	     260 B/op	       3 allocs/op
BenchmarkMerge/selectn/512-8          	     304	   3865518 ns/op	     350 B/op	       4 allocs/op
BenchmarkMerge/selectn/1024-8         	     178	   6767176 ns/op	     573 B/op	       6 allocs/op
BenchmarkMerge/selectn/2048-8         	      96	  12775303 ns/op	    1932 B/op	      20 allocs/op
BenchmarkMerge/selectn/4096-8         	      49	  25543083 ns/op	    4819 B/op	      51 allocs/op

解决方案

• 管理问题:
  • 采取统一的管理器,管理所有的channel
  • 当channel结束之后,自动进行复用
• 性能问题
  • 自动提供升级和回滚策略,既当chan 达到某个数量时,自动升级(goroutine=>reflect=>selectn的形式),回滚同理