解决办法:
给主RecyclerView设置属性
android:descendantFocusability="blocksDescendants"
<android.support.v7.widget.RecyclerView
android:id="@+id/recycler_view"
android:layout_width="match_parent"
android:descendantFocusability="blocksDescendants"
android:layout_height="match_parent"
android:cacheColorHint="@android:color/transparent" />