Substrate - ImportQueue or BlockImport Pipeline

Trait

/// Blocks import queue API.
///
/// The `import_*` methods can be called in order to send elements for the import queue to verify.
/// Afterwards, call `poll_actions` to determine how to respond to these elements.
pub trait ImportQueue<B: BlockT>: Send {
	/// Import bunch of blocks.
	fn import_blocks(&mut self, origin: BlockOrigin, blocks: Vec<IncomingBlock<B>>);
	/// Import block justifications.
	fn import_justifications(
		&mut self,
		who: Origin,
		hash: B::Hash,
		number: NumberFor<B>,
		justifications: Justifications,
	);
	/// Polls for actions to perform on the network.
	///
	/// This method should behave in a way similar to `Future::poll`. It can register the current
	/// task and notify later when more actions are ready to be polled. To continue the comparison,
	/// it is as if this method always returned `Poll::Pending`.
	fn poll_actions(&mut self, cx: &mut futures::task::Context, link: &mut dyn Link<B>);
}

BasicQueue

A common implementation of ImportQueue.

It will start a task to listen to the network events polled by network worker.

‘fn import_many_blocks()’ will really do the import. The import handler/verifier is depending on the consensus the node is using, Aura, Babe or smething else.

Note that the import result will be send to the BufferedLinkReceiver held by the BasicQueue. This BufferedLinkReceiver will also be polled by Network worker to proceee the results of import. See: buffered_link::buffered_link(). The BufferLinkSender implements the Link trait which can send result message to the Link. A NetworkLink is used in Network Worker, but a WaitLink is used in CLI::CheckImportBlocks.

/// The process of importing blocks.
///
/// This polls the `block_import_receiver` for new blocks to import and than awaits on
/// importing these blocks. After each block is imported, this async function yields once
/// to give other futures the possibility to be run.
///
/// Returns when `block_import` ended.
async fn block_import_process<B: BlockT, Transaction: Send + 'static>(
	mut block_import: BoxBlockImport<B, Transaction>,
	mut verifier: impl Verifier<B>,
	mut result_sender: BufferedLinkSender<B>,
	mut block_import_receiver: TracingUnboundedReceiver<worker_messages::ImportBlocks<B>>,
	metrics: Option<Metrics>,
	delay_between_blocks: Duration,
) {
	loop {
		let worker_messages::ImportBlocks(origin, blocks) = match block_import_receiver.next().await
		{
			Some(blocks) => blocks,
			None => {
				log::debug!(
					target: "block-import",
					"Stopping block import because the import channel was closed!",
				);
				return
			},
		};

		let res = import_many_blocks(
			&mut block_import,
			origin,
			blocks,
			&mut verifier,
			delay_between_blocks,
			metrics.clone(),
		)
		.await;

		result_sender.blocks_processed(res.imported, res.block_count, res.results);
	}
}

NetworkLink

Callbacks for sending import results to the network peers.

// Implementation of `import_queue::Link` trait using the available local variables.
struct NetworkLink<'a, B: BlockT> {
	protocol: &'a mut Swarm<B>,
}

impl<'a, B: BlockT> Link<B> for NetworkLink<'a, B> {
	fn blocks_processed(
		&mut self,
		imported: usize,
		count: usize,
		results: Vec<(Result<BlockImportStatus<NumberFor<B>>, BlockImportError>, B::Hash)>,
	) {
		self.protocol
			.behaviour_mut()
			.user_protocol_mut()
			.on_blocks_processed(imported, count, results)
	}
	fn justification_imported(
		&mut self,
		who: PeerId,
		hash: &B::Hash,
		number: NumberFor<B>,
		success: bool,
	) {
		self.protocol.behaviour_mut().user_protocol_mut().justification_import_result(
			who,
			hash.clone(),
			number,
			success,
		);
	}
	fn request_justification(&mut self, hash: &B::Hash, number: NumberFor<B>) {
		self.protocol
			.behaviour_mut()
			.user_protocol_mut()
			.request_justification(hash, number)
	}
}

How it works

ImportQueue is created from BlockImport & JustificationImport. Also a BlockImportWorker is spawned to listen to the channel for any incoming blocks.

babe::import_queue() -> BasicQueue::new(block_import…) -> spawn BlockImportWorker and return the queue handler.

NetworkWorker is polling import_queue handler in the worker main loop. The import queue was passed at initialization.

The Network event BlockImport/JustificationImport will trigger import_queue::…, which in turn send messgaes to the inner BlockImport for further block processing.

				Poll::Ready(SwarmEvent::Behaviour(BehaviourOut::BlockImport(origin, blocks))) => {
					if let Some(metrics) = this.metrics.as_ref() {
						metrics.import_queue_blocks_submitted.inc();
					}
					this.import_queue.import_blocks(origin, blocks);  // this will send the blocks to the BasicQueue worker task...
				},
				Poll::Ready(SwarmEvent::Behaviour(BehaviourOut::JustificationImport(
					origin,
					hash,
					nb,
					justifications,
				))) => {
					if let Some(metrics) = this.metrics.as_ref() {
						metrics.import_queue_justifications_submitted.inc();
					}
					this.import_queue.import_justifications(origin, hash, nb, justifications);
				},