线上使用php的客户端是 https://github.com/weiboad/kafka-php 版本:dev-master
线上经常报错误:
Could not write 1422 bytes to stream
追查源码后发现在CommonSocket.php 文件中 writeBlocking方法
public function writeBlocking(string $buffer): int
{
// fwrite to a socket may be partial, so loop until we
// are done with the entire buffer
$failedAttempts = 0;
$bytesWritten = 0;
$bytesToWrite = strlen($buffer);
while ($bytesWritten < $bytesToWrite) {
// wait for stream to become available for writing
$writable = $this->select([$this->stream], $this->sendTimeoutSec, $this->sendTimeoutUsec, false);
if ($writable === false) {
throw new Exception\Socket('Could not write ' . $bytesToWrite . ' bytes to stream');
}
if ($writable === 0) {
$res = $this->getMetaData();
if (! empty($res['timed_out'])) {
throw new Exception('Timed out writing ' . $bytesToWrite . ' bytes to stream after writing ' . $bytesWritten . ' bytes');
}
throw new Exception\Socket('Could not write ' . $bytesToWrite . ' bytes to stream');
}
if ($bytesToWrite - $bytesWritten > self::MAX_WRITE_BUFFER) {
// write max buffer size
$wrote = fwrite($this->stream, substr($buffer, $bytesWritten, self::MAX_WRITE_BUFFER));
} else {
// write remaining buffer bytes to stream
$wrote = fwrite($this->stream, substr($buffer, $bytesWritten));
}
if ($wrote === -1 || $wrote === false) {
throw new Exception\Socket('Could not write ' . strlen($buffer) . ' bytes to stream, completed writing only ' . $bytesWritten . ' bytes');
}
if ($wrote === 0) {
// Increment the number of times we have failed
$failedAttempts++;
if ($failedAttempts > $this->maxWriteAttempts) {
throw new Exception\Socket('After ' . $failedAttempts . ' attempts could not write ' . strlen($buffer) . ' bytes to stream, completed writing only ' . $bytesWritten . ' bytes');
}
} else {
// If we wrote something, reset our failed attempt counter
$failedAttempts = 0;
}
$bytesWritten += $wrote;
}
return $bytesWritten;
}
w r i t a b l e = = = f a l s e 这 一 行 抛 出 的 异 常 , 追 查 代 码 发 现 调 用 的 是 s t r e a m s e l e c t ( writable === false 这一行抛出的异常, 追查代码发现调用的是 stream_select( writable===false这一行抛出的异常,追查代码发现调用的是streamselect(null, $sockets, $null, $timeoutSec, $timeoutUsec); 方法
stream_select 是一个事件模型, 看文档知道 需要传入$socktes, readSocket, writeSocket, ExceptionSocket,超时秒数 和超时毫秒数 那为什么会返回false 呢
文档解释
https://www.php.net/manual/en/function.stream-select.php
成功时,stream_select()返回已修改数组中包含的流资源的数量,如果超时在任何有趣的事件发生之前到期,则可能为零。FALSE 返回错误并引发警告(如果系统调用被传入信号中断,则可能发生此情况)。
那说明有错误引起的
据需看代码看 到创建socket
protected function createSocket(string $remoteSocket, $context, ?int &$errno, ?string &$errstr)
{
return stream_socket_client(
$remoteSocket,
$errno,
$errstr,
$this->sendTimeoutSec + ($this->sendTimeoutUsec / 1000000),
STREAM_CLIENT_CONNECT,
$context
);
}
传入的timeout时间是
t
h
i
s
−
>
s
e
n
d
T
i
m
e
o
u
t
S
e
c
+
(
this->sendTimeoutSec + (
this−>sendTimeoutSec+(this->sendTimeoutUsec / 1000000), 计算默认默认值是0.1s
这个值太小了 会导致socket超时直接断开, 在stream_select时候 socket已经断开 返回false
将值改大后实验 不在报错误了 确实是这个值引起的
将源码值改为:
**
* Send timeout in seconds.
*
* @var int
*/
protected $sendTimeoutSec = 300;
/**
* Send timeout in microseconds.
*
* @var int
*/
protected $sendTimeoutUsec = 100000;
/**
* Recv timeout in seconds
*
* @var int
*/
protected $recvTimeoutSec = 300;
/**
* Recv timeout in microseconds
*
* @var int
*/
protected $recvTimeoutUsec = 750000;
附我的kafka配置:
$config = \Kafka\ProducerConfig::getInstance();
$config->setMetadataRefreshIntervalMs(10000);
$config->setMessageMaxBytes(1000000000);
$config->setMetadataBrokerList($kafka_config['url']);
$config->setBrokerVersion('1.1.1');
$config->setSecurityProtocol($kafka_config['securityprotocol']);
$config->setSaslMechanism($kafka_config['saslmechanism']);
$config->setSaslUsername($kafka_config['saslusername']);
$config->setSaslPassword($kafka_config['saslpassword']);
$config->setRequiredAck(1);
$config->setIsAsyn(false);
$config->setProduceInterval(500);
$config->setRequestTimeout(900000);
$config->setTimeout(900000); //kafka 请求超时控制
$producer = new \Kafka\Producer();
$producer->setLogger($logger);
默认值都设置最大吧 保证不会出错 哈哈哈
我将项目拉了一份修改后放到了自己的github中
https://github.com/jeevi-cao/kafka-php
码字不易