ubuntu 主机虚拟交流自用

当然可以。为了诊断问题,我们可以在Mapper和Reducer的关键部分添加一些打印语句来确认流程和输出。这样,当你运行程序时,可以在Hadoop的日志输出中看到这些打印语句。

下面是修改后的部分代码:

1. **Mapper**:

```java
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, Text>.Context context)
        throws IOException, InterruptedException {
    String line = value.toString();
    String[] values = line.split(" ");
    
    System.out.println("Mapper Input Line: " + line);
    context.write(new Text(values[0]), new Text(values[1]+"0"));
    System.out.println("Mapper Output Key-Value: " + values[0] + " -> " + values[1]+"0");
    
    context.write(new Text(values[1]), new Text(values[0]+"1"));
    System.out.println("Mapper Output Key-Value: " + values[1] + " -> " + values[0]+"1");
}
```

2. **Reducer**:

```java
protected void reduce(Text key, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context)
        throws IOException, InterruptedException {
    List<String> grand = new ArrayList<String>();
    List<String> child = new ArrayList<String>();
    String a = "1";
    String b = "1";

    System.out.println("Reducer Input Key: " + key.toString());

    for(Text val:values){
        System.out.println("Reducer Input Value: " + val.toString());
        if(val.toString().endsWith("0")){
            grand.add(val.toString());
            a = "2";
        }else if(val.toString().endsWith("1")){
            child.add(val.toString());
            b = "2";
        }
    }

    if(a.equals("2") && b.equals("2")) {
        for(String grap:grand){
            for(String chis:child){
                context.write(new Text(chis.substring(0, chis.length()-1)), new Text(grap.substring(0, grap.length()-1)));
                System.out.println("Reducer Output Key-Value: " + chis.substring(0, chis.length()-1) + " -> " + grap.substring(0, grap.length()-1));
            }
        }
    }
}
```

这些打印语句会帮助你了解Mapper和Reducer在处理输入数据时的具体步骤和输出。

不过,请注意,由于Hadoop是一个分布式框架,这些打印信息可能会散布在不同的日志文件中,具体取决于哪个节点正在执行该任务。所以,当你检查日志时,确保查看所有相关的节点日志。

023-10-26 18:27:53,844 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2023-10-26 18:27:53,943 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Mapper Input Line: child  parent
Mapper Output Key-Value: child -> 0
Mapper Output Key-Value:  -> child1
Mapper Input Line: Steven  Lucy
Mapper Output Key-Value: Steven -> 0
Mapper Output Key-Value:  -> Steven1
Mapper Input Line: Steven  Jack
Mapper Output Key-Value: Steven -> 0
Mapper Output Key-Value:  -> Steven1
Mapper Input Line: Jone  Lucy
Mapper Output Key-Value: Jone -> 0
Mapper Output Key-Value:  -> Jone1
Mapper Input Line: Jone  Jack
Mapper Output Key-Value: Jone -> 0
Mapper Output Key-Value:  -> Jone1
Mapper Input Line: Lucy  Mary
Mapper Output Key-Value: Lucy -> 0
Mapper Output Key-Value:  -> Lucy1
Mapper Input Line: Lucy  Frank
Mapper Output Key-Value: Lucy -> 0
Mapper Output Key-Value:  -> Lucy1
Mapper Input Line: Jack  Alice
Mapper Output Key-Value: Jack -> 0
Mapper Output Key-Value:  -> Jack1
Mapper Input Line: Jack  Jesse
Mapper Output Key-Value: Jack -> 0
Mapper Output Key-Value:  -> Jack1
Mapper Input Line: David  Alice
Mapper Output Key-Value: David -> 0
Mapper Output Key-Value:  -> David1
Mapper Input Line: David  Jesse
Mapper Output Key-Value: David -> 0
Mapper Output Key-Value:  -> David1
Mapper Input Line: Philip  David
Mapper Output Key-Value: Philip -> 0
Mapper Output Key-Value:  -> Philip1
Mapper Input Line: Philip  Alma
Mapper Output Key-Value: Philip -> 0
Mapper Output Key-Value:  -> Philip1
Mapper Input Line: Mark  David
Mapper Output Key-Value: Mark -> 0
Mapper Output Key-Value:  -> Mark1
Mapper Input Line: Mark  Alma
Mapper Output Key-Value: Mark -> 0
Mapper Output Key-Value:  -> Mark1
2023-10-26 18:27:54,258 INFO mapred.LocalJobRunner:
2023-10-26 18:27:54,263 INFO mapred.MapTask: Starting flush of map output
2023-10-26 18:27:54,263 INFO mapred.MapTask: Spilling map output
2023-10-26 18:27:54,263 INFO mapred.MapTask: bufstart = 0; bufend = 232; bufvoid = 104857600

Reducer Input Key:
Reducer Input Value: Jone1
Reducer Input Value: Lucy1
Reducer Input Value: Mark1
Reducer Input Value: Steven1
Reducer Input Value: Philip1
Reducer Input Value: Steven1
Reducer Input Value: Philip1
Reducer Input Value: Jone1
Reducer Input Value: David1
Reducer Input Value: David1
Reducer Input Value: Jack1
Reducer Input Value: child1
Reducer Input Value: Jack1
Reducer Input Value: Mark1
Reducer Input Value: Lucy1
Reducer Input Key: David
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Jack
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Jone
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Lucy
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Mark
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Philip
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: Steven
Reducer Input Value: 0
Reducer Input Value: 0
Reducer Input Key: child
Reducer Input Value: 0

if (values.length < 2) { System.err.println("Invalid line: " + line); return; }

if (values[0].trim().isEmpty() || values[1].trim().isEmpty()) { System.err.println("Empty value detected in line: " + line); return; }

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值