日均百亿数据入库，Flink如何让这家骑行俱乐部实现精准推荐？

RunningShare

于 2025-10-05 23:07:27 发布

阅读量597

点赞数 14

CC 4.0 BY-SA版权

分类专栏： Flink 大数据文章标签： flink 大数据

本文链接：https://blog.csdn.net/JacksonKing/article/details/152565868

大数据同时被 2 个专栏收录

104 篇文章

订阅专栏

Flink

22 篇文章

订阅专栏

#【双节征文】月满华诞 · 码向未来--代码寄明月，指尖庆华诞#

日均百亿数据入库，Flink如何让这家骑行俱乐部实现精准推荐？

一、业务背景与技术架构

某骑行俱乐部拥有百万会员，每日产生超百亿条骑行数据。传统批处理架构无法满足实时统计和个性化推荐需求，基于此我们构建了Flink实时计算平台。

技术栈组成：

数据采集：Kafka
实时计算：Flink
特征存储：Redis + HBase
监控告警：Prometheus

二、实时数据统计核心实现

2.1 多维度骑行统计

public class RidingStatisticsJob {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        
        // Kafka数据源
        DataStream<RidingRecord> ridingStream = env
            .addSource(KafkaSource.createRidingSource())
            .name("riding-data-source");
            
        // 区域热度统计 - 5分钟滚动窗口
        DataStream<AreaHeat> areaHeatStream = ridingStream
            .keyBy(RidingRecord::getAreaId)
            .window(TumblingProcessingTimeWindows.of(Time.minutes(5)))
            .aggregate(new AreaHeatAggregator());
            
        // 用户能力分析 - 1小时滑动窗口
        DataStream<MemberAbility> abilityStream = ridingStream
            .keyBy(RidingRecord::getMemberId)
            .window(SlidingProcessingTimeWindows.of(Time.hours(1), Time.minutes(10)))
            .process(new MemberAbilityCalculator());
    }
}

核心聚合逻辑：

public class AreaHeatAggregator implements AggregateFunction<RidingRecord, AreaHeat, AreaHeat> {
    @Override
    public AreaHeat add(RidingRecord record, AreaHeat accumulator) {
        accumulator.setRidingCount(accumulator.getRidingCount() + 1);
        accumulator.setTotalDistance(accumulator.getTotalDistance() + record.getDistance());
        
        // 实时计算平均速度
        double newAvgSpeed = (accumulator.getAverageSpeed() * (accumulator.getRidingCount() - 1) + 
                            record.getSpeed()) / accumulator.getRidingCount();
        accumulator.setAverageSpeed(newAvgSpeed);
        
        return accumulator;
    }
}

2.2 实时用户画像构建

public class MemberProfileProcessor extends KeyedProcessFunction<String, RidingRecord, MemberProfile> {
    private transient ValueState<MemberProfile> profileState;
    
    @Override
    public void processElement(RidingRecord record, Context ctx, Collector<MemberProfile> out) throws Exception {
        MemberProfile profile = profileState.value();
        if (profile == null) {
            profile = new MemberProfile(record.getMemberId());
        }
        
        // 更新用户骑行偏好
        profile.updateRidingPreference(record.getRouteType(), record.getTimestamp().getHour());
        
        // 计算能力等级
        profile.calculateAbilityLevel(record.getSpeed(), record.getDistance(), record.getHeartRate());
        
        profileState.update(profile);
        out.collect(profile);
    }
}

三、智能推荐系统源码解析

3.1 多因子推荐算法

public class MultiFactorRecommender {
    // 权重配置
    private static final double BEHAVIOR_WEIGHT = 0.40;
    private static final double LOCATION_WEIGHT = 0.25;
    private static final double SOCIAL_WEIGHT = 0.20;
    private static final double HOTNESS_WEIGHT = 0.15;
    
    public List<RouteRecommendation> recommend(String memberId, MemberContext context) {
        // 四维因子计算
        double behaviorScore = calculateBehaviorSimilarity(memberId);
        double locationScore = calculateLocationProximity(context.getCurrentLocation());
        double socialScore = calculateSocialRecommendation(memberId);
        double hotnessScore = calculateRealTimeHotness();
        
        // 加权合并推荐结果
        return weightedMergeRecommendations(
            behaviorScore * BEHAVIOR_WEIGHT,
            locationScore * LOCATION_WEIGHT,
            socialScore * SOCIAL_WEIGHT,
            hotnessScore * HOTNESS_WEIGHT
        );
    }
    
    private List<RouteRecommendation> weightedMergeRecommendations(double... weights) {
        // 使用优先队列按综合评分排序
        PriorityQueue<RouteRecommendation> queue = new PriorityQueue<>(
            (a, b) -> Double.compare(b.getFinalScore(), a.getFinalScore())
        );
        
        // 合并所有推荐结果
        // ... 具体合并逻辑
        
        // 返回Top10推荐
        List<RouteRecommendation> result = new ArrayList<>();
        while (!queue.isEmpty() && result.size() < 10) {
            result.add(queue.poll());
        }
        return result;
    }
}

3.2 实时推荐流水线

public class RealTimeRecommendationPipeline {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        
        // 1. 数据源
        DataStream<MemberEvent> eventStream = env
            .addSource(KafkaSource.createMemberEventSource())
            .name("member-event-source");
            
        // 2. 特征工程
        DataStream<MemberFeature> featureStream = eventStream
            .keyBy(MemberEvent::getMemberId)
            .process(new FeatureEngineeringProcessor());
            
        // 3. 模型推理
        DataStream<Recommendation> recommendationStream = featureStream
            .keyBy(MemberFeature::getMemberId)
            .process(new ModelScoringProcessor());
            
        // 4. 结果存储
        recommendationStream.addSink(new RedisSink());
        
        env.execute("Real-time Recommendation Pipeline");
    }
}

特征工程核心：

public class FeatureEngineeringProcessor extends KeyedProcessFunction<String, MemberEvent, MemberFeature> {
    private transient ValueState<MemberFeature> featureState;
    
    @Override
    public void processElement(MemberEvent event, Context ctx, Collector<MemberFeature> out) throws Exception {
        MemberFeature features = featureState.value();
        if (features == null) {
            features = new MemberFeature(event.getMemberId());
        }
        
        // 实时更新特征向量
        features.updateFromEvent(event);
        
        // 添加时间窗口特征
        features.addTimeWindowFeatures(event.getTimestamp());
        
        featureState.update(features);
        out.collect(features);
    }
}

四、性能优化关键配置

4.1 状态管理优化

// RocksDB状态后端配置
RocksDBStateBackend rocksDBBackend = new RocksDBStateBackend("hdfs://checkpoints/", true);
rocksDBBackend.setIncrementalCheckpoints(true);

// 状态TTL自动清理
StateTtlConfig ttlConfig = StateTtlConfig.newBuilder(Time.days(30))
    .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
    .cleanupInBackground()
    .build();

4.2 精准一次语义保障

// 检查点配置
env.enableCheckpointing(30000); // 30秒间隔
CheckpointConfig checkpointConfig = env.getCheckpointConfig();
checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
checkpointConfig.setMinPauseBetweenCheckpoints(500);