批量删除版本化数据及关联项的高效实现

您指出的问题很对。当前代码在处理LinkItemVersionedItem等模型时，虽然做到了分批删除，但实际上是一次性查询了所有符合条件的记录ID，然后分批删除。这在大数据量情况下会导致内存问题。我们需要改为真正的分批查询和删除。

以下是优化后的代码，实现了真正的分批查询和删除：

def delete_versioned_items_and_related_items(all_vi_query, valid_vi_query, batch_size=2000):
    # 获取所有有效的VersionedItems的trace_uuid（分批处理）
    valid_trace_uuids = []
    valid_vi_count = valid_vi_query.count()
    for offset in range(0, valid_vi_count, batch_size):
        batch = valid_vi_query[offset:offset + batch_size]
        valid_trace_uuids.extend(item.trace_uuid for item in batch)
    
    # 分批删除Search索引
    search_util = SearchUtil()
    for i in range(0, len(valid_trace_uuids), batch_size):
        batch_uuids = valid_trace_uuids[i:i + batch_size]
        search_util.delete_index(ids=batch_uuids)
        print(f"Deleted search index for batch {i // batch_size + 1} (size: {len(batch_uuids)})")

    # 获取所有VersionedItems的trace_uuid（分批处理）
    all_trace_uuids = []
    all_vi_count = all_vi_query.count()
    for offset in range(0, all_vi_count, batch_size):
        batch = all_vi_query[offset:offset + batch_size]
        all_trace_uuids.extend(item.trace_uuid for item in batch)

    # 获取所有相关的LinkItems（分批处理）
    link_uuids = []
    link_items_query = LinkItem.objects.filter(target_uuid__in=all_trace_uuids)
    link_items_count = link_items_query.count()
    for offset in range(0, link_items_count, batch_size):
        batch = link_items_query[offset:offset + batch_size].values_list('link_uuid', flat=True)
        link_uuids.extend(batch)

    # 定义要删除的模型和查询条件
    models_to_delete = [
        ('LinkItemVersionedItem', lambda: VersionedItem.objects.filter(trace_uuid__in=link_uuids), 
        ('LinkItem', lambda: LinkItem.objects.filter(target_uuid__in=all_trace_uuids)),
        ('ProfileData', lambda: ProfileData.objects.filter(trace_id__in=valid_trace_uuids)),  # MongoDB模型
        ('VersionItemHistory', lambda: VersionItemHistory.objects.filter(trace_uuid__in=all_trace_uuids)),
        ('FileSystemAccessRule', lambda: FileSystemAccessRule.objects.filter(object_trace_uuid__in=all_trace_uuids)),
        ('Reference', lambda: Reference.objects.filter(item_uuid__in=all_trace_uuids)),
        ('FileVersion', lambda: FileVersion.objects.filter(trace_uuid__in=all_trace_uuids)),
        ('FileChecksum', lambda: FileChecksum.objects.filter(trace_uuid__in=all_trace_uuids)),
        ('AllVersionedItem', lambda: all_vi_query)
    ]

    # 批量删除每个模型的数据
    for model_name, query_func in models_to_delete:
        try:
            with transaction.atomic():
                total_deleted = 0
                query = query_func()
                
                if model_name == 'ProfileData':
                    # MongoDB特殊处理
                    while True:
                        batch = query.limit(batch_size)
                        deleted_count = batch.delete()
                        total_deleted += deleted_count
                        print(f"Deleted {deleted_count} {model_name} records in this batch (total: {total_deleted})")
                        if deleted_count < batch_size:
                            break
                else:
                    # Django ORM处理 - 真正的分批查询和删除
                    while True:
                        # 使用values_list和iterator来节省内存
                        batch_ids = list(query.order_by('pk').values_list('pk', flat=True)[:batch_size])
                        if not batch_ids:
                            break
                        deleted_count, _ = query.filter(pk__in=batch_ids).delete()
                        total_deleted += deleted_count
                        print(f"Deleted {deleted_count} {model_name} records in this batch (total: {total_deleted})")
                        # 如果删除数量小于批次大小，说明已经是最后一批了
                        if deleted_count < batch_size:
                            break

                print(f"Finished deleting {model_name}. Total deleted: {total_deleted}")
        except Exception as e:
            print(f"Failed to delete {model_name}: {str(e)}")

主要优化点：

1. 对于Django ORM模型，现在使用values_list和iterator来分批获取主键ID，而不是一次性获取所有ID
2. 添加了提前终止循环的条件：当删除数量小于批次大小时，说明已经是最后一批了，可以提前退出循环
3. 对于MongoDB的ProfileData，直接使用limit和delete组合，而不是先获取所有记录再删除
4. 在查询时添加了order_by('pk')确保分页查询的稳定性

这样修改后，代码将：

• 真正实现分批查询和删除，而不是一次性查询所有记录
• 显著减少内存使用，特别是在处理大量数据时
• 保持事务完整性，每个模型的所有删除操作在一个事务中完成

对于您特别指出的LinkItemVersionedItem问题，现在它会：

1. 每次只查询最多batch_size个记录的主键
2. 只删除这一小批记录
3. 然后继续查询下一批
4. 直到没有更多记录需要删除

这种方式更加高效且内存友好。