本文是北美模拟面试题 Design Dropbox 的笔记,原视频可以在 System Design Guru 频道查看。
北美的 System Design 面试没有标准答案,全部为开放式问答,只要言之有理能讲清楚各种选择下的 tradeoff 即可。
原题:
请设计一个类似 Dropbox 的文件系统,其支持
1. 用户上传并下载文件
2. 多客户端/用户之间同步文件
考点:
- 数据存储(Data Storage)
- 元数据存储:
- 使用关系数据库(Relational Database, RDBMS)存储元数据,因为元数据结构良好、具有固定模式(schema),并且从ACID事务中受益。这有助于避免不一致性并有效地处理并发性。
- 文件存储:
- 使用Blob存储(Blob Store)来存储文件,因为它在成本上更具效益,并且支持非结构化数据。
- 提及Blob存储而不是具体技术,如S3,以避免具体实现的局限性。
- 元数据存储:
- 数据分片(Data Sharding)
- 按项ID分片(Shard by Item ID):
- 优点:
- 均匀分布: 确保跨分片的负载均衡。
- 容错性: 如果一个分片宕机,只有一些文件不可访问,而不是某个用户的所有文件。
- 缺点:
- 复杂查询: 跨分片查询更复杂,并可能导致更高的网络延迟。
- 优点:
- 按用户ID分片(Shard by User ID):
- 优点:
- 数据本地化: 一个用户上传的所有文件存储在一个分片中,简化了用户特定查询。
- 缺点:
- 潜在热点: 重度用户可能会创建热点,例如某个用户有数百万个文件。
- 容错性: 如果一个分片宕机,受影响的用户将失去对所有文件的访问。
- 数据移动: 在重新平衡期间,将整个用户数据集移动到不同的分片是复杂的。
- 优点:
- 按项ID分片(Shard by Item ID):
- 文件上传
- 异步上传(Async Uploading):
- 使用消息队列处理异步作业,以高效地处理大规模和并发文件上传。
- 数据分块(Data Chunking):
- 原因:
- 处理大文件: 分块将大文件分解成可管理的部分,改善资源管理和可靠性。
- 并行上传: 通过允许多个分块同时上传来加速上传过程,最大化吞吐量。
- 恢复能力: 允许在中断后从最后一个成功上传的分块恢复上传。
- 可扩展性: 将上传负载分布在多个服务器或存储节点上。
- 分块大小:
- 小分块: 最小化重试影响,但增加开销。
- 大分块: 减少开销,但增加大数据重传的风险。
- 典型分块大小: 4MB到8MB是一个常见的平衡点。
- 原因:
- 偏移量(Offset):
- 使用偏移量来确定分块在文件中的位置,从而在所有分块上传完成后准确地重构文件。
- 身份验证(Authentication):
- 更倾向于使用上传服务来实现更好的身份验证、验证和对文件上传的控制。
- 直接连接可以通过使用具有短TTL的预签名URL(pre-signed URLs)来管理。
- 故障恢复(重试机制):
- 使用指数退避(exponential backoff)、抖动(jitter)和重试限制(retry limits)来高效地处理故障。
- 文件校验(File Checksum):
- 原因: 通过验证上传的文件与原始文件匹配来确保数据完整性。
- 实现: 在元数据中存储校验和(checksum)。上传完成后,比较原始校验和与实际校验和。
- 上传更新的推送与拉取(Push vs. Pull for Uploading Updates) (通知系统(Notification System)):
- 推送(Push):
- 优点:
- 实时更新。
- 减少不必要的网络请求。
- 缺点:
- 由于需要保持开放连接,资源消耗较大。
- 优点:
- 拉取(Pull):
- 优点:
- 客户端可以控制更新频率。
- 缺点:
- 更新依赖于轮询间隔,而不是实时的。
- 频繁的轮询会增加网络流量和服务器负载。
- 优点:
- 推送(Push):
- 版本控制(冲突解决):
- 处理冲突,例如两个客户端修改同一文件。
- 解决方案包括“先写胜”(First Write Wins)或“后写胜”(Last Write Wins)策略。
- 异步上传(Async Uploading):
以下是上文的英文版本:
Key Points:
- Two Data Storage
- Metadata Storage:
- Use a Relational Database (RDBMS) for metadata because metadata is well-structured, has a fixed schema, and benefits from ACID transactions. This helps avoid inconsistency and handle concurrency effectively.
- File Storage:
- Use a Blob Store for file storage as it is cost-effective and supports unstructured data.
- Mention Blob Store instead of specific technologies like S3 to avoid limitations of specific implementations.
- Metadata Storage:
- Data Sharding
- Shard by Item ID:
- Pros:
- Even Distribution: Ensures balanced load across shards.
- Fault Tolerance: If a shard is down, only some files are inaccessible rather than all files for a user.
- Cons:
- Complex Queries: Cross-shard queries are more complex and can result in higher network latency.
- Pros:
- Shard by User ID:
- Pros:
- Data Locality: All files uploaded by one user are stored together in one shard, simplifying user-specific queries.
- Cons:
- Potential Hot Spots: Heavy users can create hotspots, e.g., a user with millions of files.
- Fault Tolerance: If a shard is down, affected users lose access to all their files.
- Data Movement: Moving entire user datasets between shards during rebalancing is complex.
- Pros:
- Shard by Item ID:
- File Uploading
- Async Uploading:
- Use message queues for async jobs to handle large-scale and concurrent file uploads efficiently.
- Data Chunking:
- Why:
- Handling Large Files: Chunking breaks large files into manageable pieces, improving resource management and reliability.
- Parallel Uploads: Speeds up uploads by allowing multiple chunks to be uploaded simultaneously, maximizing throughput.
- Resume Capability: Allows uploads to resume from the last successfully uploaded chunk after an interruption.
- Scalability: Distributes upload load across multiple servers or storage nodes.
- Chunk Size:
- Small Chunks: Minimize retry impact but increase overhead.
- Large Chunks: Reduce overhead but increase the risk of large data retransmission.
- Typical Chunk Size: 4MB to 8MB is a common balance.
- Why:
- Offset:
- Use offsets to determine the location of a chunk in the file, enabling accurate file reconstruction after all chunks are uploaded.
- Authentication:
- Prefer an uploading service for better authentication, validation, and control over file uploads.
- Direct connection can be managed using pre-signed URLs with short TTLs.
- Failure Recovery (Retry Mechanism):
- Use exponential backoff, jitter, and retry limits to handle failures efficiently.
- File Checksum:
- Why: Ensure data integrity by verifying that the uploaded file matches the original file.
- Implementation: Store checksums in the metadata. Compare original and actual checksums after upload.
- Push vs. Pull for Uploading Updates (Notification System):
- Push:
- Pros:
- Real-time updates.
- Reduces unnecessary network requests.
- Cons:
- Resource-intensive due to maintaining open connections.
- Pros:
- Pull:
- Pros:
- Clients control update frequency.
- Cons:
- Updates depend on polling intervals, not real-time.
- Frequent polling increases network traffic and server load.
- Pros:
- Push:
- Versioning (Conflict Resolution):
- Handle conflicts, such as two clients modifying the same file.
- Solutions include "First Write Wins" or "Last Write Wins" strategies.
- Async Uploading: