Introduction to ByteDance Pitaya

What is ByteDance Pitaya?

1. Pitaya Defined

Over the past decade, with the increased computing power, more efficient hardware, as well as rapid advances in AI algorithms, we are witnessing a paradigm shift in how on-device AI is gradually put into large-scale productions. Major players in the industry like FAANG, Alibaba, Tencent, and Baidu have been investing significantly in applying on-device AI to products and to new domains, and consider it a game-changer in creating impactful experiences for their customers and improving their business intelligence efficiency.

Client AI is an on-device AI research team under the umbrella of ByteDance Infra. Client AI brings together experts in on-device AI framework, kernel and system programming, advanced algorithm, and machine learning to create an on-device AI infrastructure that powers ByteDance.

ByteDance Pitaya is the on-device AI infrastructure introduced by the Client AI team and MLX team collaboratively. It provides complete end-to-end support for the whole cycle of on-device AI applications from development to deployment.

Pitaya’s vision is to develop a state-of-the-art on-device AI infrastructure in the field. We center around advancing on-device AI techniques, driving the democratization of machine learning, and supporting ByteDance’s business intelligence with AI-driven solutions to boost businesses, increase operational efficiency, optimize cost, and improve user experience.

Up to now, Pitaya has provided tailored on-device AI solutions for more than 30 ByteDance products such as Douyin, TikTok, and Toutiao in large-scale production and is currently supporting AI models and strategies to operate at 10-billion scale per day on smartphone devices.

2. Pitaya Architecture

Pitaya consists of two main components: the Pitaya platform and the Pitaya SDK.

• Pitaya platform provides a comprehensive, managed, end-to-end workbench for on-device AI applications, which enables ML engineers to build and manage AI projects, access data from various sources, train, optimize, and deploy AI models that can operate on devices with ease. Pitaya platform also comes with integrated services that enable us to develop, debug and experiment on AI-powered solutions, deploy AI packages into mobile devices, monitor services in real-time, customize alerting, and visualize the performance of your AI tasks and models.

• Pitaya SDK offers an integrated runtime environment for us to seamlessly deploy and run machine learning models on smartphone devices. It also provides a strongly consistent, composable on-device Feature Engineering framework, allowing for easy implementation of data processing, storing, and managing. To ensure user privacy and security, Pitaya SDK comes with a Federated Learning service that allows training of both cloud-based and on-device AI models without the risk of a data breach.

3. Pitaya Platform

3.1 Pitaya Workbench

MLX: ByteDance’s unified end-to-end machine learning platform

Libra: ByteDance’s large-scale online AB experimentation platform

Slardar: ByteDance’s comprehensive monitoring APM platform

Pitaya platform provides a fully-managed, scalable, foundational Pitaya Workbench that covers the entire lifecycle of on-device AI and machine learning, from development to experimentation, to managing and monitoring the AI models. It includes:

• An integrated Web IDE production environment serves a built-in web UI and a web editor in which machine learning engineers and data scientists can develop, manage, debug, release, deploy, experiment, and monitor ML tasks in billion-scale mobile devices. The web IDE houses the workflow system used to orchestrate data, models, intelligence strategies, task triggers, and release policies through an easy configuration of an algorithm package.

• A set of debugger tools for ML engineers to effectively debug their algorithm packages on mobile devices before releasing them into production. The debugger tools allow you to start a debugging session where you can view and filter the system messages in a logcat window and set several types of breakpoints in the algorithm code that pauses the execution of your app at any code location. While paused, you can examine variables, evaluate expressions, then continue execution line by line to determine the causes of runtime errors. The debugger tools also support executing mock SQL queries for inspecting and testing the value and the format of real-time feature data stored in the on-device repository.

• An algorithm package deployment and release system which supports you