优化方案优化思路_优化战的故事

优化方案优化思路

As a Developer Relations Engineer in our EMEA Consulting & Development team, I spend most of my time visiting Unity’s largest customers and helping them resolve performance issues on their projects. If you’re interested in learning how we do that so that you can apply this knowledge and techniques to your own projects, please read on.

作为我们EMEA咨询和开发团队的开发人员关系工程师,我大部分时间都在拜访Unity的最大客户,并帮助他们解决项目中的性能问题。 如果您有兴趣了解我们的操作方法,以便可以将这些知识和技术应用于自己的项目,请继续阅读。

At Unite Copenhagen 2019 I presented a session titled ‘Tales From the Optimization Trenches’. My intention was to help intermediate Unity users who might have already seen advice from our Best practice guides, but lack the practical knowledge required in order to diagnose and resolve other performance issues on their own by using profiling and analysis tools.

在2019年哥本哈根联合会上,我做了题为``优化战线的故事''的会议。 我的意图是帮助中级Unity用户,他们可能已经从我们的 最佳实践指南中 获得了建议 ,但缺乏使用概要分析工具来自行诊断和解决其他性能问题所需的实践知识。

My talk covers:

我的演讲内容包括:

  • An overview of the Developer Relations Engineer role and our core activity: delivering Project Reviews.

    开发人员关系工程师角色以及我们的核心活动概述:提供项目评论。

  • An introduction to optimization and profiling.

    优化和性能分析简介。

  • Three separate sections on CPU, GPU, and memory footprint optimization, each one of them featuring two practical examples based on real issues we’ve seen in the field, along with a breakdown of the tools and techniques we used to overcome them.

    关于CPU,GPU和内存占用空间优化的三个独立部分,每个部分均基于我们在该领域中遇到的实际问题,提供了两个实际示例,以及用于克服这些问题的工具和技术的细分。

  • A series of general optimization rules.

    一系列常规优化规则。

  • Q&A.

    问与答。

You can find the video below and the accompanying slides here.

你可以找到下面的视频和幻灯片随行 这里

There wasn’t enough time to cover everything, and I’ve had some great follow-up discussions with Unite attendees that aren’t captured in the video, so I wrote this blog post to share all of this extra material with you. Still, I highly recommend that you watch the video first.

没有足够的时间来介绍所有内容,并且我已经与Unite与会者进行了一些很好的后续讨论,但视频中没有记录这些讨论,因此我写了这篇博客文章与您分享所有这些额外的材料。 不过,我还是强烈建议您先观看视频。

演示地址

关于项目评论 (About the Project Reviews)

Project Reviews comprise the core of our work. We travel to our customers’ offices and typically spend two full days with them familiarizing ourselves with their projects, asking them several questions in order to understand their requirements and the design decisions they’ve made, and using various profiling tools in order to detect performance bottlenecks. For well-architected projects that have low build times (modular scenes, heavy usage of AssetBundles, etc), we actually perform changes while onsite and reprofile in order to uncover new issues. This is why it’s so important to optimize build times: doing so will enable more frequent iterations. This is even more true for projects whose target hardware differs significantly from the one used during development, such as mobile devices and game consoles.

项目审查是我们工作的核心。 我们前往客户办公室,通常要花整整两天的时间与他们一起熟悉他们的项目,询问他们几个问题以了解他们的要求和他们做出的设计决策,并使用各种性能分析工具来检测性能瓶颈。 对于构建时间短的结构良好的项目(模块化场景,大量使用AssetBundles等),我们实际上是在现场进行更改并重新配置,以发现新问题。 这就是为什么优化构建时间如此重要的原因:这样做可以实现更频繁的迭代。 对于目标硬件与开发过程中使用的目标硬件有显着差异的项目(例如移动设备和游戏机)而言,更是如此。

Luckily for us, Project Reviews are never the same, as our pool of customers is extremely diverse and the type of projects they work on encompasses a wide range of platforms and requirements. When there are complex problems that we don’t manage to resolve during the visit, we capture as much information as we can and we conduct further investigation back at the Unity offices, asking questions to specialized developers across our R&D departments if need be. The deliverables depend on the needs of the customers, but typically they are a written report that summarizes our findings and provides recommendations. When deciding what to focus on, our goal is to always deliver something that provides the greatest value to our customers.

对我们来说幸运的是,项目评论从来都不一样,因为我们的客户群非常多样化,并且他们从事的项目类型涵盖了广泛的平台和要求。 如果在访问过程中遇到无法解决的复杂问题,我们会收集尽可能多的信息,然后我们会在Unity办事处进行进一步调查,并在必要时向我们研发部门的专业开发人员提出问题。 可交付成果取决于客户的需求,但通常它们是一份书面报告,概述了我们的发现并提供了建议。 在决定重点关注什么时,我们的目标是始终交付能够为客户带来最大价值的产品。

Even though we have access to Unity’s source code, when conducting Project Reviews we try to put ourselves in the same position as our customers. That is, we optimize their projects using publicly available profiling tools and best practices. If we actually need to peek under the hood in order to get to the bottom of a performance issue, we do our best to update our documentation afterward in order to make this new knowledge available to all our users and have a greater impact.

即使我们可以访问Unity的源代码,但是在进行项目审查时,我们仍会尝试将自己置于与客户相同的位置。 也就是说,我们使用公开可用的概要分析工具和最佳实践来优化其项目。 如果我们确实需要深入了解才能发现性能问题的根源,那么我们将尽力在以后更新我们的文档,以使所有用户都可以使用此新知识并产生更大的影响。

CPU绑定vs GPU绑定 (CPU bound vs GPU bound)

As discussed during the presentation, before we start optimizing our project we need to find out the actual bottlenecks. One way we can do that is by inspecting the breakdown of our CPU usage using the Unity Profiler. If most of our frame time is spent on ‘Rendering’, as illustrated in the image below, we then need to determine whether we’re CPU bound or GPU bound.

正如演示期间所讨论的,在开始优化项目之前,我们需要找出实际的瓶颈。 我们可以做到这一点的一种方法是使用Unity Profiler检查CPU使用率的细分。 如果我们的大部分帧时间都花费在“渲染”上,如下图所示,那么我们需要确定是CPU绑定还是GPU绑定。

Rendering is a process that is performed in conjunction with both the CPU and the GPU. A comprehensive description of this process is outside the scope of this article but, in a nutshell, the rendering of a scene comprises the following steps:

渲染是与CPU和GPU一起执行的过程。 对此过程的全面描述不在本文讨论的范围之内,但概括地说,场景的渲染包括以下步骤:

  1. For each group of objects that share a material:

    对于共享材料的每组对象:

    1. The CPU sends a series of commands to the GPU in order to set it its internal state (e.g., shader, bound textures, vertex formats, etc). This step is also known as ‘set pass’ call.

      CPU将一系列命令发送给GPU,以设置其内部状态(例如,着色器,绑定纹理,顶点格式等)。 此步骤也称为“设置通过”呼叫。

    2. The CPU sends a batch of geometry to the GPU so that it can be rendered using the state set in 1.a. This step is also referred to as a ‘draw call’ and it’s quite expensive.

      CPU将一批几何图形发送到GPU,以便可以使用1.a中设置的状态对其进行渲染。 此步骤也称为“抽签”,非常昂贵。

    3. If more geometry under the same type of material needs to be rendered, go to step 1.B.

      如果需要在相同类型的材料下渲染更多几何图形,请转到步骤1.B。

    For each group of objects that share a material:

    对于共享材料的每组对象:

Again, there are several details and caveats to the algorithm above, but the key takeaway is that rendering is an activity conducted between the CPU and the GPU. As illustrated in the screenshot below certain tools, such as Xcode, can give us detailed information on how much time is actually spent by both resources.

同样,上面的算法有几个细节和警告,但是关键要点在于渲染是在CPU和GPU之间进行的活动。 如下面的屏幕快照所示,某些工具(例如Xcode)可以为我们提供有关这两种资源实际花费多少时间的详细信息。

This type of information can also be found in the Unity Profiler, though note that the GPU metrics are not always available, as they depend on the support provided by the graphics card and its drivers:

此类信息也可以在Unity Profiler中找到,但是请注意,GPU指标并非始终可用,因为它们取决于图形卡及其驱动程序提供的支持:

If we cannot get CPU and GPU timings using our profiling tools, we can always inspect a random frame in the Unity Profiler. If there’s a call to Gfx.WaitForPresent in there and says ‘call is taking a considerable amount of time’, it means that the CPU is waiting for the GPU to finish processing all the rendering commands and, thus, we’re GPU bound (please refer to this manual page in order to understand the meaning behind other markers, such as WaitForTargetFPS and Gfx.PresentFrame):

如果我们无法使用性能分析工具获得CPU和GPU的时序,则我们始终可以在Unity Profiler中检查随机帧。 如果在其中调用了 Gfx.WaitForPresent 并说“调用花费了大量时间”,则意味着CPU正在等待GPU完成所有渲染命令的处理,因此,我们受GPU约束(请参阅 此手册页 ,以了解其他标记(例如 WaitForTargetFPSGfx.PresentFrame ) 背后的含义 :

There are many factors that could have an impact on the GPU workload, such as:

有许多因素可能会影响GPU工作负载,例如:

  • Fill rate: our application is coloring an excessive number of pixels multiple times on a given frame, a process known as ‘overdraw’.

    填充率:我们的应用程序在给定的帧上多次对过多的像素进行着色,这一过程称为“过度绘制”。

  • Memory bandwidth: our application is sending a large amount of texture data to the GPU. This can be alleviated by reducing the number of textures (via atlasing, for example), reducing their size, and setting them to a compressed format when applicable.

    内存带宽:我们的应用程序正在向GPU发送大量纹理数据。 可以通过减少纹理的数量(例如通过地图集),减小纹理的大小以及在适用时将它们设置为压缩格式来缓解这种情况。

  • Vertex processing: our application is sending too much geometry to the GPU. We covered this scenario as part of one of our examples during the presentation at Unite.

    顶点处理:我们的应用程序向GPU发送了太多的几何图形。 在Unite的演讲中, 我们将这种情况 作为示例之一进行了介绍

Alternatively, if we’re CPU bound, there could be many things contributing to CPU time (e.g. physics, gameplay code, etc.), and we should check the profiler. If the profiler says we’re spending a lot of time in Rendering, it probably means that the CPU is busy sending too many commands to the GPU. This can be optimized by reducing both the number of state changes (or ‘SetPass’ calls) and the number of batches. Please refer to our ‘Fixing Performance Problems’ tutorial for a deeper discussion on this subject.

另外,如果我们受CPU限制,那么可能有很多因素会影响CPU时间(例如,物理,游戏代码等),我们应该检查探查器。 如果探查器说我们在渲染上花费了大量时间,则可能意味着CPU忙于向GPU发送太多命令。 可以通过减少状态更改(或“ SetPass”调用)的数量和批处理的数量来进行优化。 请参阅我们的 “修复性能问题” 教程,以对此主题进行更深入的讨论。

案例研究:加载数据时CPU峰值 (Case study: CPU spikes when loading data)

A performance problem we typically see in customer projects are performance hiccups during the startup phase of their application or when they are transitioning to a new level. These hiccups manifest themselves as spikes in the Unity Profiler:

我们通常在客户项目中看到的性能问题是在应用程序启动阶段或过渡到新级别时出现的性能问题。 这些打h表现为Unity Profiler中的峰值:

And they are typically caused due to both expensive computational processing and large memory allocations. In this example, the CPU spike causes a stall of nearly 10 seconds and a managed allocation of 3.8 GB, as seen in the screenshot below:

它们通常是由于昂贵的计算处理和大量的内存分配而引起的。 在此示例中,CPU峰值导致将近10秒钟的停顿和3.8 GB的托管分配,如以下屏幕截图所示:

These spikes are undesirable mainly for two reasons. The first reason is that their excessive length interrupts the flow of the application. One way to ‘mask’ the stall caused by the CPU spike is to use a loading screen, though note that this solution won’t work if we need to show animated elements on screen, as the animations will stall during the loading process. The second reason that makes these spikes undesirable is that their large allocations permanently increase the size of the managed heap. Unity’s automatic memory management system works in such a way that unreferenced memory is reused in subsequent allocations, but the overall size of the managed heap never decreases, it can only go up. This is known as ‘non-compacting garbage collection’. Please refer to this entry in our documentation and this article from Unity’s Learn website.

这些尖峰是不希望的,主要有两个原因。 第一个原因是它们的长度过长会中断应用程序的流程。 一种“掩盖” CPU高峰导致的停顿的方法是使用加载屏幕,但是请注意,如果我们需要在屏幕上显示动画元素,则此解决方案将不起作用,因为动画会在加载过程中停滞。 导致这些峰值不理想的第二个原因是,它们的大量分配永久增加了托管堆的大小。 Unity的自动内存管理系统的工作方式是,未引用的内存可以在后续分配中重用,但是托管堆的总大小永远不会减少,只能增加。 这就是所谓的“非压缩垃圾收集”。 请参考 我们文档中的条目 以及 Unity学习网站上的本文

The spikes are normally caused by a combination of factors. Based on what we see in the field, it’s because the application is storing data in a non-optimized format (e.g., JSON or XML) and the parsers need to allocate a significant amount of memory in order to process their content. Those allocations, coupled with the intensive computations required to operate on said data (and their associated memory allocations) are often the main culprits.

峰值通常是由多种因素引起的。 根据我们在现场看到的结果,这是因为应用程序以非优化格式(例如JSON或XML)存储数据,并且解析器需要分配大量内存才能处理其内容。 这些分配,加上对所述数据进行操作所需的大量计算(及其关联的内存分配),通常是主要的罪魁祸首。

In order to alleviate these problems, we usually recommend customers to implement a ‘budgeted time manager’ system which instantiates and initializes objects within a per-frame limit and adding support for a binary format. The ‘budgeted time manager’ spreads the cost across multiple frames, whereas support for a binary format helps minimize the size of the allocations.

为了减轻这些问题,我们通常建议客户实施“预算时间管理器”系统,该系统在每帧限制内实例化和初始化对象,并增加对二进制格式的支持。 “预算时间管理器”将成本分布在多个框架上,而对二进制格式的支持有助于最小化分配的大小。

This idea of having a ‘budgeted time manager’ instead of loading all the data in a single method is analogous to the difference between the regular garbage collector and the incremental garbage collector: while the first one stalls the frame until the whole list of managed objects has been processed, the second one spreads the work across multiple frames.

拥有“预算时间管理器”而不是用一种方法加载所有数据的想法类似于常规垃圾收集器和增量垃圾收集器之间的区别:而第一个垃圾收集器会暂停帧直到整个托管对象列表已经处理完毕,第二个将工作分散到多个框架中。

Due to their nature, binary formats are usually harder to work with during development. So our recommendation to customers is not to remove support for text formats entirely. Instead, we advise them to support both and use the text or binary formats depending on whether they are executing the development or release versions of their applications, respectively.

由于其性质,二进制格式通常在开发过程中较难使用。 因此,我们建议客户不要完全删除对文本格式的支持。 相反,我们建议他们同时支持这两种格式,并分别使用文本格式或二进制格式,具体取决于它们是执行应用程序的开发版本还是发行版本。

关于垃圾收集的一些评论 (Some comments regarding Garbage Collection)

In the ‘GC spikes in a fast-paced game’ example, we advised the customer to enable the Incremental Garbage Collector and reduce the frame time as much as possible in order to give the algorithm enough room to operate at the end of every frame. One point that wasn’t stressed enough during the presentation is that the incremental garbage collector is not an excuse to become lax when it comes to minimizing the amount and size of managed memory allocations: the main benefit of the tool compared to the regular garbage collector is that it spreads its workload across multiple frames instead of stalling the frame until the entire pool of managed objects is processed, which is especially important in order to ensure a steady framerate.

“快节奏游戏中的GC峰值” 示例中,我们建议客户启用 增量垃圾收集器 并尽可能减少帧时间,以便为算法提供足够的空间在每帧结束时进行操作。 在演示过程中,没有足够强调的一点是,增量垃圾收集器并不是最小化托管内存分配的数量和大小的松懈借口:与常规垃圾收集器相比,该工具的主要优点这是因为它将工作负载分散在多个帧上,而不是停顿帧直到处理完所有托管对象池,这对于确保稳定的帧率尤为重要。

Note that the Garbage Collector can actually be disabled via scripting by setting the value of the GarbageCollector.GCMode static field to GarbageCollector.Mode.Disabled:

请注意,垃圾收集器实际上可以通过脚本通过设置的值禁用 GarbageCollector.GCMode 静电场 GarbageCollector.Mode.Disabled

1

GarbageCollector.GCMode = GarbageCollector.Mode.Disabled;

1

GarbageCollector . GCMode = GarbageCollector . Mode . Disabled ;

This technique can be useful in scenarios where we don’t want to pay any processing costs associated with the garbage collection algorithm. Though please note that, in order to do that, we need to ensure that no allocations are taking place when the garbage collector is disabled because, as discussed during the presentation, the operating system will happily pull the plug on our application if our memory usage goes above a certain threshold. This is especially true on mobile platforms such as Android and iOS.

在我们不想支付与垃圾回收算法相关的任何处理成本的情况下,此技术很有用。 尽管请注意,为了做到这一点,我们需要确保在禁用垃圾收集器时不进行任何分配,因为如演示期间所讨论的,如果内存使用率过高,操作系统将很乐意将其插入应用程序超过某个阈值。 在Android和iOS等移动平台上尤其如此。

案例研究:具有权威服务器的FPS (Case study: FPS with authoritative server)

A few months ago we conducted a Project Review of a multiplayer first-person shooter game featuring an authoritative server architecture, with the server running in headless mode. We performed a memory capture using the Unity Memory Profiler and discovered that there were hundreds of MBs allocated to meshes, light probes, audio clips, mesh renderers, and various other types of objects that were not actually required in a headless server.

几个月前,我们对具有权威服务器架构且服务器以无头模式运行的多人第一人称射击游戏进行了项目审查。 我们使用 Unity Memory Profiler 进行了内存捕获 ,发现有数百MB分配给无头服务器实际上不需要的网格,光探针,音频剪辑,网格渲染器和各种其他类型的对象。

While this extra memory footprint didn’t prevent the server from running a single multiplayer session, it was clearly impacting its ability to scale. More specifically, being able to increase the number of active instances in a given server required a significant increase in memory.

尽管这种额外的内存占用并不会阻止服务器运行单个多人游戏会话,但显然会影响其扩展能力。 更具体地说,要增加给定服务器中活动实例的数量,就需要显着增加内存。

In this scenario, we advised the customer to break out every game level scene in two parts and store them in separate AssetBundles. The first entity is the ‘logical scene’, and contains all the information required by the headless server, whereas the second entity is the ‘visual scene’ and contains all the information that is exclusively used by the clients.

在这种情况下,我们建议客户将每个游戏关卡场景分为两个部分,并将它们存储在单独的AssetBundles中。 第一个实体是“逻辑场景”,并且包含无头服务器所需的所有信息,而第二个实体是“可视场景”,并且包含客户端专有的所有信息。

Note that this division can cause some workflow problems. More specifically, artists and level designers can no longer work in a single scene. Instead of introducing disruptions in the content creators’ workflow, we recommended our customers to leave them as they are and add support for breaking down scenes into the ‘logical’ and ‘visual’ as part of the build process.

请注意,这种划分可能会导致一些工作流程问题。 更具体地说,艺术家和关卡设计师不能再在单个场景中工作。 建议不要在内容创建者的工作流程中引入干扰,我们建议客户将其保持原样,并在构建过程中增加对将场景分解为“逻辑”和“视觉”的支持。

深度剖析和探查器标记 (Deep profiling and profiler markers)

As we’ve discussed before, we should aim for nearly zero per-frame allocations in our applications’ core loops. Doing so will significantly reduce the overhead caused by the garbage collection algorithm. The Unity Profiler is the best tool for the job, but the default level of depth in the reported call stack will only go as deep as the first call stack depth of invocations from the engine’s native code into the application’s scripting code (e.g., MonoBehaviour.Start(), MonoBehaviour.Update() and similar methods). In practice, this means that if our scripts are invoking methods from other scripts (which they usually do), we won’t be able to easily identify the exact place where the managed allocations are taking place.

正如我们之前讨论的那样,我们应该争取在应用程序的核心循环中将几乎每帧分配为零。 这样做将大大减少由垃圾收集算法引起的开销。 Unity Profiler是完成这项工作的最佳工具,但是所报告的调用堆栈中的默认深度级别只会达到从引擎本机代码到应用程序脚本代码(例如 MonoBehaviour) 中调用的第一个调用堆栈深度 。 Start()MonoBehaviour.Update() 和类似方法)。 实际上,这意味着,如果我们的脚本从其他脚本调用方法(通常这样做),则我们将无法轻松识别进行托管分配的确切位置。

One way to work around this problem is by explicitly adding Profiler Markers to our scripts. Doing so will record extra information during the profiling process and will help us narrow down the source of our allocations.

解决此问题的一种方法是将 Profiler Markers 明确添加 到我们的脚本中。 这样做将在性能分析过程中记录更多信息,并有助于我们缩小分配来源。

A second way is to enable Deep Profiling. You can find specific instructions on how to do that in this article from Unity’s Learn website. Bear in mind that deep profiling adds a lot of overhead which significantly slows down the application, and so the reported timings will no longer be accurate. Our recommendation is to conduct a profiling session with deep profiling disabled, take notes on which scenarios cause unwanted managed allocations and if the reported call stacks are not detailed enough to track down the source of the allocations, conduct a second session with deep profiling enabled in order to find the source of the allocations.

第二种方法是启用 深度剖析 。 您可以在 Unity的Learn网站 上找到有关如何执行 操作的特定说明 。 请记住,深度剖析会增加很多开销,这会大大降低应用程序的速度,因此报告的时间将不再准确。 我们的建议是在禁用深度分析的情况下进行性能分析会话,并记下哪些情况会导致不必要的托管分配,并且如果所报告的调用堆栈不够详细,无法追踪到分配源,则在启用了深度性能分析的情况下进行第二次会话为了找到分配的来源。

Please note that before Unity 2019.3, deep profiling was only available when using the Mono scripting backend. This limitation has been lifted in the beta cycle of Unity 2019.3, which provides support for both Mono and IL2CPP backends. From the release notes:

请注意,在Unity 2019.3之前,深度分析仅在使用Mono脚本后端时可用。 在Unity 2019.3的beta周期中已解除了此限制,该周期提供对Mono和IL2CPP后端的支持。 从 发行说明中

Profiler: Added Deep Profiler support to Mono and IL2CPP players.Profiler: Added Deep Profiling support build option to players. When you build a Player with Deep Profiling, C# code instrumentation can be dynamically enabled and disabled.Profiler: Added managed allocation callstacks support in players. When you enable callstacks collection, GC.Alloc samples contain C# code callstack.

Profiler:为Mono和IL2CPP播放器添加了Deep Profiler支持。 Profiler:向播放器添加了深度剖析支持构建选项。 使用深度剖析构建Player时,可以动态启用和禁用C#代码检测。 探查器:在播放器中添加了托管分配调用堆栈支持。 启用调用堆栈收集时,GC.Alloc示例包含C#代码调用堆栈。

The fact that deep profiling is now available for the IL2CPP backend means that developers will now be able to perform deep profile captures on platforms that only support IL2CPP, such as iOS. On top of that, the added support for managed allocation call stacks in players should help developers find the source of their allocations without having to resort to deep profiling.

现在可以对IL2CPP后端进行深度分析,这意味着开发人员现在将能够在仅支持IL2CPP的平台(例如iOS)上执行深度配置文件捕获。 最重要的是,播放器中对托管分配调用堆栈的额外支持应有助于开发人员找到其分配来源,而不必诉诸深度分析。

下一步 (Next steps)

Performance optimization is a large topic that requires a wide range of skills. Skills such as understanding how the underlying hardware operates, along with its limitations. Understanding the various classes and components provided by Unity, algorithms and data structures, how to use profiling tools, and you also need to have creativity in order to find efficient solutions that also satisfy the design requirements.

性能优化是一个大主题,需要广泛的技能。 诸如了解底层硬件的工作方式及其局限性之类的技能。 了解Unity提供的各种类和组件,算法和数据结构,如何使用概要分析工具,并且还需要具有创造力才能找到也满足设计要求的有效解决方案。

We want to help you make your Unity’s applications be as performant as they can be, so if there’s any optimization topic that you’d like more information on, please let us know via the comments section.

我们希望帮助您使Unity的应用程序尽可能地保持高性能,因此,如果您需要任何优化主题,请通过评论部分告诉我们。

翻译自: https://blogs.unity3d.com/2019/11/14/tales-from-the-optimization-trenches/

优化方案优化思路

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值