RemoteServiceException: can‘t deliver broadcast 问题分析

一、问题背景

    最近测试跑monkey连续压测,报了一个应用稳定性的问题。因为该问题比较典型,并且需要我们编码上也要注意规避该问题。我在分析过程中一直没找到根因,最后求助于leader,非常感谢不吝指教(一块周五加班分析到11点多)。
    分析崩溃log,核心堆栈如下(已脱敏,出问题的android系统版本是api11,AndroidR):
ps: 本次涉及的应用包名统一用com.my.app代替

11-28 03:57:20.326 12039 12039 E AndroidRuntime: FATAL EXCEPTION: main
11-28 03:57:20.326 12039 12039 E AndroidRuntime: Process: com.my.app, PID: 12039
11-28 03:57:20.326 12039 12039 E AndroidRuntime: android.app.RemoteServiceException: can't deliver broadcast
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2073)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:111)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:250)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:7860)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:591)
11-28 03:57:20.326 12039 12039 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1123)

网上查了一番,发现有一些前人的分析:

  1. viewpager嵌套fragment或其他大量动态广播注册注销出现的崩溃: https://blog.csdn.net/qq_27381325/article/details/82811079
    但上述可能算是避开该问题的方法(避开Binder的IPC调用),能规避问题,并不是根因,于是仍然需要跟进该问题,分析根因。如果业务场景真的是需要全局广播,那是无法换成LocalBroadcastManager的。

    同理,如果是应用内通信的话,完全不用广播也是可以的,使用EventBus等就可以,当然EventBus在业内各位大佬的拓展下也出了支持跨进程的版本。

二、异常抛出位置分析

    一眼看过去像是系统问题,刚开始抛给系统看了,结果系统讲这可能是广播传输的数据量过大,导致framework层报出的异常。

2.1 sendBroadcast方法是否本就可能抛出RemoteServiceException?

    查阅代码,初步看到RemotionServiceException可以先推测跟binder跨进程服务有关。这里错误message的提醒是传递广播不正确,看下sendBroadcast()方法的定义,确实是有可能抛出RemoteServiceException异常。

  • android.app.ContextImpl#sendBroadcast(android.content.Intent)源码定义:
   @Override
    public void sendBroadcast(Intent intent) {
        warnIfCallingFromSystemProcess();
        String resolvedType = intent.resolveTypeIfNeeded(getContentResolver());
        try {
            intent.prepareToLeaveProcess(this);
            ActivityManager.getService().broadcastIntentWithFeature(
                    mMainThread.getApplicationThread(), getAttributionTag(), intent, resolvedType,
                    null, Activity.RESULT_OK, null, null, null, AppOpsManager.OP_NONE, null, false,
                    false, getUserId());
        } catch (RemoteException e) { 
        // 这里有声明抛出RemoteException异常,说明发送广播本身就可能抛出异常。
        // 然鹅并不是RemoteServiceException类型异常。
            throw e.rethrowFromSystemServer();
        }
    }

android.os.RemoteException#rethrowFromSystemServer:看抛出的方法也没有做异常类型的包装。

    @NonNull
    public RuntimeException rethrowFromSystemServer() {
        if (this instanceof DeadObjectException) {
            throw new RuntimeException(new DeadSystemException());
        } else {
            throw new RuntimeException(this);
        }
    }

再看RemoteExcetion和本次的RemoteServiceException有没有继承关系,发现是没有的,看来这里的异常声明和这次崩溃的堆栈可能关系不大,以下是类的继承关系:
在这里插入图片描述

2.2 在google源码平台查询崩溃message确定崩溃抛出位置

    排除了sendBroadcast本身会抛出RemoteServiceException的可能性。下一步还是要确定异常从哪儿报出来的,通过https://cs.android.com/直接搜索“can’t deliver broadcast”,发现能搜出来,google的源码查看网站果然强大,瞬间感觉了有了源码查看利器,强烈推荐使用!
    下面Android源码版本,选择一个当前出问题的Android版本即可。

  • tips: 选择不同Android版本的方式,不过同一个版本也有很多个小的分支。
    选择不同Android版本的方式
  • 搜索结果:
    在这里插入图片描述
    简单遴选下,发现这个错误最终是在BroadcastQueue中报出来的。
  • com.android.server.am.BroadcastQueue#performReceiveLocked
    看catch语句中调用app.scheduleCrash()的地方。
    void performReceiveLocked(ProcessRecord app, IIntentReceiver receiver,
            Intent intent, int resultCode, String data, Bundle extras,
            boolean ordered, boolean sticky, int sendingUser)
            throws RemoteException {
        // Send the intent to the receiver asynchronously using one-way binder calls.
        if (app != null) {
            if (app.thread != null) {
                // If we have an app thread, do the call through that so it is
                // correctly ordered with other one-way calls.
                try {
                	// 下面没有任何代码了,所以catch中异常只能是下面这行抛出的。
                	// 其实IPC调用中RemoteException是很常见的的,
                    app.thread.scheduleRegisteredReceiver(receiver, intent, resultCode,
                            data, extras, ordered, sticky, sendingUser, app.getReportedProcState());
                // TODO: Uncomment this when (b/28322359) is fixed and we aren't getting
                // DeadObjectException when the process isn't actually dead.
                //} catch (DeadObjectException ex) {
                // Failed to call into the process.  It's dying so just let it die and move on.
                //    throw ex;
                } catch (RemoteException ex) {
                	// 下面这句注释也很重要!!英语功底很重要!!
                	// 但是这句真的很难理解意思!!老外你不能表达清楚点吗,真的让人费解呀,有理解的哥们请不吝赐教。
                	// 翻译过来:无法转调这个方法。 它(应该是指app.thread这个binder接口,是由应用实现的)要么快死了,
                	// 要么被楔住了。 平缓地杀死它。
                	// wedged:查了词典楔形的,跟这里有啥关系。把...楔住,最相近的是卡住,姑且理解为卡住、阻塞的意思吧。
                    // Failed to call into the process. It's either dying or wedged. Kill it gently.
                    synchronized (mService) {
                        Slog.w(TAG, "Can't deliver broadcast to " + app.processName
                                + " (pid " + app.pid + "). Crashing it.");
                        // 这里通过scheduleCrash抛出去的message跟本次要追踪的吻合。
                        // 可以确定就是从这里抛出去的异常。
                        app.scheduleCrash("can't deliver broadcast");
                    }
                    // 这里还是抛出了RemoteException,最终异常会往上一层层抛,
                    // 到ActivityManagerService的MainHandler中,最后抛到Loper.loop()中谁处理的?
                    // 不是本题重点,先存疑
                    throw ex;
                }
            } else {
                // Application has died. Receiver doesn't exist.
                // 如果是在调用到performReceiveLocked()时应用进程已经挂掉了的状态,应该是这个log才对。
                // 因为应用进程kill过程也是需要流程的,上面的app.thread.scheduleRegisteredReceiver()
                // 在调用过程中app进程挂掉是有可能的,注释里也有相应的提醒
                throw new RemoteException("app.thread must not be null");
            }
        } else {
        	// app为null还要转调onReceive方法??还不清楚何种情况下会传为null的ProcessRecord,先存疑。
            receiver.performReceive(intent, resultCode, data, extras, ordered,
                    sticky, sendingUser);
        }
    }
  • com.android.server.am.ProcessRecord#scheduleCrash:
    void scheduleCrash(String message) {
        // Checking killedbyAm should keep it from showing the crash dialog if the process
        // was already dead for a good / normal reason.
        if (!killedByAm) {
            if (thread != null) {
                if (pid == Process.myPid()) {
                    Slog.w(TAG, "scheduleCrash: trying to crash system process!");
                    return;
                }
                long ident = Binder.clearCallingIdentity();
                try {
                	// thread即app的IApplicationThread对象,指向ActivityThread.ApplicationThread对象。
                	// 在ActivityThread中有new的过程。
                    thread.scheduleCrash(message);
                } catch (RemoteException e) {
                    // If it's already dead our work is done. If it's wedged just kill it.
                    // We won't get the crash dialog or the error reporting.
                    // 这里catch的是thread的binder调用,如果再抛出异常,直接Process.kill()了
                    // 通知应用崩溃的message发送过程异常了啊,系统的设计选择了直接将app进程kill了
                    // 回传的kill reason是REASON_CRASH = 4:Application process died because of 
                    // an unhandled exception in Java code.
                    kill("scheduleCrash for '" + message + "' failed",
                            ApplicationExitInfo.REASON_CRASH, true);
                } finally {
                    Binder.restoreCallingIdentity(ident);
                }
            }
        }
    }

    可以看到ProcessRecord#scheduleCrash()是通过ApplicationThread抛给了应用层,ApplicationThread接口是SystemServer进程和应用进程通信的AIDL接口,定义在ActivityThread.java文件中。

    注意:在这里的场景下,AMS是client端,应用进程是服务端。IApplicationThread的实现是在应用进程中,AMS运行在system_server进程中,通过ProcessRecord持有了IApplicationThread对象,以此实现应用进程的方法调用(通信)。

  • android.app.ActivityThread.ApplicationThread#scheduleCrash:
    通过该方法将framework层system_server进程的异常传递给了应用进程,并通过Handler的方式跑到了主线程。
public void scheduleCrash(String msg) {
   sendMessage(H.SCHEDULE_CRASH, msg);
}
  • H#handleMessage()
case SCHEDULE_CRASH:
     throw new RemoteServiceException((String)msg.obj);

上面的代码,抛出了RemoteServiceException,然后在Looper.loop()的dispatchMessage()中抛出,形成了上面讲的堆栈。

由此,堆栈就对应上了。

2.3 从整体流程看,在哪一步抛出异常?

    到目前为止,我们明确了异常抛出的位置。但到底是在广播发送流程抛出,还是接收流程抛出仍未明确。

这里直接贴广播的发送流程uml序列图了,感兴趣的可以对照翻代码(基于Android Api30,很早之前画的,比较粗糙见谅)。
ContextImpl.sendBroadcast()过程
    由上图能看出对于应用进程(发生崩溃的进程)是接收流程,也就是说崩溃的进程是在接收广播时,AMS通过iApplicationThread转调应用进程的onReceive()方法的过程中抛出异常了。

三、根因分析

3.1 app.thread.scheduleRegisteredReceiver()为何失败?

    也就是说AMS调用IApplicationThread的一个AIDL方法为何失败?
    这个问题跟随便写一个aidl调用进行IPC调用,但是抛出了RemoteException异常本质一样。
Binder调用失败,刚好log中在发生崩溃前有大量的binder调用阻塞和失败记录。

  • Binder调用频繁失败
11-28 03:56:59.642   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.645   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.648   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.657   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.668   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.676   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.689   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.707   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.719   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.720   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.721   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.738   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.743   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.746   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.767   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.773   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.782   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.788   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.805   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.809   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.812   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.829   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.833   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.844   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.847   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.851   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.854   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.859   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.872   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.876   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.880   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.881   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.882   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.883   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.901   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.904   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.908   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.922   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.924   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.929   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.930   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.949   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.953   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.971   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.974   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.977   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.994   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:56:59.997   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.000   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.007   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.018   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.022   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232) 
11-28 03:57:00.027   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.030   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.033   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.043   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.052   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.055   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
11-28 03:57:00.068   599 20783 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 232)
  • Binder调用阻塞很长时间
    Binder调用阻塞
  • Binder泄漏导致资源耗尽,system_server异常
    在Binder泄漏
    跟进源码发现抛出崩溃的位置:frameworks/base/core/java/android/os/BinderProxy.java#set()
    BinderProxy泄漏的位置

四、小结

  • 异常不确定时可通过https://cs.android.com/直接搜索message确定异常抛出位置;
  • 无论复杂or简单的问题,逐步分析的思路和方向要明确:初步确定有哪些可能抛出该问题?->在源码中搜索异常message确定抛出位置?->分析源码哪些情况下会走异常逻辑?-> A情况?、B情况?C情况?-> 改写代码凑条件复现该问题-> 总结、复盘该题结论。->后面如何避免该问题?
  • 理性分析,尊重客观事实。
  • 5
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 8
    评论
<!DOCTYPE html> <html> <head> <title>iXigua</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <style> body{ margin: 0; padding: 0; } .header{ background-color: #131313; height: 70px; display: flex; align-items: center; justify-content: space-between; padding: 0 40px; } .logo{ font-size: 30px; font-weight: bold; color: white; } .navigation{ display: flex; align-items: center; } .navigation a{ margin-left: 30px; font-size: 18px; color: white; text-decoration: none; transition: all 0.3s ease; } .navigation a:hover{ color: #FF3B3F; } .hero-section{ background-image: url("https://cdn.pixabay.com/photo/2017/02/01/22/02/mountains-2031539_1280.jpg"); height: 600px; background-size: cover; background-position: center center; display: flex; flex-direction: column; justify-content: center; align-items: center; text-align: center; color: white; } .hero-section h1{ font-size: 60px; line-height: 80px; margin-bottom: 30px; } .hero-section p{ font-size: 28px; line-height: 40px; margin-bottom: 50px; } .hero-section button{ background-color: #FF3B3F; color: white; padding: 15px 40px; font-size: 18px; font-weight: bold; border-radius: 50px; transition: all 0.3s ease; cursor: pointer; } .hero-section button:hover{ background-color: white; color: #FF3B3F; border: 2px solid #FF3B3F; } .features{ padding: 100px 0; display: flex; flex-wrap: wrap; align-items: center; justify-content: center; background-color: #F9F9F9; text-align: center; } .feature{ flex-basis: 300px; padding: 50px; margin: 30px; background-color: white; border-radius: 10px; box-shadow: 0 0 10px rgba(0,0,0,0.2); } .feature h2{ font-size: 24px; margin-bottom: 30px; } .feature p{ font-size: 18px; line-height: 30px; } .get-started{ background-color: #FF3B3F; color: white; padding: 100px; display: flex; flex-direction: column; align-items: center; text-align: center; } .get-started h2{ font-size: 48px; margin-bottom: 50px; } .get-started button{ background-color: white; color: #FF3B3F; padding: 15px 40px; font-size: 24px; font-weight: bold; border-radius: 50px; transition: all 0.3s ease; cursor: pointer; margin-top: 30px; border: none; box-shadow: 0 0 10px rgba(0,0,0,0.2); } .get-started button:hover{ background-color: #FF3B3F; color: white; border: none; } .footer{ background-color: #131313; color: white; padding: 50px; display: flex; flex-wrap: wrap; align-items: center; justify-content: center; } .footer p{ flex-basis: 100%; text-align: center; } .footer a{ color: white; text-decoration: none; margin-left: 10px; margin-right: 10px; font-size: 18px; transition: all 0.3s ease; } .footer a:hover{ color: #FF3B3F; } @media only screen and (max-width: 768px){ .header{ height: auto; flex-direction: column; padding: 20px; } .navigation{ margin-top: 20px; } .hero-section h1{ font-size: 42px; line-height: 60px; } .hero-section p{ font-size: 20px; line-height: 30px; } .hero-section button{ font-size: 16px; padding: 10px 30px; } .features{ padding: 50px 0; } .feature{ flex-basis: auto; padding: 30px; margin: 20px; } .get-started{ padding: 50px; } .get-started h2{ font-size: 36px; } .get-started button{ font-size: 20px; padding: 10px 30px; } .footer{ padding: 30px; } } </style> </head> <body> <header class="header"> <h1 class="logo">iXigua</h1> <nav class="navigation"> <a href="#">Home</a> <a href="#">Features</a> <a href="#">Pricing</a> <a href="#">Contact</a> </nav> </header> <section class="hero-section"> <h1>The Ultimate Video Platform</h1> <p>Experience the power of our state-of-the-art platform.</p> <button>Get Started</button> </section> <section class="features"> <div class="feature"> <h2>Fast and Reliable</h2> <p>Our platform is built to deliver videos quickly and reliably, so you can enjoy a seamless viewing experience.</p> </div> <div class="feature"> <h2>HD Quality</h2> <p>With iXigua, you can watch your favorite videos in stunning high-definition, bringing every detail to life.</p> </div> <div class="feature"> <h2>Great Selection</h2> <p>From trending clips to full-length movies, our vast library of content has something for everyone.</p> </div> </section> <section class="get-started"> <h2>Ready to Get Started?</h2> <button>Get Started Now</button> </section> <footer class="footer"> <p>© 2022 iXigua. All rights reserved. | <a href="#">Privacy Policy</a> | <a href="#">Terms of Service</a> | <a href="#">FAQs</a></p> </footer> </body> </html>

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 8
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

TechMix

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值