Android-audio playback/record Monitor

前言:

安卓在Q上支持了多应用同时录音，当两个应用试图捕获音频时，它们都可以接收输入信号，或者其中一个可能会受到静默处理。当多个应用同时捕获音频时，只有一个或两个应用处于“活动”状态(正在接收音频)，其他应用则处于静音状态（接收静音）。当活动应用发生更改时，音频框架可能会根据以下规则重新配置音频路径：

每个活动应用的音频输入设备可能会更改（例如，从内置麦克风更改为已连接的蓝牙耳机）。启用与最高优先级活动应用相关联的预处理。其他预处理都将被忽略。当优先级较高的应用处于活动状态时，活动应用可能会受到静默处理，因此您可以在 AudioRecord 或 MediaRecorder 对象上注册一个 AudioManager.AudioRecordingCallback，以便在配置发生更改时收到通知。

Playback/record Monitor常见使用场景:

• 多app音量调节
• 应用通话录音

google官方说明

2017/Bootcamp 2017 - Core Audio.pdf

Recording notification:

 

Playback notification: 

 

 

APP调用方法

https://developer.android.google.cn/reference/android/media/AudioManager.AudioRecordingCallback.html

1.1 APP监听record的变化

public void onCreate() {
   mHandler = new VoipRecorderHandler();
   mAudioManager = (AudioManager) 
           getApplicationContext().getSystemService(Context.AUDIO_SERVICE);
   mAudioManager.registerAudioRecordingCallback(mAudioRecordingCallback,
           mHandler);
}

public void onDestroy() {
    mAudioManager.unregisterAudioRecordingCallback(mAudioRecordingCallback);
    mHandler.removeCallbacksAndMessages(null);
}

private AudioManager.AudioRecordingCallback mAudioRecordingCallback =
    new AudioManager.AudioRecordingCallback() {
    @Override
    public void onRecordingConfigChanged(List<AudioRecordingConfiguration> configs) {
        ....
        }
    }
};

private class VoipRecorderHandler extends Handler {
        @Override
        public void handleMessage(Message msg) {
            super.handleMessage(msg);
            ......
        }
    }

1.2 APP监听playback的变化

public void onCreate() {
   mHandler = new VoipRecorderHandler();
   mAudioManager = (AudioManager) 
           getApplicationContext().getSystemService(Context.AUDIO_SERVICE);
   registerAudioPlaybackCallback(mAudioPlaybackCallback, 
           new Handler(mLooper));
}

public void onDestroy() {
    mAudioManager.unregisterAudioPlaybackCallback(mAudioPlaybackCallback);
}

private final AudioManager.AudioPlaybackCallback mAudioPlaybackCallback =
         new AudioManager.AudioPlaybackCallback() {
    @Override
    public void onPlaybackConfigChanged(List<AudioRecordingConfiguration> configs) {
        .....
    }
};

Recorder callback具体实现流程

2.1 AudioManager流程

// frameworks/base/media/java/android/media/AudioManager.java
public void registerAudioRecordingCallback(cb, handler) {
    // 此处维护了一个列表mRecordCallbackList
    mRecordCallbackList.add(new AudioRecordingCallbackInfo(cb,
          new ServiceEventHandlerDelegate(handler).getHandler()));
    final IAudioService service = getService();
    // getService的方式: IBinder b = ServiceManager.getService(Context.AUDIO_SERVICE);
    // sService = IAudioService.Stub.asInterface(b); return sService;
    service.registerRecordingCallback(mRecCb);
}

private final IRecordingConfigDispatcher mRecCb = new IRecordingConfigDispatcher.Stub() { // 此处IRecordingConfigDispatcher 是aidl接口，会跨进程通信
    @Override
    public void dispatchRecordingConfigChange(List<AudioRecordingConfiguration> configs) {
        synchronized(mRecordCallbackLock) {
            if (mRecordCallbackList != null) {
                for (int i=0 ; i < mRecordCallbackList.size() ; i++) {
                    final AudioRecordingCallbackInfo arci = mRecordCallbackList.get(i);
                    if (arci.mHandler != null) {
                        final Message m = arci.mHandler.obtainMessage(
                            MSSG_RECORDING_CONFIG_CHANGE/*what*/,
                            new RecordConfigChangeCallbackData(arci.mCb, configs)/*obj*/);  // 这里会发送消息MSSG_RECORDING_CONFIG_CHANGE，handler里面收到这个消息就会处理: onRecordingConfigChanged,从而调用到app里面的onRecordingConfigChanged函数
                            arci.mHandler.sendMessage(m);
                    }
                }
            }
        }
    }
};

public void unregisterAudioRecordingCallback(@NonNull AudioRecordingCallback cb) {
    removeRecordCallback_sync(cb)；// 这里实际就是从mRecordCallbackList删除
}

总结: AudioManager里面的实现其实重点就是mRecordCallbackList，当有app注册callback的时候AudioManager就会把app的callback和handler信息存放到mRecordCallbackList里面。unregister的时候就从mRecordCallbackList里面把对应的信息删除掉。当dispatchRecordingConfigChange的时候就表示有record的变化，此时AudioManager就会for循环从mRecordCallbackList取出各个app的信息，并调用他们的callback把消息传过去。

audiomanager实际就是一个clinet端，实际的实现都是在audio server进程.

2.2 audio server流程

// fw/base/services/core/java/com/android/server/audio/AudioService.java
public void registerRecordingCallback(IRecordingConfigDispatcher rcdb) {
    mRecordMonitor.registerRecordingCallback(rcdb, isPrivileged);
}

这里所有的实现都是在RecordingActivityMonitor.java

// fw/bs/sv/core/java/com/android/server/audio/RecordingActivityMonitor.java
void registerRecordingCallback(IRecordingConfigDispatcher rcdb, boolean isPrivileged) {
    final RecMonitorClient rmc = new RecMonitorClient(rcdb, isPrivileged);
    mClients.add(rmc);
    // 这里其实就是维护了一个mClinets数组，存放client数据
}

unregister其实就是从mClients删除了这个记录

private void dispatchCallbacks(List<AudioRecordingConfiguration> configs) {
    final List<AudioRecordingConfiguration> configsPublic = mHasPublicClients
            ? anonymizeForPublicConsumption(configs) :
            new ArrayList<AudioRecordingConfiguration>();
    for (RecMonitorClient rmc : mClients) {
        rmc.mDispatcherCb.dispatchRecordingConfigChange(configs);
    }
}

总结: audioserver里面的实现其实和audiomanger是一样的，也是通过维护mClients列表实现register和unregister，并在dispatchCallbacks函数里面循环为每个RecMonitorClient调用callback。

问题: dispatchCallbacks这个是在什么时机调用的?

当native层发现record有变化时，会调用recordingCallbackFromNative，这个函数通过jni调用到AudioSystem.java的onRecordingConfigurationChanged，然后调用audio server dispatchCallbacks。

2.3 native 流程

// frameworks/av/media/libaudioclient/AudioSystem.cpp
/*static*/ void AudioSystem::setRecordConfigCallback(record_config_callback cb)
{
    Mutex::Autolock _l(gLock);
    gRecordConfigCallback = cb;
}

void AudioSystem::AudioPolicyServiceClient::onRecordingConfigurationUpdate(
          int event,
          const record_client_info_t *clientInfo,
          const audio_config_base_t *clientConfig,
          std::vector<effect_descriptor_t> clientEffects,
          const audio_config_base_t *deviceConfig,
          std::vector<effect_descriptor_t> effects,
          audio_patch_handle_t patchHandle,
          audio_source_t source) {
    record_config_callback cb = NULL;
    {
        Mutex::Autolock _l(AudioSystem::gLock);
        cb = gRecordConfigCallback;
    }

    if (cb != NULL) {
        cb(event, clientInfo, clientConfig, clientEffects,
           deviceConfig, effects, patchHandle, source);
        // 这里调用了cb函数也就是android_media_AudioSystem_recording_callback函数
    }
}

问题: AudioSystem.cpp的setRecordConfigCallback是什么时候调用的?

AudioService构造函数调用RecordingActivityMonitor.java的initMonitor。

initMonitor调用AudioSystem.setRecordingCallback(this)

setRecordingCallback调用AudioSystem.java setRecordingCallback

setRecordingCallback最终调用的就是AudioSystem.cpp的setRecordConfigCallback。// 此处传入的参数是jni的android_media_AudioSystem_recording_callback函数，也就是这个函数调用的上面的recordingCallbackFromNative。

问题: 是谁调用的AudioSystem::AudioPolicyServiceClient::onRecordingConfigurationUpdate

av/services/audiopolicy/common/managerdefinitions/src/AudioInputDescriptor.cpp
void AudioInputDescriptor::updateClientRecordingConfiguration (){}
//  mClientInterface->onRecordingConfigurationUpdate .....

// frameworks/av/services/audiopolicy/service/AudioPolicyClientImpl.cpp
AudioPolicyService::AudioPolicyClient::onRecordingConfigurationUpdate
// 这个函数里面调用了AudioPolicyService的onRecordingConfigurationUpdate

AudioPolicyService::onRecordingConfigurationUpdate调用commandthread并发送消息RECORDING_CONFIGURATION_UPDATE
// recordingConfigurationUpdateCommand发送消息RECORDING_CONFIGURATION_UPDATE 
然后调用AudioPolicyService::NotificationClient::onRecordingConfigurationUpdate
在这个函数里面调用mAudioPolicyServiceClient->onRecordingConfigurationUpdate

// /frameworks/av/services/audiopolicy/service/AudioPolicyService.cpp
void AudioPolicyService::onRecordingConfigurationUpdate(......) {
    mAudioPolicyServiceClient->recordingConfigurationUpdateCommand(event,
        clientInfo, clientConfig, clientEffects, 
        deviceConfig, effects,patchHandle, source);
}

然后调用的就AudioSystem::AudioPolicyServiceClient::onRecordingConfigurationUpdate

问题: AudioInputDescriptor::updateClientRecordingConfiguration的调用:

void AudioInputDescriptor::updateClientRecordingConfiguration(int event, const sp<RecordClientDescriptor>& client)

这个函数有两个参数，一个是event，一个是client.

event一共有三种类型: RECORD_CONFIG_EVENT_START / RECORD_CONFIG_EVENT_STOP / RECORD_CONFIG_EVENT_UPDATE

上传的数据类型: session_id / source / deviceformat / clientformat / packagename / uid / handle / port_id / silenced / devicesource / effect等

updateClientRecordingConfiguration的调用时机:

AudioInputDescriptor::setPatchHandle(audio_patch_handle_t handle) // 发送start or update

// audio patch表示音频中端到端的连接关系。这里setPatchHandle应该是切换了设备，或者开始连接设备的时候会调用的。

AudioInputDescriptor::setClientActive(const sp<RecordClientDescriptor>& client, bool active) // 发送start or stop

// setClientActive是标记当前Client是否活跃，当startinput startsource等函数中标记为true，stop的时候标记为false

AudioInputDescriptor::trackEffectEnabled(const sp<EffectDescriptor> &effect, bool enabled) // 发送 update

// trackEffectEnabled是切换音效的

AudioInputDescriptor::setAppState(audio_port_handle_t portId, app_state_t state) // 发送 update

// setAppState这个是设置app state，有三种state: APP_STATE_IDLE & APP_STATE_FOREGROUND & APP_STATE_TOP, 第一个表示client是空闲状态，不能录音。第二个表示client有一个前台service，可以录音。第三个表示client有一个可见的ui，可以录音也可以选择use case。

PlaybackCallBack具体实现流程:

 updateState有6种state，后面调用中的event就是这里的state。

typedef enum {
    PLAYER_STATE_UNKNOWN  = -1, 
    PLAYER_STATE_RELEASED = 0,  // 调用release之后state是released
    PLAYER_STATE_IDLE     = 1,  // 创建playerbase的时候state是idle
    PLAYER_STATE_STARTED  = 2,
    PLAYER_STATE_PAUSED   = 3,
    PLAYER_STATE_STOPPED  = 4,
} player_state_t;

// frameworks/base/media/java/android/media/AudioManager.java
收到消息MSSG_PLAYBACK_CONFIG_CHANGE就会调用cbData.mCb.onPlaybackConfigChanged(cbData.mConfigs)

public void dispatchPlaybackConfigChange // 发出消息MSSG_PLAYBACK_CONFIG_CHANGE

// frameworks/base/services/core/java/com/android/server/audio/PlaybackActivityMonitor.java
private void dispatchPlaybackChange(boolean iplayerReleased)
playerAttributes && playerEvent 都会调用dispatchPlaybackChange

// 先看看playerAttributes的调用
// frameworks/base/services/core/java/com/android/server/audio/AudioService.java
public void playerAttributes(int piid, AudioAttributes attr)

// frameworks/base/media/java/android/media/PlayerBase.java
void baseUpdateAudioAttributes(@NonNull AudioAttributes attr)
setAudioStreamType && setAudioAttributes 都会调用baseUpdateAudioAttributes

// /frameworks/base/media/java/android/media/MediaPlayer.java
public void setAudioAttributes(AudioAttributes attributes) // 都是在mediaplayer creat的时候调用的

// 再看看playerEvent的调用
// frameworks/base/services/core/java/com/android/server/audio/AudioService.java
public void playerEvent(int piid, int event) 

// frameworks/base/media/java/android/media/PlayerBase.java
private void updateState(int state) 
baseStart() & basePause() & baseStop() 分别发送start / stop / pause消息

// frameworks/base/media/java/android/media/AudioTrack.java
private void startImpl()
// frameworks/base/media/java/android/media/MediaPlayer.java
private void startImpl() {
// playerbase就是一个接口类，实现了APP通过mediaplayer播放还是通过audiotrack播放，都能把消息传给PlaybackActivityMonitor

多个APP复用通路，怎么收到通知:

output复用逻辑:

// /frameworks/av/services/audiopolicy/managerdefault/AudioPolicyManager.cpp
status_t AudioPolicyManager::checkOutputsForDevice(device, state,outputs) {
    status_t status = dupOutputDesc->openDuplicating(mPrimaryOutput, desc,
                   &duplicatedOutput);
}

//frameworks/av/services/audiopolicy/common/managerdefinitions/src/AudioOutputDescriptor.cpp
status_t SwAudioOutputDescriptor::openDuplicating(output1,output2,*ioHandle) {
    *ioHandle = mClientInterface->openDuplicateOutput(output2->mIoHandle, output1->mIoHandle);
}

// frameworks/av/services/audiopolicy/service/AudioPolicyClientImpl.cpp
audio_io_handle_t AudioPolicyService::AudioPolicyClient::openDuplicateOutput(
       audio_io_handle_t output1, audio_io_handle_t output2) {
    sp<IAudioFlinger> af = AudioSystem::get_audio_flinger();
    return af->openDuplicateOutput(output1, output2);
    // audioflinger中会把这两个output加入同一个复用thread里面(DuplicatingThread)
}

output复用都是在native层做处理，但是playbackcallback的检测都是在java层，所以复用不会影响消息接收。只要有一个audiotrack/mediaplayer播放，playbackmonitor就会发送消息。

input复用逻辑:

// frameworks/av/services/audiopolicy/managerdefault/AudioPolicyManager.cpp
audio_io_handle_t AudioPolicyManager::getInputForDevice(......) {
    sp<IOProfile> profile;
    profile = getInputProfile(device, profileSamplingRate, profileFormat, 
              profileChannelMask,profileFlags);
    if (!profile->canOpenNewIo()|| (curInputCount >= 2)) { // 需要复用
        for (size_t i = 0; i < mInputs.size(); ) {
            sp<AudioInputDescriptor> desc = mInputs.valueAt(i); 
            RecordClientVector clients = desc->clientsList();
            for (const auto& client : clients) {
                if (client->active() && client->appState() != APP_STATE_IDLE) {
                    ALOGD("%s() resue input: %d", __func__, desc->mIoHandle);
                    return desc->mIoHandle; // 复用成功
                }
            }
        }
    }
    // 后面是如果不复用，打开新input的逻辑
}

record的复用是在AudioManager里面判断的，如果复用则两个app的录音会复用同一个AudioInputDescriptor。record callback的消息也是从AudioInputDescriptor发出的，那么call back可以区分是哪个app的消息吗？

void AudioInputDescriptor::setAppState(audio_port_handle_t portId, app_state_t state) {
    RecordClientVector clients = clientsList(false /*activeOnly*/);
    RecordClientVector updatedClients;
    for (const auto& client : clients) {
        if (portId == client->portId()) {
            bool wasSilenced = client->isSilenced();
            client->setAppState(state);
            if (client->active() && wasSilenced != client->isSilenced()) {
                updatedClients.push_back(client);
            }
        }
    }
    checkSuspendEffects();
    for (const auto& client : updatedClients) {
        updateClientRecordingConfiguration(RECORD_CONFIG_EVENT_UPDATE, client);
    }
}

audiopolicyservice里面存放了每个client的信息: mAudioRecordClients。当有变化时，会调用对每个client调用setAppState，此处会带上client的portId，所以AudioInputDescriptor也可以区分具体是哪个client，因此即使多个APP复用了input通路，record callback也可以正常的接收到每个app的消息。