环信Android端实时语音视频通话时在本地实现通话录音录像
- 背景
- 主要思路
- EMCallSurfaceView获取视频流
- WebRtcAudioRecord
- WebRtcAudioTrack
- 音视频编码混合
- 参考资料
背景
最近一个项目需求，需要将环信实时通话的过程录制保存为音频/视频文件。查找环信API之后发现，环信的makeVideoCall方法仅提供一个recordOnServer参数，将音视频录制保存在环信服务器。未提供本地录制音视频的API。
主要思路
查找环信底层调用Android系统API播放音频和显示视频图像的位置，在相应的位置将音频流和视频流拦截下来，并提交给Android系统的MediaCodec和MediaMuxer进行编码和混合。
EMCallSurfaceView获取视频流
用Android Studio打开hyphenatechat_3.6.1.jar包，找到EMCallSurfaceView，该类位于com.hyphenate.media包中。该视图用于环信的视频显示。该类继承自SurfaceViewRenderer。SurfaceViewRenderer类定义了一个方法:
1
2public void onFrame(VideoFrame frame)
当获取到视频图像帧时，该方法会被调用，并传入参数VideoFrame。我们只需要继承EMCallSurfaceView，并重新实现onFrame方法。再调用VideoFrame.getBuffer().toI420()方法可以获取到I420格式的视频帧数据，之后传递给MediaCodec即可完成视频的编码过程
WebRtcAudioRecord
这两个类都位于com.superrtc.audio包中。WebRtcAudioTrack负责实时语音播放，具体是调用AudioTrack类完成播放过程。WebRtcAudioRecord类负责实时语音的采集，调用AudioRecord类完成具体的采集过程。
这两个类在JavaAudioDeviceModule.Builder的createAudioDeviceModule方法中被创建。如果想获取这两个类的实例，可以修改jar包中这两个类的代码，将他们改成类似单例模式，通过getInstance()方法获取这两个类的实例。WebRtcAudioRecord
1
2
3
4
5
6
7
8
9
10
11
12//添加静态成员变量 private static WebRtcAudioRecord mWebRtcAudioRecord; //添加静态方法 public static WebRtcAudioRecord getInstance() { return mWebRtcpAudioRecord; } //构造方法中修改如下 public WebRtcAudioRecord(...) { ... mWebRtcAudioRecord = this }
WebRtcAudioRecord的内部类AudioRecordThread负责采集的具体实时，其中的run方法如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41public void run() { Process.setThreadPriority(-19); Logging.d("WebRtcAudioRecordExternal", "AudioRecordThread" + WebRtcAudioUtils.getThreadInfo()); WebRtcAudioRecord.assertTrue(WebRtcAudioRecord.this.audioRecord.getRecordingState() == 3); long var1 = System.nanoTime(); while(this.keepAlive) { int bytesReadx = false; int bytesRead = WebRtcAudioRecord.this.audioRecord.read(WebRtcAudioRecord.this.byteBuffer, WebRtcAudioRecord.this.byteBuffer.capacity()); if (bytesRead == WebRtcAudioRecord.this.byteBuffer.capacity()) { if (WebRtcAudioRecord.this.listener != null) { WebRtcAudioRecord.this.listener.onAudioRecordReceived(WebRtcAudioRecord.this.byteBuffer, bytesRead); } if (WebRtcAudioRecord.this.microphoneMute) { WebRtcAudioRecord.this.byteBuffer.clear(); WebRtcAudioRecord.this.byteBuffer.put(WebRtcAudioRecord.this.emptyBytes); } if (this.keepAlive) { WebRtcAudioRecord.this.nativeDataIsRecorded(WebRtcAudioRecord.this.nativeAudioRecord, bytesRead); } if (WebRtcAudioRecord.this.audioSamplesReadyCallback != null) { byte[] data = Arrays.copyOfRange(WebRtcAudioRecord.this.byteBuffer.array(), WebRtcAudioRecord.this.byteBuffer.arrayOffset(), WebRtcAudioRecord.this.byteBuffer.capacity() + WebRtcAudioRecord.this.byteBuffer.arrayOffset()); WebRtcAudioRecord.this.audioSamplesReadyCallback.onWebRtcAudioRecordSamplesReady(new AudioSamples(WebRtcAudioRecord.this.audioRecord.getAudioFormat(), WebRtcAudioRecord.this.audioRecord.getChannelCount(), WebRtcAudioRecord.this.audioRecord.getSampleRate(), data)); } } else { String errorMessage = "AudioRecord.read failed: " + bytesRead; Logging.e("WebRtcAudioRecordExternal", errorMessage); if (bytesRead == -3) { this.keepAlive = false; WebRtcAudioRecord.this.reportWebRtcAudioRecordError(errorMessage); } } } try { if (WebRtcAudioRecord.this.audioRecord != null) { WebRtcAudioRecord.this.audioRecord.stop(); } } catch (IllegalStateException var5) { Logging.e("WebRtcAudioRecordExternal", "AudioRecord.stop failed: " + var5.getMessage()); } }
可以看出AudioRecordThread通过循环调用AudioRecord的read方法读取音频数据。在WebRtcAudioRecord类中添加如下代码
1
2
3
4
5
6
7
8public interface IWebRtcAudioRecordListener { void onAudioRecordReceived(ByteBuffer var1, Integer var2); } protected WebRtcAudioRecord.IWebRtcAudioRecordListener listener; public void setListener(WebRtcAudioRecord.IWebRtcAudioRecordListener listener) { this.listener = listener; }
并在AudioRecordThread的run方法中添加如下代码
1
2
3
4
5
6
7if (bytesRead == WebRtcAudioRecord.this.byteBuffer.capacity()) { //新增代码 if (WebRtcAudioRecord.this.listener != null) { WebRtcAudioRecord.this.listener.onAudioRecordReceived(WebRtcAudioRecord.this.byteBuffer, bytesRead); } ...
并且设置WebRtcRecord类为public即可。即可通过调用WebRtcRecord.getInstance().setListener()方法将listener传递进去并监听音频数据的采集过程，获取音频数据
WebRtcAudioTrack
录制实时通话时，需要将通信双方的语音数据都录制下来，因此除了通过WebRtcAudioRecord获取到本地的音频数据外，还需要通过WebRtcAudioTrack获取对方传过来的音频数据。WebRtcAudioTrack在环信中用于音频播放。同样找到AudioTrackThread内部类。run方法如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38public void run() { Process.setThreadPriority(-19); Logging.d("WebRtcAudioTrackExternal", "AudioTrackThread" + WebRtcAudioUtils.getThreadInfo()); WebRtcAudioTrack.assertTrue(WebRtcAudioTrack.this.audioTrack.getPlayState() == 3); for(int sizeInBytes = WebRtcAudioTrack.this.byteBuffer.capacity(); this.keepAlive; WebRtcAudioTrack.this.byteBuffer.rewind()) { WebRtcAudioTrack.nativeGetPlayoutData(WebRtcAudioTrack.this.nativeAudioTrack, sizeInBytes); WebRtcAudioTrack.assertTrue(sizeInBytes <= WebRtcAudioTrack.this.byteBuffer.remaining()); if (WebRtcAudioTrack.this.speakerMute) { WebRtcAudioTrack.this.byteBuffer.clear(); WebRtcAudioTrack.this.byteBuffer.put(WebRtcAudioTrack.this.emptyBytes); WebRtcAudioTrack.this.byteBuffer.position(0); } /*** ***新增代码 ***/ if (WebRtcAudioTrack.this.webRtcAudioTrackListener != null) { WebRtcAudioTrack.this.webRtcAudioTrackListener.onWebRtcAudioTrackReceived(WebRtcAudioTrack.this.byteBuffer, sizeInBytes); } int bytesWritten = this.writeBytes(WebRtcAudioTrack.this.audioTrack, WebRtcAudioTrack.this.byteBuffer, sizeInBytes); if (bytesWritten != sizeInBytes) { Logging.e("WebRtcAudioTrackExternal", "AudioTrack.write played invalid number of bytes: " + bytesWritten); if (bytesWritten < 0) { this.keepAlive = false; WebRtcAudioTrack.this.reportWebRtcAudioTrackError("AudioTrack.write failed: " + bytesWritten); } } } if (WebRtcAudioTrack.this.audioTrack != null) { Logging.d("WebRtcAudioTrackExternal", "Calling AudioTrack.stop..."); try { WebRtcAudioTrack.this.audioTrack.stop(); Logging.d("WebRtcAudioTrackExternal", "AudioTrack.stop is done."); } catch (IllegalStateException var3) { Logging.e("WebRtcAudioTrackExternal", "AudioTrack.stop failed: " + var3.getMessage()); } } }
通过nativeGetPlayoutData方法获取音频数据，然后writeBytes方法将音频数据写入AudioTrack完成播放。同理我们只需要定义listener并在run方法中实现回调，即可获取WebRtcAudioTrack的音频数据。与WebRtcAudioRecord类似。
音视频编码混合
通过上述方法获取到的视频数据是I420p格式的，需要转化为I420sp格式。可以参考libyuv。音频格式是PCM格式。将音频和视频格式传递给MediaCodec模块，编码后再传递给MediaMuxer模块生成mp4文件即可。音视频的编码过程可以查看参考资料
谢谢阅读
参考资料
音视频编码混合
[1]: https://blog.csdn.net/luoyouren/article/details/52135476
修改第三方jar包内的类
[2]: https://www.jianshu.com/p/b54368815d45
最后
以上就是丰富萝莉最近收集整理的关于环信Android端实时语音视频通话时在本地实现通话录音录像背景主要思路参考资料的全部内容，更多相关环信Android端实时语音视频通话时在本地实现通话录音录像背景主要思路参考资料内容请搜索靠谱客的其他文章。
发表评论 取消回复