Android Camera原理之setRepeatingRequest與capture模組
Camera操作過程中最重要的四個步驟:
- CameraManager-->openCamera ---> 開啟相機
- CameraDeviceImpl-->createCaptureSession ---> 建立捕獲會話
- CameraCaptureSession-->setRepeatingRequest ---> 設定預覽介面
- CameraDeviceImpl-->capture ---> 開始捕獲圖片
之前我們介紹了openCamera流程和createCaptureSession流程,如下:
《Android Camera原理之openCamera模組(一)》
《Android Camera原理之openCamera模組(二)》
《Android Camera原理之createCaptureSession模組》
至此,Camera 會話已經建立成功,接下來我們可以開始預覽了,預覽回撥onCaptureCompleted之後就可以拍照(回撥到onCaptureCompleted,說明capture 完整frame資料已經返回了,可以捕捉其中的資料了。),由於預覽和拍照的很多流程很相似,拍照只是預覽過程中的一個節點,所以我們把預覽和拍照放在一文裡講解。
1.預覽
預覽發起的函式就是CameraCaptureSession-->setRepeatingRequest
CameraCaptureSession-->setRepeatingRequest
是createCaptureSession(List<Surface> outputs, CameraCaptureSession.StateCallback callback, Handler handler)
中輸出流配置成功之後執行CameraCaptureSession.StateCallback.onConfigured(@NonNull CameraCaptureSession session)
函式中執行的。
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()), new CameraCaptureSession.StateCallback() { @Override public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) { // The camera is already closed if (null == mCameraDevice) { return; } // When the session is ready, we start displaying the preview. mCaptureSession = cameraCaptureSession; try { // Auto focus should be continuous for camera preview. mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE); // Flash is automatically enabled when necessary. setAutoFlash(mPreviewRequestBuilder); // Finally, we start displaying the camera preview. mPreviewRequest = mPreviewRequestBuilder.build(); mCaptureSession.setRepeatingRequest(mPreviewRequest, mCaptureCallback, mBackgroundHandler); } catch (CameraAccessException e) { e.printStackTrace(); } } @Override public void onConfigureFailed( @NonNull CameraCaptureSession cameraCaptureSession) { showToast("Failed"); } }, null );
最終執行了
mCaptureSession.setRepeatingRequest(mPreviewRequest, mCaptureCallback, mBackgroundHandler);
來執行camera preview操作。像對焦等操作就可以在這個onConfigured
回撥中完成。
onConfigured
回調錶示當前的配置流已經完成,相機已經顯示出來了,可以預覽了。onConfigureFailed
配置流失敗,相機黑屏。
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback,
Handler handler) throws CameraAccessException {
checkRepeatingRequest(request);
synchronized (mDeviceImpl.mInterfaceLock) {
checkNotClosed();
handler = checkHandler(handler, callback);
return addPendingSequence(mDeviceImpl.setRepeatingRequest(request,
createCaptureCallbackProxy(handler, callback), mDeviceExecutor));
}
}
- 第一個引數CaptureRequest 標識當前capture 請求的屬性,是請求一個camera還是多個camera,是否複用之前的請求等等。
- 第二個引數CaptureCallback 是捕捉回撥,這是開發者直接接觸的回撥。
public interface CaptureCallback {
public static final int NO_FRAMES_CAPTURED = -1;
public void onCaptureStarted(CameraDevice camera,
CaptureRequest request, long timestamp, long frameNumber);
public void onCapturePartial(CameraDevice camera,
CaptureRequest request, CaptureResult result);
public void onCaptureProgressed(CameraDevice camera,
CaptureRequest request, CaptureResult partialResult);
public void onCaptureCompleted(CameraDevice camera,
CaptureRequest request, TotalCaptureResult result);
public void onCaptureFailed(CameraDevice camera,
CaptureRequest request, CaptureFailure failure);
public void onCaptureSequenceCompleted(CameraDevice camera,
int sequenceId, long frameNumber);
public void onCaptureSequenceAborted(CameraDevice camera,
int sequenceId);
public void onCaptureBufferLost(CameraDevice camera,
CaptureRequest request, Surface target, long frameNumber);
}
這需要開發者自己實現,這些回撥是如何呼叫到上層的,請看《Android Camera原理之CameraDeviceCallbacks回撥模組》,都是通過CameraDeviceCallbacks回撥調上來的。
下面我們從camera 呼叫原理的角度分析一下
mCaptureSession.setRepeatingRequest
--->CameraDeviceImpl.setRepeatingRequest
--->CameraDeviceImpl.submitCaptureRequest
其中CameraDeviceImpl.setRepeatingRequest
中第3個引數傳入的是true。之所以這個強調一點,因為接下來執行CameraDeviceImpl.capture的時候也會執行setRepeatingRequest,這裡第3個引數傳入的就是false。第3個引數boolean repeating
如果為true,表示當前捕獲的是一個過程,camera frame不斷在填充;如果為false,表示當前捕獲的是一個瞬間,就是拍照。
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback,
Executor executor) throws CameraAccessException {
List<CaptureRequest> requestList = new ArrayList<CaptureRequest>();
requestList.add(request);
return submitCaptureRequest(requestList, callback, executor, /*streaming*/true);
}
private int submitCaptureRequest(List<CaptureRequest> requestList, CaptureCallback callback,
Executor executor, boolean repeating) {
//......
}
CameraDeviceImpl.submitCaptureRequest
核心工作就是3步:
- 1.驗證當前CaptureRequest列表中的request是否合理:核心就是驗證與request繫結的Surface是否存在。
- 2.向底層傳送請求資訊。
- 3.將底層返回的請求資訊和傳入的CaptureCallback 繫結,以便後續正確回撥。
而這三步中,第二步卻是核心工作。
1.1 向底層傳送captureRequest請求
SubmitInfo requestInfo;
CaptureRequest[] requestArray = requestList.toArray(new CaptureRequest[requestList.size()]);
// Convert Surface to streamIdx and surfaceIdx
for (CaptureRequest request : requestArray) {
request.convertSurfaceToStreamId(mConfiguredOutputs);
}
requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating);
if (DEBUG) {
Log.v(TAG, "last frame number " + requestInfo.getLastFrameNumber());
}
for (CaptureRequest request : requestArray) {
request.recoverStreamIdToSurface();
}
- 執行
request.convertSurfaceToStreamId(mConfiguredOutputs);
將本地已經快取的surface和stream記錄在記憶體中,並binder傳輸到camera service層中,防止camera service端重複請求。requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating);
這兒直接呼叫到camera service端。這兒需要重點講解一下的。request.recoverStreamIdToSurface();
回撥成功,清除之前在記憶體中的資料。
CameraDeviceClient::submitRequest
--->CameraDeviceClient::submitRequestList
這個函式程式碼很多,前面很多執行都是在複用檢索之前的快取是否可用,我們關注一下核心的執行:預覽的情況下傳入的streaming
是true,執行上面;如果是拍照的話,那就執行下面的else。
err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber));
傳入的submitInfo就是要返回上層的回撥引數,如果是預覽狀態,需要不斷更新當前的的frame資料,所以每次更新最新的frame number。
if (streaming) {
err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to set streaming request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
} else {
Mutex::Autolock idLock(mStreamingRequestIdLock);
mStreamingRequestId = submitInfo->mRequestId;
}
} else {
err = mDevice->captureList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to submit capture request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
}
ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId);
}
Camera3Device::setStreamingRequestList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper(
const List<const PhysicalCameraSettingsList> &requests,
const std::list<const SurfaceMap> &surfaceMaps,
bool repeating,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock il(mInterfaceLock);
Mutex::Autolock l(mLock);
status_t res = checkStatusOkToCaptureLocked();
if (res != OK) {
// error logged by previous call
return res;
}
RequestList requestList;
res = convertMetadataListToRequestListLocked(requests, surfaceMaps,
repeating, /*out*/&requestList);
if (res != OK) {
// error logged by previous call
return res;
}
if (repeating) {
res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
} else {
res = mRequestThread->queueRequestList(requestList, lastFrameNumber);
}
//......
return res;
}
預覽的時候會執行mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
拍照的時候執行mRequestThread->queueRequestList(requestList, lastFrameNumber);
mRequestThread->setRepeatingRequests
status_t Camera3Device::RequestThread::setRepeatingRequests(
const RequestList &requests,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock l(mRequestLock);
if (lastFrameNumber != NULL) {
*lastFrameNumber = mRepeatingLastFrameNumber;
}
mRepeatingRequests.clear();
mRepeatingRequests.insert(mRepeatingRequests.begin(),
requests.begin(), requests.end());
unpauseForNewRequests();
mRepeatingLastFrameNumber = hardware::camera2::ICameraDeviceUser::NO_IN_FLIGHT_REPEATING_FRAMES;
return OK;
}
將當前提交的CaptureRequest請求放入之前的預覽請求佇列中,告知HAL層有新的request請求,HAL層連線請求開始工作,源源不斷地輸出資訊到上層。這兒是跑在Camera3Device中定義的RequestThread執行緒中,可以保證在預覽的時候不斷地捕獲資訊流,camera就不斷處於預覽的狀態了。
1.2 將返回請求資訊和 CaptureCallback 繫結
if (callback != null) {
mCaptureCallbackMap.put(requestInfo.getRequestId(),
new CaptureCallbackHolder(
callback, requestList, executor, repeating, mNextSessionId - 1));
} else {
if (DEBUG) {
Log.d(TAG, "Listen for request " + requestInfo.getRequestId() + " is null");
}
}
/** map request IDs to callback/request data */
private final SparseArray<CaptureCallbackHolder> mCaptureCallbackMap =
new SparseArray<CaptureCallbackHolder>();
1.向底層傳送captureRequest請求--->回撥的requestIinfo表示當前capture request的結果,將requestInfo.getRequestId()
與CaptureCallbackHolder
繫結,因為Camera 2架構支援傳送多次CaptureRequest請求,如果不使用這種繫結機制,後續的回撥會造成嚴重的錯亂,甚至回撥不上來,那麼開發者無法繼續使用了。
我們看看使用這些回撥的地方的程式碼:
《Android Camera原理之CameraDeviceCallbacks回撥模組》已經說明了CameraDeviceCallbacks.aidl才是camera service程序與使用者程序通訊的回撥,到這個回撥裡面,再取出CaptureRequest繫結的CaptureCallback回撥,呼叫到CaptureCallback回撥函式,這樣開發者可以直接使用。
下面看看CameraDeviceCallbacks
的onCaptureStarted
回撥---->
public void onCaptureStarted(final CaptureResultExtras resultExtras, final long timestamp) {
int requestId = resultExtras.getRequestId();
final long frameNumber = resultExtras.getFrameNumber();
if (DEBUG) {
Log.d(TAG, "Capture started for id " + requestId + " frame number " + frameNumber);
}
final CaptureCallbackHolder holder;
synchronized(mInterfaceLock) {
if (mRemoteDevice == null) return; // Camera already closed
// Get the callback for this frame ID, if there is one
holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId);
if (holder == null) {
return;
}
if (isClosed()) return;
// Dispatch capture start notice
final long ident = Binder.clearCallingIdentity();
try {
holder.getExecutor().execute(
new Runnable() {
@Override
public void run() {
if (!CameraDeviceImpl.this.isClosed()) {
final int subsequenceId = resultExtras.getSubsequenceId();
final CaptureRequest request = holder.getRequest(subsequenceId);
if (holder.hasBatchedOutputs()) {
// Send derived onCaptureStarted for requests within the
// batch
final Range<Integer> fpsRange =
request.get(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE);
for (int i = 0; i < holder.getRequestCount(); i++) {
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(i),
timestamp - (subsequenceId - i) *
NANO_PER_SECOND/fpsRange.getUpper(),
frameNumber - (subsequenceId - i));
}
} else {
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(resultExtras.getSubsequenceId()),
timestamp, frameNumber);
}
}
}
});
} finally {
Binder.restoreCallingIdentity(ident);
}
}
}
holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId);
然後直接呼叫
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(i),
timestamp - (subsequenceId - i) *
NANO_PER_SECOND/fpsRange.getUpper(),
frameNumber - (subsequenceId - i));
簡單明瞭,脈絡清楚。
2.拍照
開發者如果想要拍照的話,直接呼叫
mCaptureSession.capture(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
拍照的呼叫流程和預覽很相似,只是在呼叫函式中個傳入的引數不同。
public int capture(CaptureRequest request, CaptureCallback callback, Executor executor)
throws CameraAccessException {
if (DEBUG) {
Log.d(TAG, "calling capture");
}
List<CaptureRequest> requestList = new ArrayList<CaptureRequest>();
requestList.add(request);
return submitCaptureRequest(requestList, callback, executor, /*streaming*/false);
}
拍照的時候也是呼叫submitCaptureRequest
,只不過第3個引數傳入的是false,表示不用迴圈獲取HAL呼叫上來的幀資料,只獲取瞬間的幀資料就可以。
拍照和預覽呼叫的區分在:CameraDeviceClient::submitRequestList
if (streaming) {
//......
} else {
err = mDevice->captureList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to submit capture request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
}
ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId);
}
接下里呼叫到
mDevice->captureList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper(
const List<const PhysicalCameraSettingsList> &requests,
const std::list<const SurfaceMap> &surfaceMaps,
bool repeating,
/*out*/
int64_t *lastFrameNumber) {
//......
RequestList requestList;
//......
if (repeating) {
res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
} else {
res = mRequestThread->queueRequestList(requestList, lastFrameNumber);
}
//......
return res;
}
執行Camera3Device::RequestThread
執行緒中的queueRequestList
。
status_t Camera3Device::RequestThread::queueRequestList(
List<sp<CaptureRequest> > &requests,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock l(mRequestLock);
for (List<sp<CaptureRequest> >::iterator it = requests.begin(); it != requests.end();
++it) {
mRequestQueue.push_back(*it);
}
if (lastFrameNumber != NULL) {
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
ALOGV("%s: requestId %d, mFrameNumber %" PRId32 ", lastFrameNumber %" PRId64 ".",
__FUNCTION__, (*(requests.begin()))->mResultExtras.requestId, mFrameNumber,
*lastFrameNumber);
}
unpauseForNewRequests();
return OK;
}
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
這裡有關鍵的執行程式碼,表示當前取最新的capture frame資料。
拍照的時候在什麼地方捕捉image?
camera1的時候提供了PictureCallback回撥方式來提供實時預覽回撥,可以在這裡獲取image資料回撥。
camera2沒有這個介面,但是提供了ImageReader.OnImageAvailableListener
來實現回撥。
public interface OnImageAvailableListener {
/**
* Callback that is called when a new image is available from ImageReader.
*
* @param reader the ImageReader the callback is associated with.
* @see ImageReader
* @see Image
*/
void onImageAvailable(ImageReader reader);
}
還記得《Android Camera模組解析之拍照》中提到openCamera之前要設定
mImageReader = ImageReader.newInstance(largest.getWidth(), largest.getHeight(),
ImageFormat.JPEG, /*maxImages*/2);
mImageReader.setOnImageAvailableListener(
mOnImageAvailableListener, mBackgroundHandler);
ImageReader中有一個getSurface()
函式,這是ImageReader的拍照輸出流,我們拍照的時候一般有兩個輸出流(outputSurface物件),一個是預覽流,還有一個是拍照流。不記得可以參考《Android Camera原理之createCaptureSession模組》,ImageReader設定的拍照流會設定到camera service端。
public Surface getSurface() {
return mSurface;
}
我們看看在什麼時候回撥這個介面。
ImageReader回撥介面.jpg
看上面的呼叫流程,呼叫到ImageReader.OnImageAvailableListener->onImageAvailable
中,我們獲取ImageReader->acquireNextImage
可以獲取採集的image圖片。其實ImageReader中也可以獲取預覽的流式資料。SurfacePlane 封裝了返回的ByteBuffer資料,可供開發者實時獲取。
private class SurfacePlane extends android.media.Image.Plane {
private SurfacePlane(int rowStride, int pixelStride, ByteBuffer buffer) {
mRowStride = rowStride;
mPixelStride = pixelStride;
mBuffer = buffer;
/**
* Set the byteBuffer order according to host endianness (native
* order), otherwise, the byteBuffer order defaults to
* ByteOrder.BIG_ENDIAN.
*/
mBuffer.order(ByteOrder.nativeOrder());
}
@Override
public ByteBuffer getBuffer() {
throwISEIfImageIsInvalid();
return mBuffer;
}
final private int mPixelStride;
final private int mRowStride;
private ByteBuffer mBuffer;
}
Note:很多開發者在camera1使用Camera.PreviewCallback的
void onPreviewFrame(byte[] data, Camera camera)
可以獲取實時資料,但是在camera2中沒有這個介面了,雖然camera1的介面方法也能用,camera2替代的介面就是ImageReader.OnImageAvailableListener->on