VTS Test Failure Analysis

The following test case failed during VTS (Vendor Test Suite) validation:

Note: Google Test filter = PerInstance/ClearOperationsTest.TooManyOperations/0_default
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from PerInstance/ClearOperationsTest
[ RUN ] PerInstance/ClearOperationsTest.TooManyOperations/0_default
hardware/interfaces/keymaster/4.0/vts/functional/keymaster_hidl_hal_test.cpp:4592: Failure
Expected equality of these values:
  ErrorCode::TOO_MANY_OPERATIONS
  Which is: TOO_MANY_OPERATIONS
  Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &op_handle_)
  Which is: OK
hardware/interfaces/keymaster/4.0/vts/functional/keymaster_hidl_hal_test.cpp:4595: Failure
Expected equality of these values:
  ErrorCode::TOO_MANY_OPERATIONS
  Which is: TOO_MANY_OPERATIONS
  Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &op_handle_)
  Which is: OK
hardware/interfaces/keymaster/4.0/vts/functional/keymaster_hidl_hal_test.cpp:4597: Failure
Expected equality of these values:
  ErrorCode::OK
  Which is: OK
  Abort(op_handles[i])
  Which is: INVALID_OPERATION_HANDLE
[ FAILED ] PerInstance/ClearOperationsTest.TooManyOperations/0_default
[ PASSED ] 0 tests.
[ FAILED ] 1 test.

Test Case Description

The TooManyOperations test verifies that:

1. The TOO_MANY_OPERATIONS error is returned when the maximum number of concurrent operations is exceeded
2. After aborting all operations, new operations can be started successfully

Test implementation:

TEST_P(ClearOperationsTest, TooManyOperations) {
    ASSERT_EQ(ErrorCode::OK, GenerateKey(AuthorizationSetBuilder()
                                             .Authorization(TAG_NO_AUTH_REQUIRED)
                                             .RsaEncryptionKey(2048, 65537)
                                             .Padding(PaddingMode::NONE)));

    auto params = AuthorizationSetBuilder().Padding(PaddingMode::NONE);
    int max_operations = SecLevel() == SecurityLevel::STRONGBOX ? 4 : 16;
    OperationHandle op_handles[max_operations];
    AuthorizationSet out_params;

    // Start max_operations number of operations
    for(int i = 0; i < max_operations; i++) {
        EXPECT_EQ(ErrorCode::OK, Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &(op_handles[i])));
    }

    // Next Begin should return TOO_MANY_OPERATIONS
    EXPECT_EQ(ErrorCode::TOO_MANY_OPERATIONS,
         Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &op_handle_));

    // Retry should also return TOO_MANY_OPERATIONS
    EXPECT_EQ(ErrorCode::TOO_MANY_OPERATIONS,
         Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &op_handle_));

    // Abort all operations
    for(int i = 0; i < max_operations; i++) {
        EXPECT_EQ(ErrorCode::OK, Abort(op_handles[i]));
    }

    // After aborting, new operation should succeed
    EXPECT_EQ(ErrorCode::OK,
         Begin(KeyPurpose::ENCRYPT, key_blob_, params, &out_params, &op_handle_));
    AbortIfNeeded();
}

Root Cause Analysis

The issue was found in the Trusted Application's operation management code. Two problems were identified:

Problem 1: Incorrect Maximum Operation Limit

The operation table iteration uses a constant that may differ from what the test expects:

keymaster_error_t try_start_crypto_operation(
                const keymaster_operation_handle_t op_handle,
                const keymaster_key_blob_t& key_blob,
                const uint32_t min_security_level,
                TEE_OperationHandle *crypto_op,
                const keymaster_purpose_t purpose,
                TEE_OperationHandle *digest_op,
                const bool require_auth,
                const keymaster_padding_t padding,
                const keymaster_block_mode_t mode,
                const uint32_t mac_length,
                const keymaster_digest_t digest,
                const keymaster_blob_t& nonce,
                uint8_t *key_id)
{
    TEE_Time current_time;
    DMSG("MAX_CONCURRENT_OPS = %d", MAX_CONCURRENT_OPS);

    for (uint32_t i = 0; i < MAX_CONCURRENT_OPS; i++) {
        if (active_operations[i].op_handle == UNDEFINED) {
            // Allocate operation slot...
            active_operations[i].op_handle = op_handle;
            // ... additional setup code ...
            return KM_ERROR_OK;
        }
    }

    DMSG("Returning KM_ERROR_TOO_MANY_OPERATIONS");
    return KM_ERROR_TOO_MANY_OPERATIONS;
}

Problem 2: Retry Mechanism Bypasses Error Check

The wrapper function attempts to kill old operations when starting fails, which causes test to fail because it allows operations beyond the limit:

keymaster_error_t start_crypto_operation(
                const keymaster_operation_handle_t op_handle,
                const keymaster_key_blob_t& key_blob,
                uint32_t min_security_level,
                TEE_OperationHandle *crypto_op,
                const keymaster_purpose_t purpose,
                TEE_OperationHandle *digest_op,
                const bool require_auth,
                const keymaster_padding_t padding,
                const keymaster_block_mode_t mode,
                const uint32_t mac_length,
                const keymaster_digest_t digest,
                const keymaster_blob_t& nonce,
                uint8_t *key_id)
{
    keymaster_error_t result = try_start_crypto_operation(op_handle, key_blob, min_security_level,
                               crypto_op, purpose,
                               digest_op, require_auth,
                               padding, mode,
                               mac_length, digest,
                               nonce, key_id);
    DMSG("try_start_crypto_operation result = %d", result);

    // This retry logic is problematic - it attempts to recover from TOO_MANY_OPERATIONS
    if (result != KM_ERROR_OK) {
        result = evict_oldest_operation();
        if (result == KM_ERROR_OK) {
            result = try_start_crypto_operation(op_handle, key_blob, min_security_level,
                             crypto_op, purpose,
                             digest_op, require_auth,
                             padding, mode,
                             mac_length, digest,
                             nonce, key_id);
        }
    }
    return result;
}

The retry mechanism was designed to handle resource pressure scenarios, but it incorrect attempts recovery even when the operation limit has been explicitly reached. This defeats the purpose of the TOO_MANY_OPERATIONS error.

Solution

Modify the retry logic to only atempt recovery for errors other than KM_ERROR_TOO_MANY_OPERATIONS:

keymaster_error_t start_crypto_operation(
                const keymaster_operation_handle_t op_handle,
                const keymaster_key_blob_t& key_blob,
                uint32_t min_security_level,
                TEE_OperationHandle *crypto_op,
                const keymaster_purpose_t purpose,
                TEE_OperationHandle *digest_op,
                const bool require_auth,
                const keymaster_padding_t padding,
                const keymaster_block_mode_t mode,
                const uint32_t mac_length,
                const keymaster_digest_t digest,
                const keymaster_blob_t& nonce,
                uint8_t *key_id)
{
    keymaster_error_t result = try_start_crypto_operation(op_handle, key_blob, min_security_level,
                               crypto_op, purpose,
                               digest_op, require_auth,
                               padding, mode,
                               mac_length, digest,
                               nonce, key_id);
    DMSG("try_start_crypto_operation result = %d", result);

    // Only retry when the error is NOT KM_ERROR_TOO_MANY_OPERATIONS
    if (result != KM_ERROR_OK && result != KM_ERROR_TOO_MANY_OPERATIONS) {
        result = evict_oldest_operation();
        if (result == KM_ERROR_OK) {
            result = try_start_crypto_operation(op_handle, key_blob, min_security_level,
                             crypto_op, purpose,
                             digest_op, require_auth,
                             padding, mode,
                             mac_length, digest,
                             nonce, key_id);
        }
    }
    return result;
}

Verification

After applying the fix, the test passes successfully:

Note: Google Test filter = PerInstance/ClearOperationsTest.TooManyOperations/0_default
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from PerInstance/ClearOperationsTest
[ RUN ] PerInstance/ClearOperationsTest.TooManyOperations/0_default
[ OK ] PerInstance/ClearOperationsTest.TooManyOperations/0_default (23289 ms)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (23291 ms total)
[ PASSED ] 1 test.

Summary

The VTS test failure was caused by an overly aggressive retry mechanism in the Keymaster Trusted Application. The fix ensures that when the maximum number of concurrent operations is reached, the TOO_MANY_OPERATIONS error is properly propagated to the caller rather than attempting to recover by evicting existing operations.