Skip to content

Flaky test - Raven.Client.Exceptions.RavenException : System.OperationCanceledException: The operation was canceled. #5748

Description

@soujay

Flaky test error in Linux-Default: Raven.Client.Exceptions.RavenException : System.OperationCanceledException: The operation was canceled.

https://github.com/Particular/ServiceControl/actions/runs/31186605177/job/92892618564?pr=5684

The test passed when the failed job was re-run

Logs:


Running ServiceControl.Audit.Persistence.Tests.RavenDB.csproj (net10.0)
    Test run for /home/runner/work/ServiceControl/ServiceControl/src/ServiceControl.Audit.Persistence.Tests.RavenDB/bin/Release/net10.0/ServiceControl.Audit.Persistence.Tests.RavenDB.dll (.NETCoreApp,Version=v10.0)
    A total of 1 test files matched the specified pattern.
    
    Error: Raven.Client.Exceptions.RavenException : System.OperationCanceledException: The operation was canceled.
       at System.Threading.CancellationToken.ThrowOperationCanceledException()
       at System.Threading.CancellationToken.ThrowIfCancellationRequested()
       at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in /_/62092/src/Voron/StorageEnvironment.cs:line 662
       at Voron.StorageEnvironment.ReadTransaction(TransactionPersistentContext transactionPersistentContext, ByteStringContext context) in /_/62092/src/Voron/StorageEnvironment.cs:line 597
       at Raven.Server.ServerWide.Context.TransactionOperationContext.CreateReadTransaction() in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 33
       at Raven.Server.ServerWide.Context.TransactionOperationContext`1.OpenReadTransaction(String caller) in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 74
       at Raven.Server.Documents.Indexes.Index.GetStats(Boolean calculateLag, Boolean calculateStaleness, Boolean calculateMemoryStats, Boolean calculateLastBatchStats, QueryOperationContext queryContext) in /_/62092/src/Raven.Server/Documents/Indexes/Index.cs:line 3168
       at Raven.Server.Documents.Handlers.Processors.Indexes.IndexHandlerProcessorForGetDatabaseIndexStatistics.HandleCurrentNodeAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/Indexes/IndexHandlerProcessorForGetDatabaseIndexStatistics.cs:line 126
       at Raven.Server.Documents.Handlers.Processors.AbstractHandlerProxyReadProcessor`3.ExecuteAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/AbstractHandlerProxyReadProcessor.cs:line 30
       at Raven.Server.Documents.Handlers.IndexHandler.Stats() in /_/62092/src/Raven.Server/Documents/Handlers/IndexHandler.cs:line 118
       at Raven.Server.Routing.RequestRouter.HandlePath(RequestHandlerContext reqCtx) in /_/62092/src/Raven.Server/Routing/RequestRouter.cs:line 417
       at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in /_/62092/src/Raven.Server/RavenServerStartup.cs:line 183
      ReachedLeader: False
      ----> System.OperationCanceledException : System.OperationCanceledException: The operation was canceled.
       at System.Threading.CancellationToken.ThrowOperationCanceledException()
       at System.Threading.CancellationToken.ThrowIfCancellationRequested()
       at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in /_/62092/src/Voron/StorageEnvironment.cs:line 662
       at Voron.StorageEnvironment.ReadTransaction(TransactionPersistentContext transactionPersistentContext, ByteStringContext context) in /_/62092/src/Voron/StorageEnvironment.cs:line 597
       at Raven.Server.ServerWide.Context.TransactionOperationContext.CreateReadTransaction() in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 33
       at Raven.Server.ServerWide.Context.TransactionOperationContext`1.OpenReadTransaction(String caller) in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 74
       at Raven.Server.Documents.Indexes.Index.GetStats(Boolean calculateLag, Boolean calculateStaleness, Boolean calculateMemoryStats, Boolean calculateLastBatchStats, QueryOperationContext queryContext) in /_/62092/src/Raven.Server/Documents/Indexes/Index.cs:line 3168
       at Raven.Server.Documents.Handlers.Processors.Indexes.IndexHandlerProcessorForGetDatabaseIndexStatistics.HandleCurrentNodeAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/Indexes/IndexHandlerProcessorForGetDatabaseIndexStatistics.cs:line 126
       at Raven.Server.Documents.Handlers.Processors.AbstractHandlerProxyReadProcessor`3.ExecuteAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/AbstractHandlerProxyReadProcessor.cs:line 30
       at Raven.Server.Documents.Handlers.IndexHandler.Stats() in /_/62092/src/Raven.Server/Documents/Handlers/In
    
      Failed Indexes_should_be_reset_on_setup [956 ms]
      Error Message:
       Raven.Client.Exceptions.RavenException : System.OperationCanceledException: The operation was canceled.
       at System.Threading.CancellationToken.ThrowOperationCanceledException()
       at System.Threading.CancellationToken.ThrowIfCancellationRequested()
       at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in /_/62092/src/Voron/StorageEnvironment.cs:line 662
       at Voron.StorageEnvironment.ReadTransaction(TransactionPersistentContext transactionPersistentContext, ByteStringContext context) in /_/62092/src/Voron/StorageEnvironment.cs:line 597
       at Raven.Server.ServerWide.Context.TransactionOperationContext.CreateReadTransaction() in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 33
       at Raven.Server.ServerWide.Context.TransactionOperationContext`1.OpenReadTransaction(String caller) in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 74
       at Raven.Server.Documents.Indexes.Index.GetStats(Boolean calculateLag, Boolean calculateStaleness, Boolean calculateMemoryStats, Boolean calculateLastBatchStats, QueryOperationContext queryContext) in /_/62092/src/Raven.Server/Documents/Indexes/Index.cs:line 3168
       at Raven.Server.Documents.Handlers.Processors.Indexes.IndexHandlerProcessorForGetDatabaseIndexStatistics.HandleCurrentNodeAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/Indexes/IndexHandlerProcessorForGetDatabaseIndexStatistics.cs:line 126
       at Raven.Server.Documents.Handlers.Processors.AbstractHandlerProxyReadProcessor`3.ExecuteAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/AbstractHandlerProxyReadProcessor.cs:line 30
       at Raven.Server.Documents.Handlers.IndexHandler.Stats() in /_/62092/src/Raven.Server/Documents/Handlers/IndexHandler.cs:line 118
       at Raven.Server.Routing.RequestRouter.HandlePath(RequestHandlerContext reqCtx) in /_/62092/src/Raven.Server/Routing/RequestRouter.cs:line 417
       at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in /_/62092/src/Raven.Server/RavenServerStartup.cs:line 183
      ReachedLeader: False
      ----> System.OperationCanceledException : System.OperationCanceledException: The operation was canceled.
       at System.Threading.CancellationToken.ThrowOperationCanceledException()
       at System.Threading.CancellationToken.ThrowIfCancellationRequested()
       at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in /_/62092/src/Voron/StorageEnvironment.cs:line 662
       at Voron.StorageEnvironment.ReadTransaction(TransactionPersistentContext transactionPersistentContext, ByteStringContext context) in /_/62092/src/Voron/StorageEnvironment.cs:line 597
       at Raven.Server.ServerWide.Context.TransactionOperationContext.CreateReadTransaction() in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 33
       at Raven.Server.ServerWide.Context.TransactionOperationContext`1.OpenReadTransaction(String caller) in /_/62092/src/Raven.Server/ServerWide/Context/TransactionOperationContext.cs:line 74
       at Raven.Server.Documents.Indexes.Index.GetStats(Boolean calculateLag, Boolean calculateStaleness, Boolean calculateMemoryStats, Boolean calculateLastBatchStats, QueryOperationContext queryContext) in /_/62092/src/Raven.Server/Documents/Indexes/Index.cs:line 3168
       at Raven.Server.Documents.Handlers.Processors.Indexes.IndexHandlerProcessorForGetDatabaseIndexStatistics.HandleCurrentNodeAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/Indexes/IndexHandlerProcessorForGetDatabaseIndexStatistics.cs:line 126
       at Raven.Server.Documents.Handlers.Processors.AbstractHandlerProxyReadProcessor`3.ExecuteAsync() in /_/62092/src/Raven.Server/Documents/Handlers/Processors/AbstractHandlerProxyReadProcessor.cs:line 30
       at Raven.Server.Documents.Handlers.IndexHandler.Stats() in /_/62092/src/Raven.Server/Documents/Handlers/IndexHandler.cs:line 118
       at Raven.Server.Routing.RequestRouter.HandlePath(RequestHandlerContext reqCtx) in /_/62092/src/Raven.Server/Routing/RequestRouter.cs:line 417
       at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in /_/62092/src/Raven.Server/RavenServerStartup.cs:line 183
      CancellationToken: System.Threading.CancellationToken
      Stack Trace:
         at Raven.Client.Exceptions.ExceptionDispatcher.Throw(JsonOperationContext context, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Exceptions.ExceptionDispatcher.Throw(JsonOperationContext context, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Http.Behaviors.DefaultCommandResponseBehavior.TryHandleUnsuccessfulResponseAsync[TResult](JsonOperationContext context, RavenCommand`1 command, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Http.RequestExecutor.HandleUnsuccessfulResponse[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, HttpRequestMessage request, HttpResponseMessage response, String url, SessionInfo sessionInfo, Boolean shouldRetry, CancellationToken token)
       at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
       at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
       at Raven.Client.Documents.Operations.MaintenanceOperationExecutor.SendAsync[TResult](IMaintenanceOperation`1 operation, CancellationToken token)
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.WaitForIndexDefinitionUpdate(IndexStats oldStats) in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 118
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.UpdateIndex(IAbstractIndexCreationTask index) in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 109
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.Indexes_should_be_reset_on_setup() in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 54
       at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
       at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
       at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
       at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
       at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
       at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
       at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
       at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
    --OperationCanceledException
    
    1)    at Raven.Client.Exceptions.ExceptionDispatcher.Throw(JsonOperationContext context, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Exceptions.ExceptionDispatcher.Throw(JsonOperationContext context, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Http.Behaviors.DefaultCommandResponseBehavior.TryHandleUnsuccessfulResponseAsync[TResult](JsonOperationContext context, RavenCommand`1 command, HttpResponseMessage response, CommandUnsuccessfulResponseBehavior unsuccessfulResponseBehavior)
       at Raven.Client.Http.RequestExecutor.HandleUnsuccessfulResponse[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, HttpRequestMessage request, HttpResponseMessage response, String url, SessionInfo sessionInfo, Boolean shouldRetry, CancellationToken token)
       at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
       at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
       at Raven.Client.Documents.Operations.MaintenanceOperationExecutor.SendAsync[TResult](IMaintenanceOperation`1 operation, CancellationToken token)
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.WaitForIndexDefinitionUpdate(IndexStats oldStats) in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 118
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.UpdateIndex(IAbstractIndexCreationTask index) in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 109
       at ServiceControl.Audit.Persistence.Tests.IndexSetupTests.Indexes_should_be_reset_on_setup() in /_/src/ServiceControl.Audit.Persistence.Tests.RavenDB/IndexSetupTests.cs:line 54
       at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
       at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
       at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
       at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
       at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
       at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
       at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
       at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
    --OperationCanceledException
    

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions