Why Does My Synchronous WCF Call Hang?

This is a limitation of WCF which is not widely known. Lets suppose you have an WCF interface which contains a mixture of Task based and non task based methods:

[ServiceContract(Namespace = "WCFDispatching")]
public interface IRemotedService
{
    [OperationContract]
    Task<bool> MakeAsyncCall(int id);

    [OperationContract]
    void SyncCall(int id);
}

What will happen when you call both methods?

    async Task Work(IRemotedService service)
    {
        await service.MakeAsyncCall(50);
        service.SyncCall(150);
    }

The sad truth is that the second call will hang indefinitely with a rather long call stack:

System.Threading.WaitHandle.WaitOne
System.Runtime.TimeoutHelper.WaitOne
System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply
System.ServiceModel.Dispatcher.DuplexChannelBinder.Request
System.ServiceModel.Channels.ServiceChannel.Call
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService
System.ServiceModel.Channels.ServiceChannelProxy.Invoke
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke
WCFDispatching.Program.Work
[Resuming Async Method]
System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext
System.Threading.ExecutionContext.RunInternal
System.Threading.ExecutionContext.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run
System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0
System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke
System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0
System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction
System.Threading.Tasks.Task.FinishContinuations
System.Threading.Tasks.Task.FinishStageThree
System.Threading.Tasks.Task<bool>.TrySetResult
System.Threading.Tasks.TaskFactory<bool>.FromAsyncCoreLogic
System.Threading.Tasks.TaskFactory<bool>.FromAsyncImpl.AnonymousMethod__0
System.Runtime.AsyncResult.Complete
System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend
System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.SendCallback
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame
System.Runtime.AsyncResult.Complete
System.ServiceModel.Dispatcher.DuplexChannelBinder.AsyncDuplexRequest.Done
System.ServiceModel.Dispatcher.DuplexChannelBinder.AsyncDuplexRequest.GotReply
System.ServiceModel.Dispatcher.DuplexChannelBinder.HandleRequestAsReplyCore
System.ServiceModel.Dispatcher.DuplexChannelBinder.HandleRequestAsReply
System.ServiceModel.Dispatcher.ChannelHandler.HandleRequestAsReply
System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest
System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump

System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame
System.Runtime.AsyncResult.Complete
System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame
System.Runtime.AsyncResult.Complete
System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete
System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete
System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete
System.ServiceModel.Channels.OverlappedContext.CompleteCallback
System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame
System.Threading._IOCompletionCallback.PerformIOCompletionCallback

The interesting thing is that the synchronous call completes on the remote endpoint but the WCF client call hangs in this call stack. The problem is that WCF runs asynchronous method completions on the WCF channel dispatcher which seems to be single threaded just like a UI application with a message pump. When a blocking synchronous call is performed WCF waits normally in a stack for the read operation to complete like this

System.Threading.WaitHandle.InternalWaitOne
System.Threading.WaitHandle.WaitOne
System.Runtime.TimeoutHelper.WaitOne
System.ServiceModel.Channels.OverlappedContext.WaitForSyncOperation
System.ServiceModel.Channels.OverlappedContext.WaitForSyncOperation
System.ServiceModel.Channels.PipeConnection.WaitForSyncRead
System.ServiceModel.Channels.PipeConnection.Read
System.ServiceModel.Channels.DelegatingConnection.Read
System.ServiceModel.Channels.SessionConnectionReader.Receive
System.ServiceModel.Channels.SynchronizedMessageSource.Receive
System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive
System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive
System.ServiceModel.Dispatcher.DuplexChannelBinder.Request
System.ServiceModel.Channels.ServiceChannel.Call
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService
System.ServiceModel.Channels.ServiceChannelProxy.Invoke
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke

But in our case we have a different wait call stack where we are not waiting for a read to complete but in DuplexChannelBinder.SyncDuplexRequest.WaitForReply we are waiting that another thread sets an event to signal completion. This assumes that another thread is still receiving input from the remote connection which is not the case. We can see this when one looks who is setting the event:

image

To release our waiting thread another thread must call GotReply which is never going to happen. To get things working again you must make in your remoted interface either all methods synchronous or asynchronous. A sync/async mixture of remoted methods will likely cause deadlocks like shown above.

Below is the full sample code to reproduce the issue if you are interested

using System;
using System.ServiceModel;
using System.Threading.Tasks;

namespace WCFDispatching
{
    [ServiceContract(Namespace = "WCFDispatching")]
public interface IRemotedService
{
    [OperationContract]
    Task<bool> MakeAsyncCall(int id);

    [OperationContract]
    void SyncCall(int id);

    [OperationContract]
    Task SyncCallAsync_(int id);
}

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession, ConcurrencyMode = ConcurrencyMode.Multiple, IncludeExceptionDetailInFaults = true)]
public class RemotedService : IRemotedService
{
    public async Task<bool> MakeAsyncCall(int id)
    {
        await Task.Delay(10);
        Console.WriteLine($"Async call with id: {id} completed.");
        return true;
    }

    public async Task SyncCallAsync_(int id)
    {
        await Task.Delay(0);
        Console.WriteLine($"SyncCallAsync call with id {id} called.");
    }

    public void SyncCall(int id)
    {
        Console.WriteLine($"Sync call with id {id} called.");
    }
}

class Program
{
    const string PipeUri = "net.pipe://localhost/WCFDispatching";

    static void Main(string[] args)
    {
        new Program().Run(args);
    }

    bool bUseAsyncVersion = false;
    readonly string Help = "WCFDispatching.exe [-server] [-client [-async]]" + Environment.NewLine +
                "    -server      Create WCF Server" + Environment.NewLine +
                "    -client      Create WCF Client" + Environment.NewLine + 
                "    -async       Call async version of both API calls" + Environment.NewLine +
                "No options means client mode which calls async/sync WCF API which produces a deadlock.";


    private void Run(string[] args)
    {
        if( args.Length == 0)
        {
            Console.WriteLine(Help);
            return;
        }
        else
        {
            for (int i = 0; i < args.Length; i++)
            {
                string arg = args[i];
                switch(arg)
                {
                    case "-server":
                        StartServer();
                        Console.WriteLine("Server started");
                        Console.ReadLine();
                        break;
                    case "-client":
                        // this is the default
                        break;
                    case "-async":
                        bUseAsyncVersion = true;
                        break;
                    default:
                        Console.WriteLine(Help);
                        Console.WriteLine($"Command line argument {args[0]} is not valid.");
                        return;
                }
            }

            var service = CreateServiceClient<IRemotedService>(new Uri(PipeUri));
            Task waiter = Work(service);
            waiter.Wait();
            return;
        }

    }

    async Task Work(IRemotedService service)
    {
        await service.MakeAsyncCall(50);
        if (bUseAsyncVersion)  // this will work
        {
            await service.SyncCallAsync_(50);
        }
        else
        {
            service.SyncCall(150);  // this call will deadlock!
        }
    }

    internal static T CreateServiceClient<T>(Uri uri)
    {
        var binding = CreateDefaultNetNamedPipeBinding();
        var channelFactory = new ChannelFactory<T>(binding, PipeUri.ToString());
        var serviceClient = channelFactory.CreateChannel();
        var channel = (IContextChannel)serviceClient;
        channel.OperationTimeout = TimeSpan.FromMinutes(10);

        return serviceClient;
    }

    internal static ServiceHost StartServer()
    {
        var host = new ServiceHost(typeof(RemotedService));
        host.AddServiceEndpoint(implementedContract: typeof(IRemotedService), binding: CreateDefaultNetNamedPipeBinding(), address: PipeUri);
        host.Open();

        return host;
    }

    internal static NetNamedPipeBinding CreateDefaultNetNamedPipeBinding()
    {
        //Default setting for NetNamedPipeBinding.MaxReceivedMessageSize = 65,536 bytes
        //Default settings for NetNamedPipeBinding.ReaderQuotas
        //MaxDepth = 32, MaxStringContentLength = 8192, MaxArrayLength = 16384, MaxBytesPerRead = 4096, MaxNameTableCharCount = 16384
        TimeSpan timeOut = TimeSpan.FromMinutes(1000);
        var binding = new NetNamedPipeBinding(NetNamedPipeSecurityMode.None)
        {
            ReceiveTimeout = timeOut,
            MaxReceivedMessageSize = Int32.MaxValue,
            ReaderQuotas =
            {
                MaxArrayLength = Int16.MaxValue,
                MaxStringContentLength = Int32.MaxValue,
                MaxBytesPerRead = Int32.MaxValue
            }
        };
        return binding;
    }
}
}

To try it out first start the server from a shell with

WCFDispatching.exe -server

and then start the client with -client as option to get the deadlock. To call the fixed version add to the client call -client -async and the deadlock will not occur.

Advertisements

One thought on “Why Does My Synchronous WCF Call Hang?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.