This project has moved and is read-only. For the latest updates, please go here.
1

Closed

AsyncReaderWriterLock causes deadlocks trying to acquire a write lock

description

We are using the AsyncReaderWriterLock to protect cached resources in an MVC4/Framework 4.5 application.

Each request takes a Read Lock, reads what it needs from the cache and then on Dispose (when the request completes as Ninject is maintaining the scope InRequestScope) the lock is released. When the cache needs to be updated, a request is made and the handler tries to take a Write Lock using the following pattern:
           Trace.WriteLine("Entering Writelock");

            try
            {
                IDisposable t = _lock.WriterLock(CancellationTokenHelpers.Timeout(timeout).Token);
                Trace.WriteLine("Enter Writelock Success");
                return t;
            }
            catch
            {
                Trace.WriteLine("Enter Writelock Failed");
                return null;
            }
What is supposed to happen is that the writer request will block until it is able to take the lock, but all other requests will finish (and release their locks) and new read requests will be queued.

However, when the application is under load and there are a number of open read locks, what actually happens is that immediately after calling _lock.WriterLock() a significant number of requests simply hang - they do not complete and their locks are never released. Once the _lock.WriterLock() is cancelled (by the timeout expiring) the other requests immediately complete and release their read locks. Of course, it is too late by this time and queuing requests waiting for Read Locks are also timing out.

Do you have any idea what might be causing this deadlock? I have stepped through the code and it is the line task.GetAwaiter().GetResult() in TaskExtensions.WaitAndUnwrapException which is hanging.
Closed Sep 5, 2014 at 2:59 AM by StephenCleary

comments

StephenCleary wrote Jan 12, 2014 at 1:00 PM

I discourage RWLs in general, since they're usually not the correct solution. In particular, it's not possible to make a RWL that is fully fair and also deadlock-free. RWLs also have quite a bit more overhead (and more potential for bugs) than a simple lock.

I'll take a look at this, and in the meantime try out a plain AsyncLock (or ConcurrentDictionary, which is what I usually use in cache scenarios).

StephenCleary wrote Jan 12, 2014 at 9:50 PM

I created some load tests for the ARWL, but have not been able to duplicate this issue.

Could you provide more information about how your code works? Things to look for:
  • Any request that would call WriterLock after it called ReaderLock.
  • Any request that calls ReaderLock multiple times.
Since your locks are being disposed when your requests complete, either of those situations would cause this deadlock.

Also, check in your web logs when this happens and see if ASP.NET is aborting the request thread. This can happen with synchronous requests, but not asynchronous requests.

thecontrarycat wrote Jan 13, 2014 at 5:14 PM

Thanks to your hints I managed to solve the problem I was having.

It turned out that replacing the ARWL for an AsyncLock immediately caused a deadlock whenever two requests were active at once. When I examined where the threads were deadlocking it became clear that the issue was related to Ninject.Extensions.Factory which was doing something clever with transparent proxy objects and in there it must have been calling the default constructor of my lock object (and thus trying to take the lock for the second time in the same request) even though there was no indication of this in the TraceLog and the lifetime scope of the object should not have required the creation of a new object.

Refactoring the code so that the lock was taken on demand instead of in the constructor fixed the issue and I am back using the ARWL as designed.

Thanks for the help - I doubt I'd have tracked this down as quickly (or at all!) without your hints.