Signaling Failure within Finagle

For a combination of historical and practical reasons, there are many ways to signal an unsuccessful response in Finagle. When extending Finagle, it is important to use the correct signaling mechanisms.

For more information on how these are presented to users outside of Finagle, see the Unsuccessful Responses page in the User’s Guide.

History

Historically, Finagle relied exclusively on a hierarchy of classes of exceptions defined in Exceptions.scala, with each exception corresponding to a specific problem. Pattern matching on the exception type was used to classify the exceptions and make decisions about how to handle them. As the number of exceptions grew, pattern matching became unwieldy and Failures, were introduced as uniform way for Finagle to signal failure internally. Failures are more general and the successor to ad-hoc Exceptions. They are composable, immutable, and allow us to send signals across process boundaries.

However, when fleshing out Finagle’s back pressure strategy, it became useful for exceptions other than just Failure to also be able to be flagged with various attributes that propagate across service boundaries. In order to accomplish this, the FailureFlags trait was introduced. FailureFlags can be applied to any Throwable, which simplifies the migration strategy as the type of Throwable no longer has to change.

Currently, a Failure can be thought of as a “Finagle runtime exception” as it contains no type-based information. Failure provides a convenient API for building FailureFlag-style exceptions, but should not be used for inspection.

Creating exceptions with FailureFlags

Any new module should ensure that the exceptions it creates have FailureFlags. This can be done by creating a new class that extends FailureFlags or by using convenience methods in Failure.scala.

If the class name of the exception is helpful to users use a custom exception class that extends FailureFlags. The class name will show up in logs and stats. This is preferred.

If class name of the exception is unimportant or it is truly a runtime-exception type error, creating a new Failure with flags set correctly is appropriate.

Responding with an arbitrary Throwable or Exception is not recommended as they will be converted generic exceptions. For instance, Thrift will convert them to TApplicationExceptions.

Examining and Modifying FailureFlags

To inspect a Throw response for flags, pattern match against the FailureFlags trait.

import com.twitter.finagle.FailureFlags
import com.twitter.util.Throw

response match {
  case Throw(f: FailureFlags[_]) if f.isFlagged(FailureFlags.$FLAG) =>
    // Logic here.
  case ...
}

FailureFlags responses should be immutable. All of the convenience methods for flagging and unflagging make copies. Use these methods to modify flags.

import com.twitter.finagle.FailureFlags
import com.twitter.util.Throw

// Flag a response as "nonRetryable"
response match {
  case Throw(f: FailureFlags[_]) => Future.exception(f.asNonRetryable)
}

FailureFlags and Failures outside of Finagle

In a typically-configured client, the ProcessFailures filter will unwrap (See Failure#unwrap) Failures and mask out FailureFlags that are unsafe to pass along to users.

Unwrapping a Failure will transform the Failure into whatever Throwable it had wrapped, discarding any additional information carried by the Failure. The reason for this unwrapping is because of the large API burden of moving over to Failure completely. By switching to a FailureFlags trait, this problem is avoided.

If a Failure does not wrap anything, the ProcessFailures filter will only mask off unsafe FailureFlags.

Propagating Responses

There’s some nuance as to how FailureFlags and unflagged exceptions traverse client/service boundaries and differ across protocols. These tables cover how unsuccessful responses propagate from service to client over the two most common protocols: HTTP and ThriftMux. The columns indicate how the response is transformed along the way.

ThriftMux

ThriftMux will currently pass the following FailureFlags: Rejected, Retryable, NonRetryable. See MuxFailure.scala.

Cause	Finagle (as Throw)	ThriftMux	Mux	Client Sees (as Throw)
SomeException(message) raised in handling request	SomeException	TApplicationException	RdispatchOk	TApplicationException(message)
Failure.rejected(message)	Failure	Failure	RdispatchNack	Failure(message) flagged Rejected & NonRetryable
any Throwable with HasFailureFlags	T: HasFailureFlags	T: HasFailureFlags	RdispatchError	Failure(throwable.toString) with supported FailureFlags

Failure.rejected creates a Failure with Rejected and Retryable flags.

If a custom Throwable is used, its message and FailureFlags (if any) will propagate but it will be converted in to a Failure. Any other information, such as stack traces, will be discarded. Due to Finagle’s asynchronous nature, stack traces are not particularly useful. Instead Finagle provides built in support for distributed tracing systems.

When sending these responses across service boundaries, toString is called on each hop. This means that messages will appear like “Failure(Failure(Failure(Underlying cause… ” when logged. This may change in the future.

HTTP

Http will pass the flag Rejected (a retryable nack) or a combination of Rejected and NonRetryable (a nonretryable nack) via headers. HTTP will completely discard any information (stack trace, message, etc) except for the flags. See HttpNackFilter.scala

Cause	Finagle (as Throw)	Http	Client Sees (as Throw)
SomeException(message) raised in handling request	SomeException	500 Internal Server Error	Return(Http.Response(500 w/ message))
any Throwable with HasFailureFlags flagged Rejected and NonRetryable	T: HasFailureFlags	503 Service Unavailable w/ ‘finagle-http-nack’ header	Failure flagged Rejected and NonRetryable
any Throwable with HasFailureFlags flagged Rejected	T: HasFailureFlags	503 Service Unavailable w/ ‘finagle-http-nonretryable-nack’ header	Failure flagged Rejected and NonRetryable