util-jackson Guide

The library builds upon the excellent jackson-module-scala for JSON support by wrapping the Jackson ScalaObjectMapper to provide an API very similar to the Jerkson Parser.

Additionally, the library provides a default improved case class deserializer which accumulates deserialization errors while parsing JSON into a case class instead of failing-fast, such that all errors can be reported at once.

🚨 This documentation assumes some level of familiarity with Jackson JSON Processing

Specifically Jackson databinding and the Jackson ObjectMapper.

There are several tutorials (and external documentation) which may be useful if you are unfamiliar with Jackson.

Additionally, you may want to familiarize yourself with the Jackson Annotations as they allow for finer-grain customization of Jackson databinding.

Case Classes

As mentioned, Jackson is a JSON processing library. We generally use Jackson for databinding, or more specifically:

  • object serialization: converting an object into a JSON String and
  • object deserialization: converting a JSON String into an object

This library integration is primarily centered around serializing and deserializing Scala case classes. This is because Scala case classes map well to the two JSON structures [reference]:

  • A collection of name/value pairs. Generally termed an object. Here, the name/value pairs are the case field name to field value but can also be an actual Scala Map[T, U] as well.
  • An ordered list of values. Typically an array, vector, list, or sequence, which for case classes can be represented by a Scala Iterable.

Library Features

Basic Usage

Let’s assume we have these two case classes:

case class Bar(d: String)
case class Foo(a: String, b: Int, c: Bar)

To serialize a case class into a JSON string, use

ScalaObjectMapper#writeValueAsString(any: Any): String // or
JSON#write(any: Any)

For example:

To deserialize a JSON string into a case class, use

ScalaObjectMapper#parse[T](s: String): T // or
JSON#parse[T](s: String): T

For example, assuming the same Bar and Foo case classes defined above:

Tip

As seen above you can use the c.t.util.jackson.JSON utility for general JSON serde operations with a default configured ScalaObjectMapper.

This can be useful when you do not need to use a specifically configured ScalaObjectMapper or do not wish to perform any Bean Validation 2.0 style validations during JSON deserialization since the c.t.util.jackson.JSON utility specifically disables validation support on its underlying c.t.util.jackson.ScalaObjectMapper.

See the documentation for more information.

You can find many examples of using the ScalaObjectMapper in the various framework tests:

As mentioned above, there is also a plethora of Jackson tutorials and HOW-TOs available online which provide more in-depth examples of how to use a Jackson ObjectMapper.

ScalaObjectMapper

The ScalaObjectMapper is a thin wrapper around a configured jackson-module-scala ScalaObjectMapper. However, the util-jackson ScalaObjectMapper comes configured with several defaults when instantiated.

Defaults

The following integrations are provided by default when using the ScalaObjectMapper:

Instantiation

Instantiation of a new ScalaObjectMapper can done either via a companion object method or via the ScalaObject#Builder to specify custom configuration.

ScalaObjectMapper#apply

The companion object defines apply methods for creation of a ScalaObjectMapper configured with the defaults listed above:

import com.twitter.util.jackson.ScalaObjectMapper
import com.fasterxml.jackson.databind.{ObjectMapper => JacksonObjectMapper}
import com.fasterxml.jackson.module.scala.{ScalaObjectMapper => JacksonScalaObjectMapper}

val objectMapper: ScalaObjectMapper = ScalaObjectMapper()

val underlying: JacksonObjectMapper with JacksonScalaObjectMapper = ???
val objectMapper: ScalaObjectMapper = ScalaObjectMapper(underlying)

Important

The above #apply which takes an underlying Jackson ObjectMapper will mutate the configuration of the underlying Jackson ObjectMapper to apply the default configuration to the given Jackson ObjectMapper. Thus it is not expected that this underlying Jackson ObjectMapper be a shared resource.

Companion Object Wrappers

The companion object also defines other methods to easily obtain some specifically configured ScalaObjectMapper which wraps an already configured Jackson ObjectMapper:

import com.twitter.util.jackson.ScalaObjectMapper
import com.fasterxml.jackson.databind.{ObjectMapper => JacksonObjectMapper}
import com.fasterxml.jackson.module.scala.{ScalaObjectMapper => JacksonScalaObjectMapper}

val underlying: JacksonObjectMapper with JacksonScalaObjectMapper = ???

// different from `#apply(underlying)`. wraps a copy of the given Jackson mapper
// and does not apply any configuration.
val objectMapper: ScalaObjectMapper = ScalaObjectMapper.objectMapper(underlying)

// merely wraps a copy of the given Jackson mapper that is expected to be configured
// with a YAMLFactory, does not apply any configuration.
val objectMapper: ScalaObjectMapper = ScalaObjectMapper.yamlObjectMapper(underlying)

// only sets the PropertyNamingStrategy to be PropertyNamingStrategy.LOWER_CAMEL_CASE
// to a copy of the given Jackson mapper, does not apply any other configuration.
val objectMapper: ScalaObjectMapper = ScalaObjectMapper.camelCaseObjectMapper(underlying)

// only sets the PropertyNamingStrategy to be PropertyNamingStrategy.SNAKE_CASE
// to a copy of the given Jackson mapper, does not apply any other configuration.
val objectMapper: ScalaObjectMapper = ScalaObjectMapper.snakeCaseObjectMapper(underlying)

These methods clone the underlying mapper, they do not mutate the given Jaskson mapper.

Note that these methods will copy the underlying Jackson mapper (not mutate it) to apply any necessary configuration to produce a new ScalaObjectMapper, which in this case is only to change the PropertyNamingStrategy accordingly. No other configuration changes are made to the copy of the underlying Jackson mapper.

Specifically, note that the ScalaObjectMapper.objectMapper(underlying) wraps a copy but does not mutate the original configuration of the given underlying Jackson ObjectMapper. This is different from the ScalaObjectMapper(underlying) which mutates the given underlying Jackson ObjectMapper to apply all of the default configuration to produce a ScalaObjectMapper.

ScalaObjectMapper#Builder

You can use the ScalaObjectMapper#Builder for more advanced control over configuration options for producing a configured ScalaObjectMapper. For example, to create an instance of a ScalaObjectMapper with case class validation via the util-validator ScalaValidator disabled:

import com.twitter.util.jackson.ScalaObjectMapper

val objectMapper: ScalaObjectMapper = ScalaObjectMapper.builder.withNoValidation.objectMapper

See the Advanced Configuration section for more information.

Advanced Configuration

To apply more custom configuration to create a ScalaObjectMapper, there is a builder for constructing a customized mapper.

E.g., to set a PropertyNamingStrategy different than the default:

Or to set additional modules or configuration:

You can also get a camelCase, snake_case, or even a YAML configured mapper.

Access to the underlying Jackson Object Mapper

The ScalaObjectMapper is a thin wrapper around a configured Jackson jackson-module-scala com.fasterxml.jackson.module.scala.ScalaObjectMapper, thus you can always access the underlying Jackson object mapper by calling underlying:

Adding a Custom Serializer or Deserializer

For more information see the Jackson documentation for Custom Serializers.

Add a Jackson Module to a ScalaObjectMapper

Follow the steps to create a Jackson Module for the custom serializer or deserializer then register the module to the underlying Jackson mapper from the ScalaObjectMapper instance:

Improved case class deserializer

The library provides a case class deserializer which overcomes some limitations in jackson-module-scala:

The case class deserializer is added by default when constructing a new ScalaObjectMapper.

Tip

Note: with the util-jackson case class deserializer, non-option fields without default values are considered required. If a required field is missing, a CaseClassMappingException is thrown.

JSON null values are not allowed and will be treated as a “missing” value. If necessary, users can specify a custom deserializer for a field if they want to be able to parse a JSON null into a Scala null type for a field. Define your deserializer, NullAllowedDeserializer then annotate the field with @JsonDeserialize(using = classOf[NullAllowedDeserializer]).

@JsonCreator Support

The util-jackson case class deserializer supports specification of a constructor or static factory method annotated with the Jackson Annotation, @JsonCreator (an annotation for indicating a specific constructor or static factory method to use for instantiation of the case class during deserialization).

For example, you can annotate a method on the companion object for the case class as a static factory for instantiation. Any static factory method to use for instantiation MUST be specified on the companion object for case class:

Or to specify a secondary constructor to use for case class instantiation:

Note

If you define multiple constructors on a case class, it is required to annotate one of the constructors with @JsonCreator.

To annotate the primary constructor (as the syntax can seem non-intuitive because the () is required):

The parens are needed because the Scala class constructor syntax requires constructor annotations to have exactly one parameter list, possibly empty.

If you define multiple case class constructors with no visible @JsonCreator constructor or static factory method via a companion, deserialization will error.

@JsonFormat Support

The util-jackson case class deserializer supports @JsonFormat-annotated case class fields to properly contextualize deserialization based on the values in the annotation.

A common use case is to be able to support deserializing a JSON string into a “time” representation class based on a specific pattern independent of the time format configured on the ObjectMapper or even the default format for a given deserializer for the type.

For instance, the library provides a specific deserializer for the com.twitter.util.Time class. This deserializer is a Jackson ContextualDeserializer and will properly take into account a @JsonFormat-annotated field.

However, the util-jackson case class deserializer is invoked first and acts as a proxy for deserializing the time value. The case class deserializer properly contextualizes the field for correct deserialization by the TimeStringDeserializer.

Thus if you had a case class defined:

The following JSON:

{
  "id": 42,
  "description": "Something happened.",
  "when": "2018-09-14T23:20:08.000-07:00"
}

Will always deserialize properly into the case class regardless of the pattern configured on the ObjectMapper or as the default of a contextualized deserializer:

Welcome to Scala 2.12.13 (JDK 64-Bit Server VM, Java 1.8.0_242).
Type in expressions for evaluation. Or try :help.

scala> import com.fasterxml.jackson.annotation.JsonFormat
import com.fasterxml.jackson.annotation.JsonFormat

scala> import com.twitter.util.Time
import com.twitter.util.Time

scala> case class Event(
     |   id: Long,
     |   description: String,
     |   @JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSXXX") when: Time
     | )
defined class Event

scala> val json = """
     | {
     |   "id": 42,
     |   "description": "Something happened.",
     |   "when": "2018-09-14T23:20:08.000-07:00"
     | }""".stripMargin
json: String =
"
{
  "id": 42,
  "description": "Something happened.",
  "when": "2018-09-14T23:20:08.000-07:00"
}"

scala> import com.twitter.util.jackson.ScalaObjectMapper
import com.twitter.util.jackson.ScalaObjectMapper

scala> val mapper = ScalaObjectMapper()
mapper: com.twitter.util.jackson.ScalaObjectMapper = com.twitter.util.jackson.ScalaObjectMapper@52dc71b2

scala> val event: Event = mapper.parse[Event](json)
event: Event = Event(42,Something happened.,2018-09-15 06:20:08 +0000)

Jackson InjectableValues Support

By default, the library does not configure any com.fasterxml.jackson.databind.InjectableValues implementation.

@InjectableValue

It does however provide the c.t.util.jackson.annotation.InjectableValue annotation which can be used to mark other java.lang.annotation.Annotation interfaces as annotations which support case class field injection via Jackson com.fasterxml.jackson.databind.InjectableValues.

That is, users can create custom annotations and annotate them with @InjectableValue. This then allows for a configured Jackson com.fasterxml.jackson.databind.InjectableValues implementation to be able to treat these annotations similar to the @JacksonInject.

This means a custom Jackson InjectableValues implementation can use the @InjectableValue marker annotation to resolve fields annotated with annotations that have the @InjectableValue marker annotation as injectable fields.

For more information on the Jackson @JacksonInject or c.f.databind.InjectableValues support see the tutorial here.

Mix-in Annotations

The Jackson Mix-in Annotations provide a way to associate annotations to classes without needing to modify the target classes themselves. It is intended to help support 3rd party datatypes where the user cannot modify the sources to add annotations.

The util-jackson case class deserializer supports Jackson Mix-in Annotations for specifying field annotations during deserialization with the case class deserializer.

For example, to deserialize JSON into the following classes that are not yours to annotate:

case class Point(x: Int, y: Int) {
  def area: Int = x * y
}

case class Points(points: Seq[Point])

However, you want to enforce field constraints with validations during deserialization. You can define a Mix-in,

import com.fasterxml.jackson.annotation.JsonIgnore
import jakarta.validation.constraints.{Max, Min}

trait PointMixIn {
  @Min(0) @Max(100) def x: Int
  @Min(0) @Max(100) def y: Int
  @JsonIgnore def area: Int
}

Then register this Mix-in for the Point class type. There are several ways to do this:

Follow the steps to create a Jackson Module for the Mix-in then register the module to the underlying Jackson mapper from the ScalaObjectMapper instance:

import com.fasterxml.jackson.databind.module.SimpleModule
import com.twitter.util.jackson.ScalaObjectMapper

object PointMixInModule extends SimpleModule {
    setMixInAnnotation(classOf[Point], classOf[PointMixIn]);
}

...

val objectMapper: ScalaObjectMapper = ???
objectMapper.registerModule(PointMixInModule)

Or register the Mix-in for the class type directly on the underlying Jackson mapper (without a Jackson Module):

import com.twitter.util.jackson.ScalaObjectMapper

val objectMapper: ScalaObjectMapper = ???
objectMapper.underlying.addMixin[Point, PointMixIn]

Deserializing this JSON would then error with failed validations:

{
  "points": [
    {"x": -1, "y": 120},
    {"x": 4, "y": 99}
  ]
}

As the first Point instance has an x-value less than the minimum of 0 and a y-value greater than the maximum of 100.

Known CaseClassDeserializer Limitations

The util-jackson case class deserializer provides a fair amount of utility but can not and does not support all Jackson Annotations. The behavior of supporting a Jackson Annotation can at times be ambiguous (or even nonsensical), especially when it comes to combining Jackson Annotations and injectable field annotations.

Java Enums

We recommend the use of Java Enums for representing enumerations since they integrate well with Jackson’s ObjectMapper and have exhaustiveness checking as of Scala 2.10.

The following Jackson annotations may be useful when working with Enums:

  • @JsonValue: can be used for an overridden toString method.
  • @JsonEnumDefaultValue: can be used for defining a default value when deserializing unknown Enum values. Note that this requires READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE feature to be enabled.

c.t.util.jackson.JSON

The library provides a utility for default mapping of JSON to an object or writing an object as a JSON string. This is largely inspired by the scala.util.parsing.json.JSON from the Scala Parser Combinators library.

However, the c.t.util.jackson.JSON utility uses a default configured ScalaObjectMapper and is thus more full featured than the scala.util.parsing.json.JSON utility.

Important

The c.t.util.jackson.JSON API does not return an exception when parsing but rather returns an Option[T] result. When parsing is successful, this is a Some(T), otherwise it is a None. But note that the specifics of any failure are lost.

It is thus also important to note that for this reason that the c.t.util.jackson.JSON uses a default configured ScalaObjectMapper with validation specifically disabled, such that no Bean Validation 2.0 style validations are performed when parsing with c.t.util.jackson.JSON.

Users should prefer using a configured ScalaObjectMapper to perform validations in order to be able to properly handle validation exceptions.

c.t.util.jackson.YAML

Similarly, there is also a utility for YAML serde operations with many of the same methods as c.t.util.jackson.JSON using a default ScalaObjectMapper configured with a YAMLFactory.

Important

Like with c.t.util.jackson.JSON, the c.t.util.jackson.YAML API does not return an exception when parsing but rather returns an Option[T] result.

Users should prefer using a configured ScalaObjectMapper to perform validations in order to be able to properly handle validation exceptions.

c.t.util.jackson.JsonDiff

The library provides a utility for comparing JSON strings, or structures that can be serialized as JSON strings, via the c.t.util.jackson.JsonDiff utility.

JsonDiff provides two functions: diff and assertDiff. The diff method allows the user to decide how to handle JSON differences by returning an Option[JsonDiff.Result] while assertDiff throws an AssertionError when a difference is encountered.

The JsonDiff.Result#toString contains a textual representation meant to indicate where the expected and actual differ semantically. For this representation, both expected and actual are transformed to eliminate insigificant lexical diffences such whitespace, object key ordering, and escape sequences. Only the first difference in this representation is indicated. When an AssertError is thrown, the JsonDiff.Result#toString is used to populate the exception message.

For example:

Normalization

Both API methods accept a “normalize function” which is a function to apply on the actual to “normalize” any fields – such as a timestamp – before comparing to the expected.