Yufan Gong

@yufangong

Posted February 20, 2019

February 2019 Release Notes - Version 19.2.0 ☔️

It is the rainy season in San Francisco, grab a cup of hot tea and take a look at our new February release!

Finagle

New Features

  • finagle-core: Added gauge is_marked_dead as an indicator of whether the host is marked as dead(1) or not(0) in FailFastFactory. 068a15e6

  • finagle-core: KeyCredentials.CertsAndKey has been added as an option for c.t.f.ssl.KeyCredentials for when the certificate and certificate chain are contained within the same file. b4a9e8cd

  • finagle-thriftmux: Additional information is now annotated in traces for clients using Scrooge generated Thrift bindings. c69a22f8, 32d9b56b, 6283c6ce. This includes:

    • RPC method name
    • Request serialization time, in nanoseconds
    • Request deserialization time, in nanoseconds
    • Response serialization time, in nanoseconds
    • Response deserialization time, in nanoseconds

Breaking API Changes

  • finagle-http: Removed Http.Client.withCompressionLevel because it wasn’t doing anything. To migrate your client, simply remove the configuration–it had absolutely no effect. 87944371

  • finagle-http: c.t.f.dispatch.ExpiringServerDispatcher was dead code. We removed it. ae0571c9

  • finagle-thrift: Removed newIface and newServiceIface methods from c.t.f.thrift.ThriftRichClient.MultiplexedThriftClient, which are deprecated in November 2017. 773b6e57

  • finagle-thrift: Removed deprecated APIs located in Thrift.scala:

    1. c.t.f.Thrift.Client.stats => use c.t.f.Thrift.Client.clientParam.clientStats
    2. c.t.f.Thrift.withProtocolFactory => use c.t.f.Thrift.client.withProtocolFactory
    3. c.t.f.Thrift.withClientId => usec.t.f.Thrift.client.withClientId
    4. c.t.f.Thrift.Server.serverLabel => use c.t.f.Thrift.Server.serverParam.serviceName
    5. c.t.f.Thrift.Server.serverStats => use c.t.f.Thrift.Server.serverParam.serverStats
    6. c.t.f.Thrift.Server.maxThriftBufferSize => use c.t.f.Thrift.Server.serverParam.maxThriftBufferSize c6eb2020

  • finagle-thrift: c.t.f.thrift.ThriftServiceIface.Filterable is removed, use c.t.f.thrift.service.Filterable instead. 23affacc

  • finagle-thrift: c.t.f.thrift.ThriftServiceIface is removed, use c.t.f.thrift.service.ThriftServicePerEndpoint instead. 23affacc

  • finagle-thriftmux: Removed deprecated APIs located in ThriftMux.scala:

    1. c.t.f.ThriftMux.Client.stats => use c.t.f.ThriftMux.Clien.clientParam.clientStats
    2. c.t.f.ThriftMux.Server.serverLabel => use c.t.f.ThriftMux.Server.serverParam.serviceName
    3. c.t.f.ThriftMux.Server.serverStats => use c.t.f.ThriftMux.Server.serverParam.serverStats
    4. c.t.f.ThriftMux.Server.maxThriftBufferSize => use c.t.f.ThriftMux.Server.serverParam.maxThriftBufferSize c6eb2020

  • finagle-thriftmux: ThriftMux.Client.pushMuxer is removed. Use ThriftMux.Client.standardMuxer instead. 90333b1a

  • finagle-thriftmux: ThriftMux.serverMuxer is removed. Use ThriftMux.Server.defaultMuxer instead. 90333b1a

  • finagle-base-http: Removed the c.t.f.http.Statuses java helper, which was deprecated two years ago in favor of using c.t.f.http.Status directly. 75a4a209

  • finagle-base-http: Removed the c.t.f.http.Versions java helper, which was deprecated two years ago in favor of using c.t.f.http.Version directly. f191f1db

  • finagle-base-http: Removed the c.t.f.http.Methods java helper, which was deprecated two years ago in favor of using c.t.f.http.Method directly. ccf10dbc

  • finagle-http: c.t.f.http.Response.Ok was removed. Use just Response() or Response.Proxy if you need to mock it. d93bb1c9

  • finagle-core: Drv.Aliased and Drv.newVose are now private, please construct a Drv instance using Drv.apply or Drv.fromWeights. 9c810dd3

  • finagle-core: c.t.f.BackupRequestLost is now removed. Please use c.t.f.Failure.ignorable instead. 02d3d524

Bug Fixes

  • finagle-http: Fix for a bug where HTTP/2 clients could retry requests that had a chunked body even if the request body was consumed. b031e757

  • finagle-http: Fix for a bug where HTTP clients could assume connections are reusable, despite having streaming requests in flight. 88a2d0ba

Runtime Behavior Changes

  • finagle-core: Faster Filters. Removes unnecessary Service.rescue proxies from the intermediate andThen-ed Filters. Previously in rare cases you might have seen a raw Exception not wrapped in a Future if the Filter threw. These will now consistently be lifted into a Future.exception. a2ddc727

  • finagle-core: MethodBuilder metrics filtering updated to now report rolled-up logical failures. 6e3bf33f

  • finagle-http: Disabling Netty3 cookies in favor of Netty4 cookies. fccd92c6

  • finagle-http: Removed the debug metrics http/cookie/dropped_samesites and http/cookie/flagless_samesites. 2de928ce

Deprecations

  • finagle-core: Multi-param legacy tls methods have been deprecated in c.t.f.param.ServerTransportParams and c.t.f.builder.ServerBuilder. Users should migrate to using the tls(SslServerConfiguration) method instead. fbfc6d1a

  • finagle-core: $client.withSession.maxIdleTime is now deprecated; use $client.withSessionPool.ttl instead to set the maximum allowed duration a connection may be cached for. 0f060e37

  • finagle-serversets: c.t.f.zookeeper.ZkResolver has been deprecated in favor of c.t.f.serverset2.Zk2Resolver. 9878a9ec

Finatra

Added

  • finatra-kafka: Expose timeout duration in FinagleKafkaConsumerBuilder dest(). abd68ddf

  • finatra-kafka-streams: Expose all existing RocksDb configurations. See c.t.f.k.config.FinatraRocksDBConfig for details on flag names, descriptions and default values. 1454867c

  • finatra-kafka-streams: Added two RocksDB flags related to block cache tuning, cache_index_and_filter_blocks and pin_l0_filter_and_index_blocks_in_cache. 3b0931e3

  • finatra-kafka: Adding an implicit implementation of c.t.app.Flaggable[c.t.finatra.kafka.domain.SeekStrategy] and c.t.app.Flaggable[org.apache.kafka.clients.consumer.OffsetResetStrategy]. ef071e54

  • finatra-http: Added support to serve c.t.io.Reader as a streaming response in c.t.finatra.http.internal.marshalling.CallbackConverter. e5bda446

  • finatra-kafka: Expose endOffsets() in FinagleKafkaConsumer. bcbb5774

  • finatra-kafka-streams: Adding missing ScalaDocs. Adding metric for elapsed state restore time. RocksDB configuration now contains a flag for adjusting the number of cache shard bits, rocksdb.block.cache.shard.bits. afd9a17c

  • finatra-jackson: Added @Pattern annotation to support finatra/jackson for regex pattern validation on string values. 862f0ab1

Changed

  • finatra-kafka-streams: Refactor package names. All classes moved from com.twitter.finatra.streams to com.twitter.finatra.kafkastreams. a2ad0ef3

  • finatra-kafka-streams: Delete deprecated and unused classes. ee948398

  • finatra-kafka-streams: c.t.finatra.streams.transformer.domain.Time is now the canonical representation of time for watermarks and timers. RichLong implicit from com.twitter.finatra.streams.converters.time has been renamed to RichFinatraKafkaStreamsLong. 093b31b8

  • finatra-jackson: Fix CaseClassField annotation reflection for Scala 2.12. 3747c1ab

  • finatra-kafka-streams: Combine FinatraTransformer with FinatraTransformerV2. cd455c43

  • finatra-thrift: The return type of ReqRepDarkTrafficFilterModule#newFilter has been changed from DarkTrafficFilter[MethodIface] to Filter.TypeAgnostic. 50184f1b

  • finatra-kafka: Add lookupBootstrapServers function that takes timeout as a parameter. e3426fb9

  • finatra-thrift: If a Controller is not configured with exactly one endpoint per method, it will throw an AssertionError instead of logging an error message. An attempt to use non-legacy functionality with a legacy Controller will throw an AssertionError. d1d6d1e0

  • finatra-kafka: Add flags for controlling rocksdb internal LOG file growth.

    • rocksdb.log.info.level Allows the setting of rocks log levels DEBUG_LEVEL, INFO_LEVEL, WARN_LEVEL, ERROR_LEVEL, FATAL_LEVEL, HEADER_LEVEL.
    • rocksdb.log.max.file.size The maximal size of the info log file.
    • rocksdb.log.keep.file.num Maximal info log files to be kept. c03a497c

  • finatra-kafka: Add admin routes for properties and topology information

    • /admin/kafka/streams/properties Dumps the KafkaStreamsTwitterServer#properties as plain text in the TwitterServer admin page.
    • /admin/kafka/streams/topology Dumps the KafkaStreamsTwitterServer#topology as plain text in the TwitterServer admin page. ecf2e54f

  • inject-server: EmbeddedTwitterServer that fails to start will now continue to throw the startup failure on calls to methods that require a successfully started server. 3ca4437c

Fixed

  • finatra-kafka-streams: FinatraTopologyTester did not set TopologyTestDriver#initialWallClockTimeMs on initialization causing diverging wall clock time when TopologyTestDriver#advanceWallClockTime advanced time. The divergence was between system time set by org.joda.time.DateTimeUtils.setCurrentMillisFixed and internal mock timer TopologyTestDriver#mockWallClockTime. FinatraTopologyTester.inMemoryStatsReceiver is reset on TopologyFeatureTest#beforeEach for all test that extend TopologyFeatureTest. 3b93a7d7

  • finatra-kafka-streams: Improve watermark assignment/propagation upon reading the first message and when caching key value stores are used. 9aa12b8d

  • finatra-jackson: Support inherited annotations in case class deserialization. Case class deserialization support does not properly find inherited Jackson annotations. This means that code like this:

  trait MyTrait {
    @JsonProperty("differentName")
    def name: String
  }
  case class MyCaseClass(name: String) extends MyTrait

would not properly expect an incoming field with name differentName to parse into the case class name field. This commit provides support for capturing inherited annotations on case class fields. Annotations processed in order, thus if the same annotation appears in the class hierarchy multiple times, the value configured on the class will win otherwise will be in the order of trait linearization with the “last” declaration prevailing. 6237ff86

  • finatra: Remove extraneous dependency on old javax.servlet ServletAPI dependency. The fixes #478. 85100952

Closed

Scrooge

No Changes

Twitter Server

No Changes

Util

New Features

  • util-core: updated Reader#fromFuture to resolve its onClose when reading of end-of-stream. f2a05474

  • util-core: Added Reader.flatten to flatten a Reader[Reader[_]] to Reader[_], and Reader.fromSeq to create a new Reader from a Seq. a49bab4d

  • util-core: Added Duration.fromMinutes to return a Duration from a given number of minutes. eda0b390

  • util-core: If given a Timer upon construction, c.t.io.Pipe will respect the close deadline and wait the given amount of time for any pending writes to be read. 0a142872

  • util-core: Optimized ConstFuture.proxyTo which brings the performance of flatMap and transform of a ConstFuture in line with map. de4cebda

  • util-core: Experimental toggle (com.twitter.util.BypassScheduler) for speeding up ConstFuture.map (transformTry). The mechanism, when turned on, runs map operations immediately (why not when we have a concrete value), instead of via the Scheduler, where it may be queued and potentially reordered, e.g.: f.flatMap { _ => println(1); g.map { _ => println(2) }; println(3) } will print 1 2 3, where it would have printed 1 3 2. aeafba1a

  • util-security: Pkcs8KeyManagerFactory now supports a certificates file which contains multiple certificates that are part of the same certificate chain. 337e270f

Bug Fixes

  • util-core: Fixed the behavior in c.t.io.Reader where Reader#flatMap fails to propagate parent reader’s onClose. f2a05474

Runtime Behavior Changes

  • util-core: Closing a c.t.io.Pipe will notify onClose when the deadline has passed whereas before the pipe would wait indefinitely for a read before transitioning to the Closed state. 0a142872

  • util-core: Don’t allow AsyncSemaphore Permits to be released multiple times. Before it was possible to release a permit more than once and incorrectly remove waiters from the queue. With this change, the release permit operation is idempotent. ea24c951

Breaking API Changes

  • util-core: Remove c.t.u.CountDownLatch which is an extremely thin shim around j.u.c.CountDownLatch that provides pretty limited value. To migrate to j.u.c.CountDownLatch, instead of c.t.u.CountDownLatch#await(Duration), please use j.u.c.CountDownLatch#await(int, TimeUnit), and instead of c.t.u.CountDownLatch#within(Duration), please throw an exception yourself after awaiting. b9914f15

  • util-core: Deprecated conversions in c.t.conversions have new implementations that follow a naming scheme of SomethingOps. df397f53

    • percent is now PercentOps
    • storage is now StorageUnitOps
    • string is now StringOps
    • thread is now ThreadOps
    • time is now DurationOps
    • u64 is now U64Ops

  • util-collection: Delete util-collection. We deleted GenerationalQueue, MapToSetAdapter, and ImmutableLRU, because we found that they were of little utility. We deleted LruMap because it was a very thin shim around a j.u.LinkedHashMap, where you override removeEldestEntry. If you need SynchronizedLruMap, you can wrap your LinkedHashMap with j.u.Collection.synchronizedMap. We moved RecordSchema into finagle-base-http because it was basically only used for HTTP messages, so its new package name is c.t.f.http.collection. 60eff0a4

  • util-core: Rename BlacklistStatsReceiver to DenylistStatsReceiver. ed3f7069

  • util-core: Buf.Composite is now private. Program against more generic, Buf interface instead. 41e24395

© 2014–2019 Twitter, Inc. and other contributors