Release of Daml Connect 1.10.0

Written by Anthony Lusardi | Feb 10, 2021

Daml Connect 1.10.0 has been released on Wednesday February 17th. You can install it using:

daml install 1.10.0

Want to know what's happening in our developer community? Check out the latest update for this month.

Highlights

  • Daml Triggers and OAuth Middleware are now Stable
  • Daml-LF 1.11 is now stable without any further changes from the beta version. It includes the following features and changes
    • Choice Observers
    • Generic Maps and DA.Map and DA.Set libraries are
    • Ledger API version is now at 1.9
  • Daml Studio now provides tooltips and go-to-definition even if the code doesn’t currently compile
  • Considerable performance improvements

Impact and Migration

  • This release is purely additive so no action needs to be taken. We recommend testing projects with Daml-LF 1.11 or fixing Daml-LF version at 1.8 (the current default) at this point. Daml-LF 1.11 will become the default with the next release of Daml Connect.

What’s New

Daml Triggers and Auth Middleware Stable

Background

Daml Triggers are a framework to add automation to your Daml application with maximum code reuse from contracts and scripts. Triggers are written in Daml, and have similar capabilities to Daml Script/REPL, but are run in reaction to ledger events rather than as one-offs. They provide an easy way to maintain functional dependencies between contracts, or to add automation of the kind that lets users auto-accept all incoming asset transfers from another party.

Daml Triggers are operated via a hosting service that is designed to run as part of Daml Connect, next to the JSON API. It allows users to authorize automation to act on their behalf. So in the above example, the user doesn’t have to host the auto-accept automation themselves, but instead authorizes the host service to run it on their behalf.

To make that authorization work smoothly we had to solve a separate problem that developers previously had to solve themselves: Keeping a long-running service supplied with valid JWT tokens to use on the Ledger API. Our solution for this problem is now also available as a stand-alone component - the OAuth 2.0 Middleware - so that everyone can profit from this development. The middleware can be deployed to sit between an OAuth 2.0 identity provider and any long-running Ledger API client. It manages initial authorization and handles refresh tokens so that its clients can easily get fresh JWTs without implementing the OAuth protocol in full by themselves.

Furthermore, solving this problem via a middleware adds a layer of abstraction between Daml Connect components and IAM systems. The Daml Trigger Host, for example, could be deployed against a completely different IAM system by implementing an auth middleware that provides the same simple API towards the client application.

Specific Changes

  • The Trigger Service is now stable
    • JAR distribution available form GH Releases
    • Documentation available on docs.daml.com
    • Changes since last Beta:
      • The trigger service now supports authorization through an auth middleware. The feature is enabled using the --auth and --auth-callback command-line flags. Please refer to the Authorization chapter of the trigger service documentation for further instructions.
  • The OAuth Middleware is now stable
  • Dockerized versions of both components are available as part of the Enterprise Edition of Daml Connect.

Impact and Migration

Users of Daml Triggers are advised to switch from using the daml trigger CLI command to using the Trigger service, either via daml trigger-service or the stand-alone JAR file as that’s the only supported way to run triggers in production. daml trigger is intended only as a development tool to spin up a single ephemeral trigger for testing.

Daml-LF 1.11 now Stable

Background

Daml-LF is Daml’s intermediary language, analogous to Java bytecode. With Daml Connect 1.10, Daml-LF 1.11 is now stable and frozen. Daml-LF 1.11 introduces two new language features: Choice Observers and Generic Maps. In addition, Daml Script, REPL and Triggers now properly convert from ContractId a value to Text using the show function.

Daml-LF 1.11 is not yet the default yet, but will be from the next Daml Connect release in March 2021. In the meantime, to use the new features, your project has to explicitly target Daml-LF 1.11. You can enable this in your daml.yaml file by adding stanza:

build-options:
- --target=1.11

Specific Changes

  • The Daml compiler can now target Daml-LF 1.11 by setting --target=1.11.
  • The Ledger API is now at version 1.9.0.
  • Choice observers, documented on the reference page for choices, add an additional keyword observer to the choice … syntax. Parties designated choice observers using that keyword are guaranteed to see the exercise of the choice. This can be useful for defining custom events, for example:

nonconsuming choice MyEvent : ()
with
message: Text
sender: Party
receivers: [Party]
observer receivers
controller sender
do
return ()

Here the parties in receivers will all see an exercise node of type MyEvent with a payload containing message.

  • Generic maps add a type GenMap and standard library modules DA.Map and DA.Set allowing maps and sets with arbitrary serializable types as keys/members. This supersedes the TextMap type and standard library modules DA.Next.Map and DA.Next.Set, which required conversion functions between the key type and Text. The DA.Next.* libraries are deprecated for use with Daml-LF >= 1.11.

    TextMap itself continues to be supported as there are use-cases where it provides a nicer representation on the API than GenMap. Generic maps are represented as lists of tuples on the APIs. In contrast, TextMap is represented as a JSON object on the JSON API.
  • All inbuilt and derived instances of Show now return the actual contract id in Daml Script, REPL, and Triggers, rather than the opaque ”<contract-id>” that’s returned in Daml contracts.

Impact and Migration

Users of DA.Next.* are advised to switch to TextMap or DA.Map/DA.Set with the upgrade to Daml-LF 1.11. However, DA.Next.* will continue to be maintained for full backwards compatibility in the 1.X release line.

Minor Improvements

  • Ongoing performance work has paid off in up to 6x transaction throughput increases (compared to Daml Connect 1.9) on some workloads and ledger integrations.
  • The JSON API server now also optimizes Websocket queries if run with persistence, leading to better performance.
  • Daml Studio now provides type information and go-to-definition even when the code currently doesn’t compile. The information provided is based on the cached results from the last successful compilation. 
  • The JSON API server now logs successful queries at debug level instead of info
  • The Trigger Host service now logs successful queries at debug level instead of info
  • Libraries generated via daml codegen have had their dependencies updated
    • iou.grpc:grpc-(api|core|netty|protobuf|services|stub) to 1.35.0.
    • io.netty:netty-(codec-http2|handler|handler-proxy|resolver)  to 4.1.58.Final.
    • io.netty:netty-tcnative-boringssl-static to 2.0.36.Final.
    • com.google.protobuf:protobuf-java to 3.14.0.

If this causes problems with dependency resolution in consuming projects, we recommend updating dependencies there, too, in order to apply the latest security patches.

  • The Sandbox now emits an easier to understand error message when accessing archived contracts.

Bugfixes

  • Under rare conditions, a multi-template query to a JSON API backed by Postgres could result in a response based on different ledger offsets for the different templates. These  conditions are now checked and accounted for. See issue #8226.
  • The Daml Driver for PostgresQL no longer logs the JDBC url to avoid potential password leaks.
  • Malformed party names are now correctly reported back with an INVALID_ARGUMENT error instead of the incorrect INTERNAL error code.
  • Three CLI options on daml damlc that were never meant to be public are now hidden from the output of various daml xxx --help commands. The options still have the same effect when present. The options did not work universally as they were primarily intended for internal use. If you do use them please let us know about your use-case so we can consider supporting it properly. The affected flags are:
    • --with-dlint
    • --without-dlint
    • --allow-overrides
  • When running Daml Script over the JSON API, the check that the parties in the Daml Script match the parties in the token has been relaxed and now allows for duplicate parties as well as parties that are only in actAs in the Daml Script but in both actAs and readAs in the token.

Daml-LF 1.12 now available in Beta

Background

See Daml-LF 1.11 above for background to Daml-LF. Daml-LF 1.12 is now included as an early access feature. Its main improvements are on performance, reducing transaction size through type erasure. It can be used in your project by adding the following stanzas to your daml.yaml file:

build-options:
- --target=1.12
sandbox-options:
- --early-access-unsafe

Note that projects using early access Daml-LF versions can not be deployed to production ledgers.

Integration Kit

  • The Ledger API Test Tool’s CommandDeduplicationIT now sets the deduplication timeout on commands using the parameter passed to --timeout-scale-factor, allowing it to run on ledger configurations that require more than 3 seconds to commit a submission.
  • Compression and decompression of stored Daml-LF values is now executed outside of the index database, allowing to make more efficient use of the participant resources when indexing.
  • Changed index method type back to B-Tree from Hash for: participant_events_template_id_idx, participant_events_contract_id_idx indices of participant_events table
  • Enabled --client-auth for kvutils config
  • Correctly reject transactions using the INCONSISTENT error code when contract keys referenced in a transaction have changed between submission and commit (previously other, less appropriate rejection reasons would be produced, such as DISPUTED)
  • Added a --tracker-retention-period parameter to the kvutils CLI to be able customization of how long the command service will keep an active command tracker for a given party.
  • Changed the default tracker retention period from 24 hours to 5 minutes
  • The database connection timeout is now configurable.

What’s Next

  • Despite the order of magnitude performance improvements we have already accomplished, this continues to be one of our top priorities. 
  • Improved exception handling in Daml is progressing well and expected to land in one of the next Daml-LF versions.
  • We are continuing work on several features for the Enterprise Edition of Daml Connect:
    • A profiler for Daml, helping developers write highly performant Daml code.
    • A mechanism to verify cryptographic signatures on commands submitted to the Ledger API, improving security in deployment topologies where the Ledger API is provided as a service.
    • Oracle DB support throughout the Daml Connect stack in addition to the current PostgreSQL support.
  • A new primitive data type in Daml that allows infinite precision arithmetic. This will make it much easier to perform accurate numeric tasks in Daml.