Release of Daml Connect 1.12.0

Daml Connect 1.12.0 has been released on Wednesday April 14th. You can install it using:

daml install latest

Want to know what's happening more broadly among Daml developers? Check out the latest Daml Developer Monthly.

Summary

• Daml projects can now depend on packages deployed to the target ledger.
• The Daml StdLib contains a new module DA.List.BuiltinOrder, which provides higher-performance versions of several functions in DA.List by using a built-in order rather than user-defined ordering. We observed up to five-fold speedups in benchmarks.
• The Daml assistant now supports the Daml Connect Enterprise Edition (DCEE), which comes with additional features:
  • The DCEE has an Alpha feature to validate and store signatures on command submissions. This provides non-repudiation in cases where the Ledger API is exposed as a service.
  • The DCEE contains a Profiler in Alpha which allows you to extract performance information for Daml interpretation.

Impact and Migration

Some of the minor changes and bugfixes may require small actions to migrate:

• Daml Triggers now use DA.Map instead of DA.Next.Map in their API. 
• If you were directly targeting .dalf packages using data-dependencies, you now need to add --package flags to make the package contents available.
• The Scala bindings now depend on Scala 2.12.13 instead of 2.12.12
• The compiler now produces Daml-LF 1.12 by default. If you want to stick to the old default, Daml-LF 1.11, please add the build-option --target=1.11.
• Some gRPC status codes on rejected commands were changed from INVALID_ARGUMENT to ABORTED.

What’s New

Remote Dependencies

Background

The data-dependencies stanza in daml.yaml project files lists the binary dependencies which the project relies on. Until now, these dependencies were file-based, requiring developers to specify a path to .dar files. Quite often the dependencies are already running on a ledger, in which case this required the developer to first get hold of a .dar file from the original source or by calling daml ledger fetch-dar. This new feature removes that extra step, allowing a Daml project to depend directly on a package already running on a ledger.

Specific Changes

Package names and versions, as well as package ID's are allowed in the data-dependencies list of daml.yaml. These packages are fetched from the project ledger. The auto-generated daml.lock file keeps track of the package name/version to the package ID's resolution and should be checked into version control of the project. For example, to depend on the package foo-1.0.0 running on a ledger on localhost:6865, your daml.yaml should contain:

ledger:

  host: localhost

  port: 6865

dependencies:

- daml-prim

- daml-stdlib

data-dependencies:

- foo:1.0.0

Impact and Migration

This is a purely additive change.

High-Performance List Operations

Background

A number of functions in the standard library for lists, DA.List, rely on the elements of the list being orderable. They use orderings defined in Daml through Ord instances and thus need Daml interpretation for every comparison. Since Daml-LF 1.11, Daml has a canonical inbuilt ordering on all values, which is considerably more performant than comparison on Daml-defined orderings. This allows the implementation of higher-performance versions of all of ordering-based algorithms. The new, high performance implementations are contained in a new module DA.List.BuiltinOrder.

As long as all orderings are derived automatically using deriving Ord, the results of the old and new implementations will agree, but the new version is substantially faster. If any values have user-defined Ord instances, these will be ignored by the new versions, hence the name BultinOrder.

Specific Changes

The Daml Standard Library has a new module DA.List.BuiltinOrder containing more efficient implementations of the sort*, unique*, and dedup* functions based on the builtin order. We saw up to five-fold speedups in our benchmarks.

Impact and Migration

This is a purely additive change.

If you don’t care about the actual ordering of values, but only that values are orderable for algorithmic use, we recommend switching to the new versions of the algorithms for performance reasons. 

Daml Assistant support for the Daml Connect Enterprise Edition

Background

Daml Connect Enterprise Edition (DCEE) is a commercial distribution of Daml Connect containing additional features particularly useful in large or complex projects. DCEE components have been distributed via Artifactory for several releases now. With this release, the Daml Assistant also becomes aware of the Enterprise Edition and is able to manage additional developer tools and components not available in the Community Edition.

This release also contains two new features in Early Access for the DCEE, described in more detail below.

Specific Changes

The assistant now supports an artifactory-api-key field in daml-config.yaml. If you have a license for DCEE you can specify this and the assistant will automatically fetch the DCEE which provides additional functionality. See the installation documentation for more detail.

Impact and Migration

If you are an Enterprise Edition user, we recommend adding the artifactory-api-key field to your daml-config.yaml to benefit from the new features. If you already have the Community Edition installed, run daml install --force VERSION after setting the API key to replace it with the Enterprise Edition instead.

[Enterprise Edition] Daml Profiler in Alpha

Background

For large and complex Daml solutions, the speed of Daml interpretation can have a significant impact on overall system performance. The Daml Profiler now provides a tool for the developers of Daml applications to extract the timing information of real-world workflows in the well-established Speedscope file format, and analyse the performance characteristics using standard tools.

Specific changes

The Daml Sandbox distributed as part of the Enterprise Edition of Daml Connect has a new command line flag --profile-dir. If set, the timings of the interpretation of every submitted command will be stored in a json file in the provided directory. These profiles are compatible with standard analysis tools like Speedscope.
Please refer to the documentation for a complete usage example.

Impact and MIgration

This is a purely additive change.

[Enterprise Edition] Non-Repudiation in Alpha

Background

There are many scenarios in which the Daml Ledger API is offered as a service, and Daml fully supports such deployment topologies through its multi-tenancy participant node design. With this new feature, the operator of a participant node that offers the Ledger API as a service to third parties gains additional security. The non-repudiation middleware captures, validates, and stores cryptographic signatures on command submissions, providing the operator with a strong audit trail to evidence that Ledger API clients did indeed submit commands matching the recorded transactions.

Specific Changes

Enterprise Edition users have access to a new runtime component on Artifactory. The component proxies the Daml Ledger API and validates and stores cryptographic signatures on any calls to the command submission services.

Please refer to the documentation for more details.

Impact and Migration

This is a purely additive change.

Minor Improvements

• When running Daml Script on the command line you will now see a Daml stack trace on failures to interact with the ledger which makes it significantly easier to track down which of the call fails. By default, you will only get the callsite of functions like submit. To extend the stack trace, add HasCallStack constraints to functions and those will also be included.
• The Daml Assistant now also allows the sandbox port to be configured via --sandbox-option=--port=12345 instead of --sandbox-port. Other tools like Navigator, the JSON API and Daml Script will pick up the modified port automatically.
• The trigger library now uses DA.Map instead of the deprecated DA.Next.Map if the targeted Daml-LF version supports it. This is a breaking change: Code that interfaced with the triggers library using DA.Next.Map, e.g. with Daml.Trigger.getCommandsInFlight or Daml.Trigger.Assert.testRule, will need to be changed to use DA.Map instead.
• The Scala 2.12 version of the Scala Ledger API Bindings now depends on Scala 2.12.13 instead of Scala 2.12.12.
• The compiler produces Daml-LF 1.12 by default. LF 1.12 significantly reduces transaction size.

Bugfixes

• gRPC status codes for inconsistency rejections and Daml-LF errors (ContractNotFound, ReplayMismatch) were brought in line with their intended meaning by changing them from INVALID_ARGUMENT to ABORTED.
• DALFs in data-dependencies that are imported directly now require corresponding --package flags to make them visible. If you specify DALFs instead of DARs you also have to list all transitive dependencies, but typically you only want to expose your direct dependencies. Previously this was impossible. With this change, DALFs that are data-dependencies are no longer treated as main DALFs so you have more control over which packages get exposed.
• The Scala Codegen now supports the generic Map type added in Daml-LF 1.11 properly. Previously there were some bugs around the variance of the key type parameter which resulted in Scala compile errors in some cases. See #8879.
• A bug in the Daml compiler was fixed where passing --ghc-option=-Werror also produced errors for warnings produced by -Wmissing-signatures even if the user did not explicitly enable this.

Integration Kit

• The implicit conversions between Raw types (and the conversions to and from ByteString) have been removed. You will need to explicitly convert if necessary. This should not be necessary for most use cases.
• Added a test suite for race conditions to the ledger-api-test-tool

What’s Next

• Despite the order of magnitude performance improvements we have already accomplished, this continues to be one of our top priorities. 
• Improved exception handling in Daml is progressing well and expected to land in one of the next Daml-LF versions.
• We are continuing work on several features for the Enterprise Edition of Daml Connect:
  • Working towards a stable release of the profiler for Daml. The profiler helps developers write highly performant Daml code.
  • Oracle DB support throughout the Daml Connect stack in addition to the current PostgreSQL support.
• A new primitive data type in Daml that allows arbitrary precision arithmetic. This will make it much easier to perform accurate numeric tasks in Daml.