Improved support for CQL Decimal

[dehall]

This PR migrates Decimals from being represented by plain JS number to being represented by a Decimal class. (CQL Integers remain represented by plain JS numbers.) Our new Decimal class is a wrapper around the decimal.js library. All interactions with the library are limited to one file so if we decide that's the wrong library, it should be straightforward to change.

Where possible, I've tried to make the changes developer-friendly, for instance, because Quantity.value is always a Decimal, the Quantity constructor accepts anything that can be converted to a Decimal, eg, a number, bigint, or string. This is primarily relevant to the unit tests where we have a lot of "result should equal(new Quantity(3, 'g')" style test expectations.

This change means that now all CQL types are represented 1:1 by their respective JS types, so passing a "type" argument around became unnecessary in several places.

Decimals need to be normalized to a max of 8 digits after the decimal point, and have a maximum and minimum value, and so my philosophy was to try to normalize and bounds check as few times as possible, and hence as late as possible: only in the ELM layer. There are some remaining instances of checking for overflow in the datatype layer that I could have removed, but that would require even more refactoring so I left them for now.

Per the spec, the string representation of a Decimal must always contain at least one digit on either side of the decimal point. (eg, 1.0 not 1 or 1., and 0.1 not .1, and not exponential notation like 1e8)

Changes here mostly fall into 3 categories:

• Arithmetic operator support
• Aggregate operator support
• Interval expansion
  • Now instead of separate logic for number/bigint/Decimal, interval expansion is always performed with Decimals and converted back to the appropriate type when each "sub-interval" is constructed
  • While looking at interval expansion, I saw some skipped spec tests that could be fixed by adding just a few lines, so true support for the "single interval" expand overload is now present

Note this PR does not implement the CQL Precision operator, and the Decimal class doesn't keep track of significant figures. (JS numbers didn't either, so this isn't a regression) This means that trailing zeros after the decimal point will not be preserved. eg:

Decimal.from("1.00000000").toString()
-->
"1.0"

decimal.js doesn't support this natively, so a future effort will have to add a second internal state field to track scale.

The two unit test failures are expected at this point (I removed the part of the expand Interval logic that covers those 2 specific tests) but I'm waiting for more direction on #cql > Interval Expand example before doing anything more on that front.

Notable Boundaries

There are a couple instances where interactions with plain JS numbers are forced:

• Quantity operations that interact with the UCUM unit conversion library
  • To try to reduce the chance of loss of precision, instead of calling the library with "n of unit A to unit B", I call the library with "1 of unit A to unit B" and then multiplying the input Decimal by the resulting factor.
• DateTime operations that interact with the library luxon, specifically the timezoneOffset field
  • I think these always should be integers or rational numbers in a small range, so I'm pretty sure every possible value can be exactly represented with number anyway

Also note that the Decimal constructor accepts JS numbers that do not need to represent integers, but there is the risk of loss of precision if the literal used cannot be represented precisely as a js number. To be safe, consumers of this library constructing a Decimal instance should generally use the string constructor which guarantees round-trip safety. Eg:

Decimal.from(1.0000000000000000000000000001).toString()
--> "1"
 
Decimal.from("1.0000000000000000000000000001").toString()
--> "1.0000000000000000000000000001"

Decimal.from(900719925474099230945).toString()
--> "900719925474099200000" 

Decimal.from("900719925474099230945").toString()
--> "900719925474099230945"

Pull requests into cql-execution require the following.
Submitter and reviewer should ✔ when done.
For items that are not-applicable, mark "N/A" and ✔.

Submitter:

• This pull request describes why these changes were made
• Code diff has been done and been reviewed (it does not contain: additional white space, not applicable code changes, debug statements, etc.)
• Tests are included and test edge cases
• Tests have been run locally and pass
• Code coverage has not gone down and all code touched or added is covered.
• Code passes lint and prettier (hint: use npm run check to run tests, lint, and prettier)
• All dependent libraries are appropriately updated or have a corresponding PR related to this change
  Reviewer:

Name:

• Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
• The tests appropriately test the new code, including edge cases
• You have tried to break the code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.99288% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.22%. Comparing base (277d26f) to head (02952ae).

Files with missing lines	Patch %	Lines
src/elm/arithmetic.ts	87.20%	6 Missing and 5 partials ⚠️
src/elm/interval.ts	83.05%	4 Missing and 6 partials ⚠️
src/datatypes/decimal.ts	93.75%	5 Missing and 2 partials ⚠️
src/util/math.ts	94.00%	3 Missing and 3 partials ⚠️
src/util/immutableUtil.ts	50.00%	2 Missing and 1 partial ⚠️
src/datatypes/datetime.ts	88.88%	1 Missing and 1 partial ⚠️
src/datatypes/interval.ts	88.88%	0 Missing and 2 partials ⚠️
src/elm/aggregate.ts	97.22%	1 Missing and 1 partial ⚠️
src/datatypes/quantity.ts	95.23%	1 Missing ⚠️
src/elm/type.ts	95.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #376      +/-   ##
==========================================
+ Coverage   88.70%   89.22%   +0.52%     
==========================================
  Files          59       60       +1     
  Lines        4933     5116     +183     
  Branches     1429     1468      +39     
==========================================
+ Hits         4376     4565     +189     
+ Misses        322      316       -6     
  Partials      235      235              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cmoesel]

Wow. This is a pretty intense PR! It's great to finally have more reasonable support for decimals and decimal-based operations. I've left some comments about some mostly small things. I think the biggest thing is I would like for us to support retention of decimal precision -- but I'm totally fine with doing that as a follow-on to this PR (rather than trying to fit it into this PR).

The other big question -- which is definitely a "for later" thing -- is if we might gain some simplicity by modeling all numeric types as CQL classes (e.g., Integer, Long, and Decimal). I think they could all be backed by decimal.js, and if they had a common base class and/or implemented a common interface it might simplify things a lot. This is just a half-baked idea, but I think it may be worth exploring -- especially if we're going to do a major release w/ breaking changes anyway.

One other thing to note: I'm guessing we may need to update cql-exec-fhir to account for the new Decimal class. Maybe after we review and merge this PR, we should do a 4.0.0-beta.1 release so we can try integrating it w/ a cql-exec-fhir beta release as well (which both will probably be needed to update / test fqm-execution). I know I said I wanted to target Connectathon for a release of this stuff, but I think a beta release would be fine (I feel ok reporting against a beta release if it's on the main branch).

[dehall]

Just pushed a series of commits based on some further testing:

• Added Decimal.truncatedDivideBy because just using regular divideBy then truncating meant that the value was first rounded, which produced incorrect results. To call this as appropriate, I added a new optional parameter in "MathUtil" divide to decide if you want truncated divide or not
• Added Decimal.truncateToBigInt as a companion to Decimal.truncate which returns a number. This is only actually used in Interval expand, but it could theoretically be used in the Truncate operation in the future
• Add Decimal.nthRoot to try to allow for more accurate results in some cases. Ideally, Decimal.power would be enough, but the way decimal.js implements the power operator can result in imprecise values when the power is a fraction like 1/3. This was already visible in examples like GeometricMean({4.0, 4.0, 4.0 }). For now the only special cases in nthRoot are 2 (square root, calls .sqrt), and 3 (cube root, calls decimal.js cubeRoot), otherwise it just defers to power as before.
• Made the parameter in Decimal.round optional. If it's omitted or null, the spec says treat it as if 0 were passed in.
• Updated the NormalizedKey definition for Decimals and used that key in Mode logic, so that it allows value equality. Since Decimal value equality ignores trailing zeros, that's done here.

I'm going to continue poking at this but as of now I'm calling it officially ready for re-review.

[cmoesel]

This is an impressive PR. Who knew decimals would affect so much! Thanks for working through it. I think we're super close.

I've only left a few small comments (some of them just to vent about things that annoy me in CQL). I also noticed that if I run npm run check:all some of the package-lock.json files change (based on the introduction of the decimal.js library).

Let me know if you want to discuss any of this!

[cmoesel]

This is correct (based on "trailing zeroes after the decimal are ignored in determining precision for equivalent comparison."), but just for the record, I don't like it! The spec generally says that precision matters, and I feel like if an input says 1.20 rather than 1.2, that means they measured to the hundredths -- and based on that, the fact that the hundredths place holds a 0 should not matter.

For example, 1.00000000 ~ 1.499999999 is true? That's ridiculous. If someone is using instruments to measure to the hundred-millionths, then obviously those small differences matter. We should not ignore them because the value happens to be even at 0s. I get why 1.00 ~ 1.00499999, because one number was measured at a different scale to begin with, but... 1.00000000 is a different story.

OK. Getting off my soapbox now. Again, there is nothing to do here. I just needed to vent.

[cmoesel]

I'm not sure about this one. The description of Except does not indicate anything about precision, nor does it define itself in terms of successor, predecessor, start, or end. We implemented it using some of those concepts, but that doesn't mean that was the correct way to implement it (and notably had different results since predecessor/successor always used .00000001 on decimals).

How did you determine that precision of the input intervals should be used here? I think we should request that this be clarified in the spec.

[dehall]

Good point. Looking at this more it may not even be Except that is or should be precision-aware, it's the other interval semantics. But it would be good for this to be more explicit.

Just to make this concrete, the skipped test DecimalIntervalExcept1to3 is Interval[1.0, 10.0] except Interval[4.0, 10.0]. The result kind of has to be [equal to] Interval[1.0, 4.0), and that's not based on the precision of the inputs. I don't think anything else makes sense. (Or at least, anything else would be intentionally dependent on a painful combination of Decimal instance precision and Decimal type point size.) But the test output presents it in a closed form, and converting that open endpoint to a closed one is based on predecessor and so it is precision-aware. If the test expected Interval[1.0, 4.0) we'd pass either way but they seem to mostly prefer closed form in test expectations.

[dehall]

Created https://jira.hl7.org/browse/FHIR-59229 and updated the skip reason text to explain a little more in b730916 (though I left it in "incorrect output")

[cmoesel]

Yeah, I guess it comes down to the question: "Is 3.95 a valid value in the interval Interval[1.0, 10.0] or does that interval only consist of the numbers 1.0, 1.1, 1.2, ..., 10.0?

I'm pretty sure the expression 3.95 in Interval[1.0, 10.0] evaluates to true (based on the CQL spec), implying it is in that interval. But at the same time, I think expand Interval[1.0, 10] (with no per) is probably { 1.0, 1.1, 1.2, ..., 10.0 } (although the spec isn't crystal clear on that, I think it's the most reasonable interpretation of "a per value will be constructed based on the coarsest precision of the boundaries of the intervals in the input set") -- which would imply it is not in that interval. Ugh.

Anyway, I think you laid out the problem well in the Jira. Thanks.

[dehall]

FYI, I updated the cql-tests again in b0b8131 after a couple of my PRs were merged, and we now have some instances of tests with a versionTo - see https://github.com/cqframework/cql-tests/pull/147/changes . In the interest of moving this along quickly, for now I just explicitly skipped those tests targeting <= 1.5.3 in a new section in skip-list.txt, rather than add generalized support for skipping "legacy version" tests. I have some other ideas for spec-test generation that I'd also like to implement at some point so I'll add "version awareness" to that list

[cmoesel]

Thanks. I think this is pretty much ready to go. Just two more things to do:

1. Rebase on main and resolve any conflicts.
2. Run npm run check:all and check in the changed lock files in the sub-projects.

Then we can merge and release a CQL 4.0 beta!

[dehall]

Rebased, and updated package-lock files in a061f94