DSBulk 1.11.0 doesn't apply partition level TTL when migrating data with '-ttl true' about dsbulk HOT 4 CLOSED

Sounds good, thanks @paayers ! I'm going to close this issue out for now. We can re-open if new information suggests there's an actual behavioural bug here.

from dsbulk.

Based on a quick analysis I believe this behaviour is consistent with the docs for 1.11.0 @paayers. The current docs indicate that -ttl sets the cell-level time-to-live value which is exactly what you see in your first example. I'll also note that the mapping process defined there (i.e. something of the form "ttl(colname)") is exactly what is used in the data file supplied to the first example. Finally I note that this flag is ignored if -query is supplied which (presumably) is why you aren't seeing cell-level values in your second example.

In the second example you're explicitly using "USING TIMESTAMP... AND TTL" in the CQL for that query. Apparently the CQL commands are setting row-level TTLs rather than a cell-specific value... which makes sense.

I'm unaware of any change in 1.11.0 which would make this happen but I'll double-check that and make sure I didn't miss anything there. That said, if the user is asking whether this behaviour has changed between an older version and the current one it would be useful to know what older version they observed the differing behaviour on.

from dsbulk.

I think I have a more detailed explanation for you @paayers which might help explain things to your customer.

In your first example above the customer includes explicitly specified writetime and TTL values for at least some vars via some of the fields in their CSV file. Let's take "version_key" as a simple example; note that the CSV snippet in the first example above (what I'm calling the "Failing" example) includes this string:

|version_key|writetime(version_key)|ttl(version_key)

These correspond to per-column writetime and TTL values as discussed in the dsbulk settings docs. After some digging it finally became clear to me that when dsbulk is presented with column-specific writetime or TTL values it actually modifies the CQL it's using to insert values. In these cases it creates a distinct INSERT statement for each column with the column-specific writetime and/or TTL values supplied via the usual "USING" keyword. For reference the relevant section of the code is here.

A quick look at git history suggests that this code came in with this commit and was first included as part of the 1.8.0 release. I'm not sure which older version the customer is comparing 1.11.0 to but if it's older than 1.8.0 then there likely is a change in behaviour from what they expect.

from dsbulk.

Perfect, thanks for the explanation on this, we should be able to modify the inserts accordingly to resolve this one

from dsbulk.