Insert/Upsert interfaces lose dynamic fields? about milvus-sdk-java HOT 9 CLOSED

InsertParam has tow methods to input data:
withFields(List<InsertParam.Field> fields) ------------------------ this is column-based data
withRows(List rows) -------------------------------- this is row-based data

For column-based data, currently it doesn't support inputting dynamic values. This is an enhancement we can implement in future versions.

This example shows how to use withRows() to input dynamic values:

from milvus-sdk-java.

Even we do the enhancement, it is inconvenient for users in some cases.
For example, assume we have a collection with id field, vector field and dynamic filed, if we want to insert some rows like this:

{"id": 1, "vector": [v1], "desc_1": 1}
{"id": 2, "vector": [v2], "desc_2": 2}
{"id": 3, "vector": [v3], "desc_3": 3}
{"id": 4, "vector": [v4], "desc_4": 4}
{"id": 5, "vector": [v5], "desc_5": 5}
......
{"id": N, "vector": [vN], "desc_N": N}

How to represent the rows in column-based data? The user needs to construct N lists to hold the data:

field("id"):  [1, 2, 3, 4, 5, ...... ,N]
field("vector"):  [v1, v2, v3, v4, v5, ...... ,vN]
field("desc_1"):  [1, None, None, None,  ......, None]
field("desc_2"):  [None, 2, None, None,  ......, None]
field("desc_3"):  [None, None, 3, None,  ......, None]
......
field("desc_N"):  [None, None, None, None,  ......, N]

from milvus-sdk-java.

Even we do the enhancement, it is inconvenient for users in some cases. For example, assume we have a collection with id field, vector field and dynamic filed, if we want to insert some rows like this:

{"id": 1, "vector": [v1], "desc_1": 1}
{"id": 2, "vector": [v2], "desc_2": 2}
{"id": 3, "vector": [v3], "desc_3": 3}
{"id": 4, "vector": [v4], "desc_4": 4}
{"id": 5, "vector": [v5], "desc_5": 5}
......
{"id": N, "vector": [vN], "desc_N": N}

How to represent the rows in column-based data? The user needs to construct N lists to hold the data:

field("id"):  [1, 2, 3, 4, 5, ...... ,N]
field("vector"):  [v1, v2, v3, v4, v5, ...... ,vN]
field("desc_1"):  [1, None, None, None,  ......, None]
field("desc_2"):  [None, 2, None, None,  ......, None]
field("desc_3"):  [None, None, 3, None,  ......, None]
......
field("desc_N"):  [None, None, None, None,  ......, N]

Thanks for your reply.
In my opinion, generally the column based insert/upsert should be more efficient than the row based, even though Milvus are using the same InsertRequest as the underlying buffer, therefor that may be the reason Python can insert/upsert dynamic fields in columnar batch in Your Official Sample Code Pieces.

So I am confused why the columnar interfaces in JAVA would throw out the dynamic fields in quiet, rather than trying to construct JSON fields or throw an exception, as that to do in Python.

Thanks. :)

from milvus-sdk-java.

In fact, all the official sample code to insert dynamic values are row-based:
https://milvus.io/docs/dynamic_schema.md#Insert-dynamic-data
https://github.com/milvus-io/pymilvus/blob/c4a05a71c0b2a84463d747bbb86ec2b9eac32157/examples/dynamic_field.py#L47

Python sdk doesn't support insert dynamic values in column-based.

from milvus-sdk-java.

In fact, all the official sample code to insert dynamic values are row-based: https://milvus.io/docs/dynamic_schema.md#Insert-dynamic-data https://github.com/milvus-io/pymilvus/blob/c4a05a71c0b2a84463d747bbb86ec2b9eac32157/examples/dynamic_field.py#L47

Python sdk doesn't support insert dynamic values in column-based.

Whatever, it's worthy to implement this feature in Java sdk? I don't think it's a difficult thing, of which code could be completed in ParamUtil:

        private void checkAndSetColumnData(List<FieldType> fieldTypes, List<InsertParam.Field> fields,
            boolean dynamicFieldEnabled) {
            Set<String> fieldsFound = new HashSet<>();
            // gen fieldData
            // make sure the field order must be consisted with collection schema
            for (FieldType fieldType : fieldTypes) {
                boolean found = false;
                for (InsertParam.Field field : fields) {
                    if (field.getName().equals(fieldType.getName())) {
                        if (fieldType.isAutoID()) {
                            String msg = "The primary key: " + fieldType.getName()
                                + " is auto generated, no need to input.";
                            throw new ParamException(msg);
                        }
                        checkFieldData(fieldType, field);

                        found = true;
                        fieldsFound.add(field.getName());
                        this.addFieldsData(genFieldData(field.getName(), fieldType.getDataType(), field.getValues()));
                        break;
                    }

                }
                if (!found && !fieldType.isAutoID()) {
                    String msg = "The field: " + fieldType.getName() + " is not provided.";
                    throw new ParamException(msg);
                }

            }
            // deal with dynamicField
            if (dynamicFieldEnabled) {
                List<InsertParam.Field> dynamicFields = fields.stream()
                    .filter(f -> !fieldsFound.contains(f.getName()) && !f.getValues().isEmpty())
                    .collect(ImmutableList.toImmutableList());
                Optional<InsertParam.Field> firstDynamicField = dynamicFields.stream().findFirst();
                if (firstDynamicField.isPresent()) {
                    io.milvus.param.ParamUtils.InsertDataInfo insertDynamicDataInfo =
                        io.milvus.param.ParamUtils.InsertDataInfo.builder()
                            .dataType(DataType.JSON)
                            .data(new LinkedList<>())
                            .build();
                    int dynamicFieldDataSize = firstDynamicField.get().getValues().size();
                    for (int i = 0; i < dynamicFieldDataSize; i++) {
                        JSONObject jsonObject = new JSONObject();
                        for (InsertParam.Field f : dynamicFields) {
                            jsonObject.put(f.getName(), f.getValues().get(i));
                        }
                        insertDynamicDataInfo.getData().add(jsonObject);
                    }

                    this.addFieldsData(genFieldData(
                        insertDynamicDataInfo.getFieldName(), DataType.JSON, insertDynamicDataInfo.getData(), true));
                }
            }
        }

from milvus-sdk-java.

@fengguangyuan

any interest in contributing one

from milvus-sdk-java.

Let me think about it.

from milvus-sdk-java.

I submit a pr to support inserting dynamic values by column-based fields.
#674

After this commit is merged, user can insert dynamic values like this:

List<JSONObject> dynamics = new ArrayList<>();
for (long i = 0L; i < rowCount; ++i) {
            
    JSONObject dynamic = new JSONObject();
    dynamic.put("any_key", i);
    dynamic.put("dynamic_key", true);
    dynamics.add(dynamic);
}

List<InsertParam.Field> fieldsInsert = new ArrayList<>();
fieldsInsert.add(new InsertParam.Field(field1Name, ids));
fieldsInsert.add(new InsertParam.Field(field2Name, vectors));
fieldsInsert.add(new InsertParam.Field(Constant.DYNAMIC_FIELD_NAME, dynamics));

InsertParam insertColumnsParam = InsertParam.newBuilder()
                .withCollectionName(CollectionName)
                .withFields(fieldsInsert)
                .build();

from milvus-sdk-java.

Fixed in java sdk v2.3.2 and this version has been released, close this issue.

from milvus-sdk-java.