When you add any value to a Processor, the added value will get wrapped with a data type like PlainScalarString.

Is there a way around this and keep the original data type?

I was kind of hacking it like

Nodes.make_new_node = lambda source_node, value, value_format, **kwargs: value

The current implementation has 2 disadvantages this way:

• if you add a value, rualmel cannot serialize this config due to the wrappers
• inconsistent to the values that are already there in the input, those will have primitive types

I tried using your module to manage yaml files. However there is no place that shows how to do that. I was able to create the processor but after that what? How do I access the elements? There is no examples in the documentation that shows how to proceed after creating the processor

Is your feature request related to a problem? Please describe.
Collectors are described as being virtual data-sets pulled from potentially disparate parts of the source YAML data. In this light, it makes sense that the results of Collectors cannot be modified; changing a virtual data set wouldn't necessarily affect the source/concrete data. However, there should be a way to allow relaying changes to virtual data back to the concrete data, particularly now that NodeCoords are being used to track every element in the document during processing.

Describe the solution you'd like
Enable use of yaml-set to change the value of leaf nodes gathered via Collectors. At present, an error is emitted, instead.

Describe alternatives you've considered
Targeting node for change without Collectors is certainly possible in many cases, notably with Search Expressions and Wildcards. However, the reason that Collectors cannot also change nodes appears surmountable.

Operating System

1. Name/Distribution: Pop OS
2. Version: 22.04

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython
2. Python Version: 3.10.4
3. Version of yamlpath installed: 3.6.4
4. Version of ruamel.yaml installed: 0.17.17

Minimum sample of YAML (or compatible) data necessary to trigger the issue

{
  "People": [
    {
      "name": "Andy",
      "hasChildren": "True",
      "hasPet": "True"
    },
    {
      "name": "Fred",
      "hasChildren": "False",
      "hasPet": "False"
    }
  ]
}

Complete steps to reproduce the issue:

#!/usr/bin/env python3
from types import SimpleNamespace

from yamlpath import Processor as YPProcessor
from yamlpath.common import Parsers as YPParsers
from yamlpath.enums import YAMLValueFormats
from yamlpath.wrappers import ConsolePrinter as YPConsolePrinter


def str_to_bool(value: str) -> bool:
    if value.lower() in ('true', 'yes', 'y', 'on'):
        return True
    if value.lower() in ('false', 'no', 'n', 'off'):
        return False
    raise ValueError


def cast_json(json: str, bools: list[str]) -> dict | list:
    log = YPConsolePrinter(SimpleNamespace(quiet=True, verbose=False, debug=False))
    doc, success = YPParsers.get_yaml_data(YPParsers.get_yaml_editor(), log, json, literal=True)
    if not success:
        raise ValueError
    processor = YPProcessor(log, doc)

    for query_path in bools:
        matches = processor.get_nodes(query_path)
        for match in matches:
            processor.set_value(
                match.path, str_to_bool(match.node), value_format=YAMLValueFormats.BOOLEAN
            )

    return doc


if __name__ == '__main__':
    data = cast_json(
        '{"People": [{"name": "Andy", "hasChildren": "True", "hasPet": "True"}, {"name": "Fred", "hasChildren": "False", "hasPet": "False"}]}',
        ['/People/hasChildren', '/People/hasPet'],
    )
    print(f"{data=}")

Expected Outcome

I can't tell if this is working as expected and I'm failing to understand the docs, or if this is unexpected.

I'm taking in json or yaml data in which all the non-list, non-hash values are strings. I'd like to end up with a dict equivalent wherein some items are True or False.

I expected the code above to print

data=ordereddict([('People', [ordereddict([('name', 'Andy'), ('hasChildren', True), ('hasPet', True)]), ordereddict([('name', 'Fred'), ('hasChildren', False), ('hasPet', False)])])])

Actual Outcome

data=ordereddict([('People', [ordereddict([('name', 'Andy'), ('hasChildren', 1), ('hasPet', 1)]), ordereddict([('name', 'Fred'), ('hasChildren', 0), ('hasPet', 0)])])])

Operating System

1. Name/Distribution: CentOS
2. Version: pre-7.7

Version of Python and packages in use at the time of the issue.

1. Distribution: Python 3.6 from EPEL
2. Python Version: Python 3.6.8
3. Version of yamlpath installed: 2.3.0
4. Version of ruamel.yaml installed: 0.16.5

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
somestring:
  string:
    someotherstring: true

otherstring:
  default:
    config:
      deploy:
        me: true

Complete steps to reproduce the issue when triggered via:

from yamlpath.exceptions import YAMLPathException
from yamlpath.func import get_yaml_data, get_yaml_editor
from yamlpath import Processor
from yamlpath.wrappers import ConsolePrinter
from ruamel.yaml import YAML

class Args:
  debug = False
  verbose = False
  quiet = True

instance = YAML()

instance.default_flow_style = False
instance.explicit_start = True

yaml = get_yaml_editor()

log = ConsolePrinter(Args())
yaml_data = get_yaml_data(yaml, log, "/tmp/input.yaml")

processor = Processor(log, yaml_data)
try:
    processor.set_value("otherstring.default.config.deploy.me", "set_value")
except YAMLPathException as ex:
    log.critical(ex, 119)

with open("/tmp/output.yaml", 'w') as yaml_dump:
       instance.dump(yaml_data, yaml_dump)

Expected Outcome

---
somestring:
  string:
    someotherstring: true

otherstring:
  default:
    config:
      deploy:
        me: set_value

Actual Outcome

---
somestring:
  string:
    someotherstring: set_value

otherstring:
  default:
    config:
      deploy:
        me: set_value

Is your feature request related to a problem? Please describe.
Problem - yaml-set removes indentation on large JSON files

yaml-set 3.7.0

test.json

{
    "people": {
        "name": "john",
        "age": 30,
        "state": {"name": "up"}
      }
 }

yaml-set --mustexist --change=people.name --value="John Doe" test.json
{"people": {"name": "John Doe", "age": 30, "state": {"name": "up"}}}

Describe the solution you'd like
yaml-set to retain JSON indent while performing operation.

Should be one-line change to add indent
https://github.com/wwkimball/yamlpath/blob/master/yamlpath/commands/yaml_set.py#L301
json.dump(Parsers.jsonify_yaml_data(yaml_data), out_fhnd, indent=4)

Describe alternatives you've considered
pprintjson python package can be used, but needs tmp copy and can not perform an in-place replace.

Operating System

1. Name/Distribution: ALL
2. Version: ALL

Version of Python and packages in use at the time of the issue.

1. Distribution: ALL
2. Python Version: ALL
3. Version of yamlpath installed: 3.6.4
4. Version of ruamel.yaml installed: ANY supported

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
# test.yaml
key:  ENC[BLAH]

# yaml-get --query='(/key) ' test.yaml
ENC[BLAH]

Assuming EYAML was installed and the keys were in one of the default locations, the output should have been decrypted but it never is.

Complete steps to reproduce the issue when triggered via:

1. Command-Line Tools (yaml-get, yaml-set, or eyaml-rotate-keys): Precise command-line arguments which trigger the defect.
2. Libraries (yamlpath.*): Minimum amount of code necessary to trigger the defect.

Expected Outcome

Decrypted EYAML value from Collector results.

Actual Outcome

yaml-get prints out the encrypted strings.

Screenshot(s), if Available

Hello,

documentation mentions it's easy to implement a logger that is API compatible to the consoleprinter and uses standard python logging.

I wonder if yamlpath ships with such a logging alternative? Re-implementing is still tedious but required for using it as a library that uses the standard python logger.

Thanks!

Operating System

1. Name/Distribution: ANY
2. Version: ANY

Version of Python and packages in use at the time of the issue.

1. Distribution: ANY
2. Python Version: ANY Supported
3. Version of yamlpath installed: 3.6.4
4. Version of ruamel.yaml installed: ANY Supported

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
# test.yaml
key1: value1

# Pre-comment
remove_this: key

# Post-comment
keep_this: other key

Complete steps to reproduce the issue when triggered via:

1. yaml-set --delete --change=/remove_this test.yaml

Expected Outcome

---
# test.yaml
key1: value1

# Post-comment
keep_this: other key

Actual Outcome

---
# test.yaml
key1: value1

# Pre-comment
keep_this: other key

As you can see, the Pre-comment was preserved while both the post-comment and all whitespace after the target key was deleted.

Screenshot(s), if Available

Operating System

1. Name/Distribution: KDE Neon (Ubuntu)
2. Version: 20.04

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython installed using asdf-vm
2. Python Version: 3.8.5
3. Version of yamlpath installed: N/A, trying to install 3.x
4. Version of ruamel.yaml installed: N/A

Complete steps to reproduce the issue when triggered via:

1. Install yamlpath in a fresh/new virtual environment with the following commands

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install yamlpath

Expected Outcome

yamlpath gets installed successfully

Actual Outcome

Install fails with the following error:

Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pddralma/yamlpath/setup.py", line 3, in <module>
        from yamlpath.common import YAMLPATH_VERSION
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/__init__.py", line 2, in <module>
        from yamlpath.yamlpath import YAMLPath
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/yamlpath.py", line 9, in <module>
        from yamlpath.types import PathSegment
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/types/__init__.py", line 2, in <module>
        from .pathattributes import PathAttributes
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/types/pathattributes.py", line 4, in <module>
        from yamlpath.path import CollectorTerms
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/path/__init__.py", line 2, in <module>
        from .collectorterms import CollectorTerms
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/path/collectorterms.py", line 6, in <module>
        from yamlpath.enums import CollectorOperators
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/enums/__init__.py", line 8, in <module>
        from .yamlvalueformats import YAMLValueFormats
      File "/tmp/pip-install-pddralma/yamlpath/yamlpath/enums/yamlvalueformats.py", line 9, in <module>
        from ruamel.yaml.scalarstring import (
    ModuleNotFoundError: No module named 'ruamel'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The error does not occur if ruamel.yaml is installed first using pip install ruamel.yaml.

My guess is that this comes from the following import in setup.py

from yamlpath.common import YAMLPATH_VERSION

Since the imports are resolved before setup_tools installs the packages listed in install_requires

Summary

If I pin yamlpath to latest version 3.8.0 alongside ruamel.yaml 0.17.32 I get
yamlpath 3.7.0 depends on ruamel.yaml!=0.17.18, <=0.17.21 and >0.17.5

This has been this way for sometime now - any updates as to when next version of yamlpath will be compat w latest ruamel.yaml?
Reason this is important is that there is a bug in older version or ruamel.yaml related to processing dates which causes unexpected exceptions.

The workaround is to pin ruamel yaml to latest and unpin yamlpath version and let pip figure things out.
However this is clunky if you need to pin versions.

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.10
3. Version of yamlpath installed: latest 3.8.0
4. Version of ruamel.yaml installed: latest 0.17.32

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version:
   3.9.1
3. Version of yamlpath installed:
   3.6.1
4. Version of ruamel.yaml installed:
   0.17.10

Minimum sample of YAML (or compatible) data necessary to trigger the issue

yaml with cyrillic chars:

# Переменные
variables:

try to call yaml-diff:

Traceback (most recent call last):
  File ".\lib\yamlpath\commands\yaml_diff.py", line 279, in main
    (lhs_docs, lhs_loaded) = get_docs(log, lhs_yaml, lhs_file)
  File ".\lib\yamlpath\commands\yaml_diff.py", line 239, in get_docs
    for (yaml_data, doc_loaded) in Parsers.get_yaml_multidoc_data(
  File ".\lib\yamlpath\common\parsers.py", line 231, in get_yaml_multidoc_data
    for document in parser.load_all(fhnd):
  File ".\lib\ruamel\yaml\main.py", line 458, in load_all
    constructor, parser = self.get_constructor_parser(stream)
  File ".\lib\ruamel\yaml\main.py", line 483, in get_constructor_parser
    self.reader.stream = stream
  File ".\lib\ruamel\yaml\reader.py", line 133, in stream
    self.determine_encoding()
  File ".\lib\ruamel\yaml\reader.py", line 193, in determine_encoding
    self.update_raw()
  File ".\lib\ruamel\yaml\reader.py", line 288, in update_raw
    data = self.stream.read(size)
  File ".\lib\encodings\cp1251.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 111: character maps to <undefined>

after this:

def get_yaml_multidoc_data(
...
with open(source, 'r) as fhnd: -> with open(source, 'r', encoding='utf-8') as fhnd:

all work fine.

Operating System

1. Name/Distribution: Ubuntu
2. Version: 18.04.5 LTS (Bionic Beaver)

Version of Python and packages in use at the time of the issue.

1. Distribution: Anaconda
2. Python Version: 3.8.12
3. Version of yamlpath installed: 3.6.6
4. Version of ruamel.yaml installed: 0.17.21

Minimum sample of YAML (or compatible) data necessary to trigger the issue

import yamlpath

Complete steps to reproduce the issue when triggered via:

conda create -y -n test python==3.8.12
conda activate test
pip install yamlpath==3.6.6
python -c "import yamlpath"

Expected Outcome

Actual Outcome

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/__init__.py", line 5, in <module>
    from yamlpath.yamlpath import YAMLPath
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/yamlpath.py", line 9, in <module>
    from yamlpath.types import PathAttributes, PathSegment
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/types/__init__.py", line 3, in <module>
    from .pathattributes import PathAttributes
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/types/pathattributes.py", line 8, in <module>
    from yamlpath.path import CollectorTerms
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/path/__init__.py", line 2, in <module>
    from .collectorterms import CollectorTerms
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/path/collectorterms.py", line 6, in <module>
    from yamlpath.enums import CollectorOperators
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/enums/__init__.py", line 9, in <module>
    from .yamlvalueformats import YAMLValueFormats
  File "/home/kani/.conda3/lib/python3.8/site-packages/yamlpath/enums/yamlvalueformats.py", line 23, in <module>
    from yamlpath.patches.timestamp import (
ModuleNotFoundError: No module named 'yamlpath.patches'

Screenshot(s), if Available

Operating System

1. Name/Distribution: Ubuntu (on WSL)
2. Version: 18.04.5 LTS

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.8.0
3. Version of yamlpath installed: 3.6.1
4. Version of ruamel.yaml installed: 0.16.12

Minimum sample of YAML (or compatible) data necessary to trigger the issue

Any file will do, here is a simple one :

---
test:
  key1: value1

Complete steps to reproduce the issue when triggered via:

yaml-get --query test.key test.yml

Expected Outcome

Actual Outcome

Traceback (most recent call last):
  File "/home/sr/.local/bin/yaml-get", line 5, in <module>
    from yamlpath.commands.yaml_get import main
  File "/home/sr/.local/lib/python3.8/site-packages/yamlpath/__init__.py", line 6, in <module>
    from yamlpath.processor import Processor
  File "/home/sr/.local/lib/python3.8/site-packages/yamlpath/processor.py", line 18, in <module>
    from yamlpath.common import Anchors, KeywordSearches, Nodes, Searches
  File "/home/sr/.local/lib/python3.8/site-packages/yamlpath/common/__init__.py", line 4, in <module>
    from .parsers import Parsers
  File "/home/sr/.local/lib/python3.8/site-packages/yamlpath/common/parsers.py", line 25, in <module>
    from yamlpath.patches.aliasstyle import MySerializer # type: ignore
ModuleNotFoundError: No module named 'yamlpath.patches'

It seems that "patches" is not included in the release's zip or tar.gz assets.

Screenshot(s), if Available

Is your feature request related to a problem? Please describe.
Having read #56 I see the complexity of modifying a nested array/hash data structure, and why it's left to yaml-merge.
However, I humbly think some simple use cases could be brought to yaml-set instead of necessitating to build a full document for a merge: E.g. appending a new element to existing arrays (without knowing how many elements are already), and possibly creating arrays to append to, too.

Describe the solution you'd like
Indexing [-1] yields the last element, as expected, so could be used for creating a new array with the given element, but not append to an existing.
AIUI indexing [*] is not yet used and could be for this purpose, alternatively [+].

Describe alternatives you've considered
The current way is to build a full yaml document, and then yaml-merge the two. According to the docs, this has the drawback of stripping all comments and empty lines.

Additional context
yamlpath version 3.6.1

Would be really useful to get the actual path of a node. Something like this:

for node in processor.get_nodes(yamlpath):
    print(node.fullpath)     # returns the path that points to this node: /a/b/c[0]

Is your feature request related to a problem? Please describe.

Newer versions of ruamel.yaml have come out since the last release of yamlpath. Please support them.

Describe the solution you'd like

Newer versions of ruamel.yaml are utilized by yamlpath.

Describe alternatives you've considered

Sticking with old -- still available -- versions of ruamel.yaml.

Is your feature request related to a problem? Please describe.
When editing large YAML files, it's sometimes difficult to determine the full path of the line you're editing. In some cases it's very common to be editing a value from a key that's not the one you want.

Describe the solution you'd like
Using the yaml-paths command, I'd like to be able to pass a specific line number instead of a search expression. The output should be the YAML path of this specific line.

Describe alternatives you've considered
I've considered using search expressions looking for the content of the line, but this solution is prone to fail.

Additional context
My intention is to use this feature with nvim to always show the YAML path of the current line being edited.

Is your feature request related to a problem? Please describe.

It will be great if the tool can support array notation [] like below

department.employees[0].name
department.employees[0].age
department.employees[1].name
department.employees[1].age

Describe the solution you'd like

Here are 2 different yaml files

yaml-1

a:
  - hello
  - world

yaml-2

a:
  0: hello
  1: world

I run below command on both of the above yamls

yaml-paths --nofile --expand --keynames --noescape --values --search='=~/.*/' yaml-1.yaml
yaml-paths --nofile --expand --keynames --noescape --values --search='=~/.*/' yaml-2.yaml

I get same output for both yaml files

a.0: hello
a.1: world

I think the correct output for yaml-1 is:-

a[0]: hello
a[1]: world

Describe alternatives you've considered

couldn't find any alternate solution

Additional context

none

Operating System

1. Name/Distribution: OracleLinux
2. Version: 7.7

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.6.8
3. Version of yamlpath installed: yamlpath==2.2.0
4. Version of ruamel.yaml installed: ruamel.yaml==0.16.5

Minimum sample of YAML (or compatible) data necessary to trigger the issue

N/A

Complete steps to reproduce the issue when triggered via:

import warnings
warnings.simplefilter("ignore")
#call get_yaml_data
get_yaml_data(.....)

any warning should be ignored but exception is thrown

warnings.warn(......

Expected Outcome

Warning ignored

Actual Outcome

Exception thrown

Screenshot(s), if Available

Operating System

1. Name/Distribution: Windows 10 Home
2. Version: 10.0.19043 Build 19043

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython (for Windows) from python.org
2. Python Version: 3.7
3. Version of yamlpath installed: 3.6.3
4. Version of ruamel.yaml installed: 0.17.10

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
Locations:
    United States:
        New York:
        Boston:
    Canada:

Complete steps to reproduce the issue when triggered via:

1. Command-Line Tools (yaml-get, yaml-set, or eyaml-rotate-keys): Precise command-line arguments which trigger the defect.
2. Libraries (yamlpath.*): Minimum amount of code necessary to trigger the defect.

#I thought that a complete unittest might be the most helpful way to demonstrate my issue. Please let me know if another format would be more helpful.

import unittest
import yamlpath
from yamlpath.wrappers import ConsolePrinter
from yamlpath.common import Parsers
from yamlpath import Processor
from yamlpath.exceptions.yamlpathexception import YAMLPathException
from types import SimpleNamespace

class IssueReportTest(unittest.TestCase):
    
    def setUp(self):
        pass
    
    def tearDown(self):
        pass
    
    def test_retrieveGrandChildren_OnlyGrandChildrenAreReturned(self):
        yamlTagHierarchy = '''---
        Locations:
            United States:
                New York:
                Boston:
            Canada:
        '''
        
        logging_args = SimpleNamespace(quiet=True, verbose=False, debug=False)
        self._log = ConsolePrinter(logging_args)
        self._editor = Parsers.get_yaml_editor()
        (yaml_data, doc_loaded) = Parsers.get_yaml_data(self._editor, self._log, yamlTagHierarchy, literal=True)
        self._processor = Processor(self._log, yaml_data)
        
        nodes = list(self._processor.get_nodes("/Locations/*/*"))
        self.assertEqual(nodes[0].parentref, "New York")
        self.assertEqual(nodes[1].parentref, "Boston")
        self.assertEqual(len(nodes), 2, f"Node '{nodes[2].parentref}' should not be part of this list, or?")

Expected Outcome

When I try to select a specific level of descendant nodes using child and wildcard operators I expect to receive only nodes at the requested level. For example, in the above sample I expect "/Locations//" to return "New York" and "Boston" (grandchildren of "Locations")

Actual Outcome

If another branch of the yaml tree ends above the requested level, the query returns the last leaf on that branch. The above example returns "Canada" in addition to "New York" and "Boston", which is surprising to me as "Canada" is merely a child of "Location", while "New York" and "Boston" are grandchildren. I haven't been able to identify an easy way to distinguish the child from the grandchild nodes.

Thank you

Thanks so much for considering this. I was thrilled to find yamlpath for a hobby project and really appreciate the library. I hope that I'm actually reporting a real issue rather than flaunting my ignorance of how the wildcard operator should work.

Operating System

1. Name/Distribution: Windows
2. Version: 10

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython
2. Python Version: 3.x
3. Version of yamlpath installed: 3.6.7
4. Version of ruamel.yaml installed: 0.17.21

Minimum sample of YAML (or compatible) data necessary to trigger the issue

test_commands_yaml_set.py

    def test_update_base_value(self, script_runner, tmp_path_factory):
        yamlin = """---
common: &base
  TEST_COMMON_SETTING: 'a1'
TestService:
  <<: *base
"""
        yamlout = """---
common: &base
  TEST_COMMON_SETTING: a2
TestService:
  <<: *base
  TEST_COMMON_SETTING: a2
"""
        yaml_file = create_temp_yaml_file(tmp_path_factory, yamlin)
        result = script_runner.run(
            self.command,
            "--change=common.TEST_COMMON_SETTING",
            "--value=a2",
            yaml_file
        )
        assert result.success, result.stderr

        with open(yaml_file, 'r') as fhnd:
            filedat = fhnd.read()
        assert filedat == yamlout

Expected Outcome

---
common: &base
  TEST_COMMON_SETTING: a2
TestService:
  <<: *base

if you change TEST_COMMON_SETTING: 'a1' -> TEST_COMMON_SETTING: a1 then everything is ok.

looks like bug in check:

def recurse(data, parent, parentref, reference_node, replacement_node):
...
    if (hasattr(val, "anchor"):

value with type SingelQuotedScalarString has an attr anchor.

I'm bypassing the usual bug report template for this because I have yet to witness the issue myself. I'm opening this as an investigation.

The original reporter was adamant that this happened, but they were unable to provide sample YAML data to reproduce the issue. All I could gather from the conversation was that something like this:

---
# lhs.yaml
- uno
- dos

---
# rhs.yaml
- dos
- dos

Somehow produces:

---
# output.yaml
- uno

During the conversation, the reporter couldn't remember what options they passed to yaml-merge to trigger this behavior. They were nonetheless adamant that the dos value was expected but was unexpectedly deleted.

Out of an abundance of care (nobody wants to lose data during a merge), and respect for anyone who reports an issue, I'm logging this is a possible bug warranting investigation.

Is your feature request related to a problem? Please describe.
YAML Merge Keys are extremely useful for eliminating structure and data duplication within YAML files. At present, these are understood and preserved during all get/set/merge operations. However, there is presently no means within yamlpath's command-line tools of creating a new YAML Merge Key within a file which does not already have one.

Describe the solution you'd like
A command like the following should be possible: yaml-set --change=/some/hierarchy/targets/* --mergekey=/merge_source --anchor=new_anchor_name

This could change:

---
merge_source:
  hash: with
  some: default
  reusable: values
some:
  hierarchy:
    targets:
      one:
        novel: structure
      two:
        novel: data
      three:
        overrides: defaults
        some: non-default

into:

merge_source: &new_anchor_name
  hash: with
  some: default
  reusable: values
some:
  hierarchy:
    targets:
      one:
        <<: *new_anchor_name
        novel: structure
      two:
        <<: *new_anchor_name
        novel: data
      three:
        <<: *new_anchor_name
        overrides: defaults
        some: non-default

Describe alternatives you've considered
I have considered another request which asks for the ability to share such default configuration by way of applying novel key-value pairs to change targets only when the specified key does not already exist. While this alternative would solve the topical issue, it does not employ the far more useful YAML Merge Keys, which can be used to set more than one default and facilitate changing defaults later as needed via a single data-point. This helps preserve the discipline of, "One version of the truth."

Hi

When trying to update a negative float value, we get a positive value with an additional leading zero, e.g. 00.9 instead of -0.9.

Steps to reproduce below.

Operating System

1. Name/Distribution: Linux
2. Version:

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.7.1
3. Version of yamlpath installed: 2.3.6
4. Version of ruamel.yaml installed: 0.16.10

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
parameters:
  param1: 0.1
  param2: 0.9

Complete steps to reproduce the issue when triggered via:

1. Command-Line Tools (yaml-get, yaml-set, or eyaml-rotate-keys): Precise command-line arguments which trigger the defect.
2. Libraries (yamlpath.*): Minimum amount of code necessary to trigger the defect.

from yamlpath.func import get_yaml_data, get_yaml_editor
from yamlpath import Processor
from yamlpath.wrappers import ConsolePrinter
from ruamel.yaml import YAML

class Args:
  debug = False
  verbose = False
  quiet = True

file = 'test.yaml'
param = '/parameters/param2'
val = -0.9
 
args = Args()
log = ConsolePrinter(args)
 
instance = YAML()
instance.default_flow_style = False
instance.explicit_start = True
 
yaml = get_yaml_editor()
yaml_editor = get_yaml_editor()
yaml_contents = get_yaml_data(yaml_editor, log, file)
 
processor = Processor(log, yaml_contents)
processor.set_value(param, val, value_format= YAMLValueFormats.FLOAT)
 
with open(file, 'w') as cfile:
    yaml_editor.dump(yaml_contents, cfile)

Expected Outcome

parameters:
  param1: 0.1
  param2: -0.9

Actual Outcome

parameters:
  param1: 0.1
  param2: 00.9

Screenshot(s), if Available

Thank you for this project, it looks very promising! :)

Is your feature request related to a problem? Please describe.

I am considering using your library in a project that has been using PyYAML. This means that it's only YAML 1.1-compliant and I would like to not change that for maximum backward compatibility (at least for now). However when I use

Describe the solution you'd like

Allow passing extra params to ruamel.yaml.

Operating System

1. Name/Distribution: ANY
2. Version: ANY

Version of Python and packages in use at the time of the issue.

1. Distribution: ANY supported
2. Python Version: ANY supported
3. Version of yamlpath installed: 3.6.4
4. Version of ruamel.yaml installed: ANY supported

Minimum sample of YAML (or compatible) data necessary to trigger the issue

---
# lhs.yaml
aliases:
  - &name_a a_name
  - &name_b b_name

ref_hash: &ref_hash
  *name_a : {}
  *name_b : {}

imp_hash_a:
  <<: *ref_hash

imp_hash_b:
  <<: *ref_hash

---
# rhs.yaml
aliases:
  - &name_a NEW_A_VALUE
  - &name_b NEW_B_VALUE

Complete steps to reproduce the issue when triggered via:

1. yaml-merge --anchors=left lhs.yaml rhs.yaml

Expected Outcome

---
# lhs.yaml
aliases:
  - &name_a NEW_A_VALUE
  - &name_b NEW_B_VALUE

ref_hash: &ref_hash
  *name_a : {}
  *name_b : {}

imp_hash_a:
  <<: *ref_hash

imp_hash_b:
  <<: *ref_hash

Actual Outcome

---
# lhs.yaml
aliases:
  - &name_a a_name
  - &name_b b_name

ref_hash:
  *name_a : {}
  *name_b : {}

imp_hash_a:
  anchor01: &ref_hash
    NEW_A_VALUE : {}
    NEW_B_VALUE : {}

imp_hash_b:
  <<: *anchor01
  NEW_A_VALUE : {}
  NEW_B_VALUE : {}

Note that the ref_hash anchor moves and gets redefined where the first reference formerly existed, then ignored for the second reference.

[This is from memory and the detail of the actual outcome may be slightly different from that hand-typed above. The point is, the merge causes a a redefinition of the original anchor and an "explosion" of its aliases.]

Great project, thanks for the hard work on it!

Is your feature request related to a problem? Please describe.
Currently, importing this library forces my projects to drop Python 3.10 support, which is a bit frustrating.

As Python 3.11 nears release (October 2022), it'd be good for this library to not fall too far behind.

Describe the solution you'd like
Any/all issues with Python 3.10 are identified and resolved, and a new minor version of yamlpath is tagged with Python 3.10 Support.

Describe alternatives you've considered
Remaining on older Python versions (not a long term solution), forking yamlpath.

Additional context
See upcoming PR for a non-exhaustive first attempt.

Is your feature request related to a problem? Please describe.
For python usage, it will be useful if the get_yaml_data implementation supports loading from string instead of requiring a file.

Describe the solution you'd like

Prefered solution is a simply API to load the yaml data, e.g.:

my_yaml_data = """
key:
  value
"""

from yamlpath import load

yaml_data = load(my_yaml_data)

Describe alternatives you've considered
Alternative, I can load directly using ruamel.yaml.

from ruamel.yaml import YAML

yaml_obj = YAML(typ='safe')
yaml_data = yaml_obj.load(my_yaml_data)

Is your feature request related to a problem? Please describe.
I'm using a Processor to replace some string values with other types. I can't seem to set a YAML date value using the Processor, but it does work using ruamel.yaml's CommentedMap's dict interface.

#!/usr/bin/env python3
from yamlpath import Processor as YPProcessor
from yamlpath.wrappers import ConsolePrinter as YPConsolePrinter
from yamlpath.common import Parsers as YPParsers
from types import SimpleNamespace
from json import dumps as jdumps
from datetime import date


if __name__ == '__main__':

    # Start with an untyped date
    data = {'when': '2022-09-23'}

    # Get a CommentedMap "doc" and Processor, "surgeon"
    YAML_EDITOR = YPParsers.get_yaml_editor()
    log = YPConsolePrinter(SimpleNamespace(quiet=True, verbose=False, debug=False))
    doc, success = YPParsers.get_yaml_data(YAML_EDITOR, log, jdumps(data), literal=True)
    surgeon = YPProcessor(log, doc)

    # Try replacing the string date with a yaml-friendly date object:

    print(doc)  # ordereddict([('when', '2022-09-23')])
    d = date.fromisoformat(doc['when'])
    print(d)    # datetime.date(2022, 9, 23)

    doc['when'] = d
    print(doc)  # ordereddict([('when', datetime.date(2022, 9, 23))])

    try:
        surgeon.set_value('/when', d)
    except TypeError:
        print("'datetime.date' object cannot be interpreted as an integer")

Describe the solution you'd like
I would like an example added to the wiki, of setting a given yamlpath node to a YAML date value.

Describe alternatives you've considered
Even though using doc[key] = d works, I'm getting arbitrary paths from a Processor.get_nodes result, and AFAIK Processor.set_value is the only way to modify data at arbitrary yaml paths.

Additional context

TypeError                                 Traceback (most recent call last)
----> 1 surgeon.set_value('/when', d)

File ~/.local/share/venvs/1be7b478d580cac73a8bca0e01d57a97/venv/lib/python3.10/site-packages/yamlpath/processor.py:187, in Processor.set_value(self, yaml_path, value, **kwargs)
    180 self.logger.debug(
    181     "Processor::set_value:  Seeking optional node at {}."
    182     .format(yaml_path)
    183 )
    184 for node_coord in self._get_optional_nodes(
    185     self.data, yaml_path, value
    186 ):
--> 187     self._apply_change(yaml_path, node_coord, value,
    188         value_format=value_format, tag=tag)

File ~/.local/share/venvs/1be7b478d580cac73a8bca0e01d57a97/venv/lib/python3.10/site-packages/yamlpath/processor.py:287, in Processor._apply_change(self, yaml_path, node_coord, value, **kwargs)
    284         return
    286 try:
--> 287     self._update_node(
    288         node_coord.parent, node_coord.parentref, value,
    289         value_format, tag)
    290 except ValueError as vex:
    291     raise YAMLPathException(
    292         "Impossible to write '{}' as {}.  The error was:  {}"
    293         .format(value, value_format, str(vex))
    294         , str(yaml_path)) from vex

File ~/.local/share/venvs/1be7b478d580cac73a8bca0e01d57a97/venv/lib/python3.10/site-packages/yamlpath/processor.py:2589, in Processor._update_node(self, parent, parentref, value, value_format, value_tag)
   2587 else:
   2588     change_node = parent[parentref]
-> 2589 new_node = Nodes.make_new_node(
   2590     change_node, value, value_format, tag=value_tag)
   2592 self.logger.debug(
   2593     f"Changing the following <{value_format}> formatted node:",
   2594     prefix="Processor::_update_node:  ",
   2595     data={ "__FROM__": change_node, "___TO___": new_node })
   2597 recurse(self.data, parent, parentref, change_node, new_node)

File ~/.local/share/venvs/1be7b478d580cac73a8bca0e01d57a97/venv/lib/python3.10/site-packages/yamlpath/common/nodes.py:167, in Nodes.make_new_node(source_node, value, value_format, **kwargs)
    165         new_node = new_type(new_value, anchor=source_node.anchor.value)
    166     elif new_type is not type(None):
--> 167         new_node = new_type(new_value)
    169 # Apply a custom tag, if provided
    170 if "tag" in kwargs:

Operating System

1. Name/Distribution: Ubuntu
2. Version: 20.10

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython
2. Python Version: 3.9.4
3. Version of yamlpath installed: 3.4.1
4. Version of ruamel.yaml installed: 0.17.4

Minimum sample of YAML (or compatible) data necessary to trigger the issue

Things:
  - name: first thing
    rank: 42
  - name: second thing
    rank: 5
  - name: third thing
    rank: null
  - name: fourth thing
    rank: 1

Complete steps to reproduce the issue when triggered via:

1. Load YAML document that has at least one leaf element explicitly set to null.

2. Preform a search using a YAML path that includes a deep traversal. The specified path does not necessarily have to match the leaf node(s) that are set to null. Example: /**/name or **.name.

3. Iterate over the returned results. An error is thrown once the first leaf node that is set to null is reached.

4. Command-Line Tools (yaml-get, yaml-set, or eyaml-rotate-keys): N/A

5. Libraries (yamlpath.*): Minimum amount of code necessary to trigger the defect.

#!/usr/bin/env python3
"""Test script."""
from os.path import dirname, join
from yamlpath.common import Parsers
from yamlpath.wrappers import ConsolePrinter
from yamlpath import Processor


class LogArgs:
    """Data object for configuring yamlpath logging."""


logargs = LogArgs()
setattr(logargs, "debug", False)
setattr(logargs, "verbose", True)
setattr(logargs, "quiet", False)


def main():
    """Print found YAML paths of a deep traversal search."""
    yaml_file = join(dirname(__file__), "bad.yaml")

    log = ConsolePrinter(logargs)
    yaml = Parsers.get_yaml_editor()
    (yaml_data, doc_loaded) = Parsers.get_yaml_data(yaml, log, yaml_file)

    if not doc_loaded:
        exit(1)

    processor = Processor(log, yaml_data)

    path = "/**/name"
    for node_coordinate in processor.get_nodes(path):
        log.info(f"{node_coordinate.path} = {str(node_coordinate.node)}")


if __name__ == "__main__":
    main()

Expected Outcome

The iteration of search results should complete successfully. If the leaf node(s) that were set to null matched the yamlpath search criteria, they should be returned with their values set to None. This was working as expected in yamlpath 3.4.0.

Actual Outcome

The following error is thrown:

$: ./test_script.py
Things[0].name = first thing
Things[1].name = second thing
Things[2].name = third thing
Traceback (most recent call last):
  File ".../test_script.py", line 38, in <module>
    main()
  File ".../test_script.py", line 33, in main
    for node_coordinate in processor.get_nodes(path):
  File ".../lib/python3.9/site-packages/yamlpath/processor.py", line 94, in get_nodes
    for opt_node in self._get_optional_nodes(
  File ".../lib/python3.9/site-packages/yamlpath/processor.py", line 1397, in _get_optional_nodes
    for node_coord in self._get_optional_nodes(
  File ".../lib/python3.9/site-packages/yamlpath/processor.py", line 1513, in _get_optional_nodes
    raise YAMLPathException(
yamlpath.exceptions.yamlpathexception.YAMLPathException: Cannot add PathSegmentTypes.KEY subreference to scalars at 'name' in '/**/name'.

yamlpath_bug.tar.gz

Screenshot(s), if Available

Operating System

1. Name/Distribution: macOS(tried on linux as well, same behaviour)
2. Version: 12.6

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.9.12
3. Version of yamlpath installed: 0.17.17
4. Version of ruamel.yaml installed: 3.6.8

Minimum sample of YAML (or compatible) data necessary to trigger the issue

from types import SimpleNamespace

from ruamel.yaml import YAML
from yamlpath.wrappers import ConsolePrinter
from yamlpath import Processor
from yamlpath.enums.yamlvalueformats import YAMLValueFormats


if __name__ == '__main__':
    log = ConsolePrinter(
        SimpleNamespace(quiet=True, verbose=False, debug=False)
    )
    config = {"key1": [1, 2, 3],
              "key2": {
                  "key3": "value1",
                  "key4": "value2",
              }
              }
    processor = Processor(log, config)
    processor.set_value("key2.key3", "asdf", value_format=YAMLValueFormats.DEFAULT)
    print(list(processor.get_nodes("key2.key3"))[0].node)

Complete steps to reproduce the issue when triggered via:

Running the script above

Expected Outcome

printing "asdf"

Actual Outcome

printing "value1"

Screenshot(s), if Available

I am not sure if this is the intended usage of this package. But I have a question about this change, why does it accepts both CommentedSeq, list but not dict?

Sorry if I am not using this package the intended way.

Is your feature request related to a problem? Please describe.
In an existing yaml file, keys may be missing for some of the items matched in a path. I would like to be able to set the values for the missing keys.

e.g.

devices:
  R1:
    os: ios
  R2:
    # os missing

Describe the solution you'd like
A way to select non-existing keys and set their value, e.g.

yaml-set -g "/devices/*/os" -a generic --mustnotexist testbed.yaml

Is your feature request related to a problem? Please describe.
Sometimes I need to set a value that may be an empty string.

Describe the solution you'd like
A note in the yaml-set and yaml-merge docs about properly passing the empty string as a value.

Describe alternatives you've considered

$ printf '%s\n' 'oldkey: "4"' >vars.yml
$ yaml-set -g .newkey -a '' vars.yml
ERROR:  Exactly one of the following must be set:  --value, --file, --stdin, or --random
Please try --help for more information.
$ yaml-merge -m .newkey vars.json - <<<''
$ # retcode 3
$ yaml-merge -m . vars.json - <<<'newkey: '
--- {"oldkey": "4", "newkey": !!null ''}

Hey there.
I can't get 2 simple yaml files merged with this error thrown:
ERROR: Impossible to add Scalar value, , to a Hash without a key. Change the value to a 'key: value' pair, a '{key: value}' Hash, or change the merge target to an Array or other Scalar value. This issue occurred at YAML Path: /
I'm using a command:
yaml-merge first.yaml second.yaml
first.yaml
replicaCount: 2
second.yaml
replicaCount: 3
Basically at this point I've tried to merge lot's of different yamls and always get the same error.
I've installed yamlPath with:
pip3 install yamlpath
Logs:

17-Jan-2021 17:02:16 | Downloading yamlpath-3.4.0.tar.gz (183 kB)
17-Jan-2021 17:02:17 | Collecting ruamel.yaml>=0.15.96
17-Jan-2021 17:02:17 | Downloading ruamel.yaml-0.16.12-py2.py3-none-any.whl (111 kB)
17-Jan-2021 17:02:17 | Building wheels for collected packages: yamlpath
17-Jan-2021 17:02:17 | Building wheel for yamlpath (setup.py): started
17-Jan-2021 17:02:18 | Building wheel for yamlpath (setup.py): finished with status 'done'
17-Jan-2021 17:02:18 | Created wheel for yamlpath: filename=yamlpath-3.4.0-py3-none-any.whl size=191879 sha256=bddb9fa117410c4ce366a7f59e4499037e45285fda782271c7ebcffe65dad516
17-Jan-2021 17:02:18 | Stored in directory: /tmp/.cache/pip/wheels/53/7d/0d/714e41a8161d84d39310352f4bc15ef1b2ceb8e14292edd3b4
17-Jan-2021 17:02:18 | Successfully built yamlpath
17-Jan-2021 17:02:18 | Installing collected packages: ruamel.yaml, yamlpath
17-Jan-2021 17:02:18 | Successfully installed ruamel.yaml-0.16.12 yamlpath-3.4.0
17-Jan-2021 17:02:19 | ERROR:  Impossible to add Scalar value, , to a Hash without a key.  Change the value to a 'key: value' pair, a '{key: value}' Hash, or change the merge target to an Array or other Scalar value.  This issue occurred at YAML Path:  /`

Do you know what could be the reason?

Originally posted by @Ndubovoi in #108 (comment)

Hello! i'm trying to use this library to read a yaml file and update some values in it.
tbh I'm struggling a bit with the doc, (would love to have a really simple example with no logger, console, etc..) and when i'm trying to read the file i get this error:

 File "/Users/luigi.tagliamonte/Projects/scripts/update_tier.py", line 58, in update_clusters
    (yaml_data, doc_loaded) = get_yaml_data(yaml, None, rel_file_path)
  File "/usr/local/lib/python3.8/site-packages/yamlpath/func.py", line 25, in get_yaml_data
    return Parsers.get_yaml_data(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/yamlpath/common/parsers.py", line 110, in get_yaml_data
    yaml_data = parser.load(fhnd)
  File "/usr/local/lib/python3.8/site-packages/yaml/__init__.py", line 109, in load
    load_warning('load')
  File "/usr/local/lib/python3.8/site-packages/yaml/__init__.py", line 55, in load_warning
    warnings.warn(message, YAMLLoadWarning, stacklevel=3)
yaml.YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Is your feature request related to a problem? Please describe.
I don't know if it's possible to use yaml-set to insert elements that don't match any existing elements, I don't know if it's possible to use yaml-set to insert hashes, etc.

Describe the solution you'd like
A few examples of yaml-set operations could be added to the readme.

Describe alternatives you've considered
I've been guessing the syntax, but either those things are not possible or I'm just bad at guessing.

Additional context
For example, with a small file default.yml:

matches:
  - trigger: :lem
    replace: Ł

Let's say I want to insert a hash as another element of the matches list, that looks like:

trigger: btw
replace: by the way

I've tried things like:

$ yaml-set -g 'matches[trigger=btw]' -a '{"trigger": "btw", "replace": "by the way"}'

and

$ data='trigger: btw
replace: by the way'
$ yaml-set -g 'matches[trigger=btw]' -a "$data" default.yml

But I think nothing changes because no matches are found. If I add an element for it to match, though, the element is replaced by a string instead of a hash.

Could you please push the git tag for 3.7.0 release? We're using them for packaging on Gentoo.

Operating System

1. Name/Distribution: Arch

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython
2. Python Version: 3.8.6
3. Version of yamlpath installed: 3.3.0
4. Version of ruamel.yaml installed: 0.16.12

Minimum sample of YAML (or compatible) data necessary to trigger the issue

Smallest:

nullthing: null

A little bigger:

---
nullthing: null
intthing: 6
floatthing: 6.8
yesthing: yes
nothing: no
truething: true
falsething: false

JSON:

{
  "falsething": false,
  "floatthing": 6.8,
  "intthing": 6,
  "nothing": "no",
  "nullthing": null,
  "truething": true,
  "yesthing": "yes"
}

Complete steps to reproduce the issue when triggered via:

1. Command-Line Tools (yaml-get, yaml-set, or eyaml-rotate-keys):

% yaml-get -p nullthing nully.yml  # or:
% yaml-get -p .nullthing nully.yml  # or:
% yaml-get -p '[. == nullthing]' nully.yml  # etc.

2. Libraries (yamlpath.*): I've not yet used yamlpath as a library, just CLI.

Expected Outcome

I don't know! Some thoughts:

• Usually single results are printed "bare" (without any type indicators like quotes), so I don't know how to distinguish between an empty string and null result, or between "null" and null.
• An empty string may be alright.
• Maybe a non-zero return code is indeed appropriate 🤷 .
• This may overlap with a separate wish: option to treat everything other than a hash or list as a string (no native numbers, booleans, dates, or nulls).

Actual Outcome

CRITICAL:  Required YAML Path does not match any nodes

(retcode 1)

Screenshot(s), if Available

I can't tell if I'm failing to find this in the docs, or if it's missing from the docs, or if it's not currently possible. Honestly I can't for the life of me get an intuitive handle on the syntaxes but I'm always trying.

Is your feature request related to a problem? Please describe.

With a doc like this:

svcs:
  coolserver:
    enabled: true
    exec: ./coolserver.py
  logsender:
    enabled: false
    exec: remote_syslog -D

I would like to build, in the shell, an array of two strings like (coolserver logsender).

Describe the solution you'd like

An easily found example of such an operation in the wiki.

I expect this is possible with yaml-get, via something vaguely similar to

$ svcs=($(yaml-get -p 'svcs[. =~ /.*/][parent()]' /path/to/yml))

Describe alternatives you've considered

Using dasel, this can be done with

$ svcs=($(dasel -m -f /path/to/yml svcs.-))

Update: Maybe only additions to the documentation are needed. See my next comment below.

I wasn't sure whether this should be a bug report or a feature request, so I opted for the more positive of the two.

Is your feature request related to a problem? Please describe.
When using yaml-paths, I'd like to be able to do case-insensitive searches. I don't always know whether letters of the string I'm searching for are in upper-, lower-, or mixed-case.

Describe the solution you'd like
An option or a filter that when searching for name will find name, Name, NAME, etc.

Describe alternatives you've considered
I saw that there wasn't an option for ignoring case, so I looked for other possible solutions. Seeing that the =~ search operator for regular expressions uses a delimiter, like /, I thought I'd try adding the case insensitive matching flag following the last delimiter, thus:

yaml-paths -Ls '=~/name/i' example.yaml

However, that didn't yield any results. When I changed name in the search expression to match the case of letters definitely occurring in example.yaml, I still didn't get results.

Next, I tried removing the i regular expression flag. That yielded results, but only those that exactly matched the case of the search filter. So, that showed that the i regular expression flag was interfering with the search.

The only workaround I could find was:

cat example.yaml | tr 'A-Z' 'a-z' | yaml-paths -Ls '=~/name/'

However, all the search results were in lower case, so it takes additional commands to find the values in their original case.

PS
If I have time, I will try to open a pull request with a possible solution for this.

If a user has multiple yaml files and would like the value for each key to change uniformly, there doesn't seem to be functionality to support this within yamlpath.

Describe the solution you'd like
Allow a user to specify multiple filesnames and/or use a wildcard.
I.E.
yaml-set --change=food.salad --values"caesar" foo.yaml bar.yaml
and
yaml-set -change=food.salad --values="Cobb" *.yaml

Obviously it is possible to find a way to loop as needed, but this this would be a nice feature to be build in.

Also, wanted to say that this is extremely helpful tool and it works wonderfully! Thank you for making it!

Operating System

1. Name/Distribution: macOS
2. Version: Monterey 12.6.1 (21G217)

Version of Python and packages in use at the time of the issue.

1. Distribution: CPython from homebrew
2. Python Version: 3.11.0
3. Version of yamlpath installed: 3.6.9
4. Version of ruamel.yaml installed: 0.17.21

Minimum sample of YAML (or compatible) data necessary to trigger the issue

projects_and_groups:

  "*":
    merge_requests_approval_rules:
      standard:
        approvals_required: 1
        name: "All eligible users"

  "foo/bar":
    merge_requests_approval_rules:
      dev-uat:
        groups:
          - some_group
        users:
          - a_user

Complete steps to reproduce the issue when triggered via:

YAML Path: "**.merge_requests_approval_rules.*[.=users]"

Expected Outcome

1 result, the array at "projects_and_groups."foo/bar".merge_requests_approval_rules.dev-uat.users".

Actual Outcome

Exception: Cannot add PathSegmentTypes.MATCH_ALL subreference to dictionaries at 'None' in '**.merge_requests_approval_rules.*[.=users]'.

Screenshot(s), if Available

Is your feature request related to a problem? Please describe.
When given a path that does not exists Processor.get_nodes() either:

1. returns None if given the parameter mustexists = False
2. raises YAMLPathException if mustexists = True

Option 1 does not give the ability to know if the path is nonexistent or if it's just None, wich is a perfectly valid value in YAML.
At the same time, using option 2, catching the exception may hide the fact that the path given contains some errors.

Describe the solution you'd like
A simple and non-breakng solution is to sublclass YAMLPathException and raise different subclasses based on context.

class UnmatchedYAMLPathException(YAMLPathException):
  pass

Describe alternatives you've considered
The current workarounds are either setting the default_value to something that can never be in YAML, or checking the description of the exception. Both solution are hacky.

I can provide a PR if needed

Currently processor.set_value() converts a list to a string when assigning to a node.

would like the ability to set the value to a list.

I have 2 yaml files with the following typed array

file1.yaml

external:
  - name: DBSERVER
    value: FILE1VALUE
  - name: DBURL
    value: FILE1VALUE
  - name: FILE1
    value: FILE1VAL

file2.yaml

external:
  - name: DBSERVER
    value: FILE2VALUE
  - name: DBURL
    value: FILE2VALUE
  - name: FILE2
    value: FILE2VALUE

Expected Results

external:
  - name: DBSERVER
    value: FILE2VALUE
  - name: DBURL
    value: FILE2VALUE
  - name: FILE2
    value: FILE2VALUE
  - name: FILE1
    value: FILE1VALUE

How do I configure the rules so that both file can merge. If I'm using a simple array with unique merge option, that works well. When I use complex type array, it is not giving the expected result.

Works for simple array like

external:
  - one
  - two

external:
  - three
  - one
  - four

result

external:
  - one
  - two
  - three
  - four

Thanks for a great project! The work you have done here is truly amazing.

This is probably not a bug but rather just me doing something wrong.

Operating System

1. Name/Distribution: Fedora
2. Version: 32

Version of Python and packages in use at the time of the issue.

1. Distribution:
2. Python Version: 3.8
3. Version of yamlpath installed: 3.4.0
4. Version of ruamel.yaml installed: 0.16.12

Minimum sample of YAML (or compatible) data necessary to trigger the issue

"status":
      "phase": Failed

Complete steps to reproduce the issue when triggered via:

I'm trying to use the library in combination with the Kubernetes client library to listen for failing pods.

I have an incoming object (raw_event['raw_object']) which I'm dumping and then reading again so I can use yaml path to assert certain values. This whole approach might be too slow but for now I'm just testing.

The problem is I'm getting the following exception

Cannot add PathSegmentTypes.KEY subreference to scalars at 'status' in 'status[phase=Failed]'.

My complete, although trimmed down, loop

    config.load_kube_config()
    core_api = k8s.client.CoreV1Api()
    watcher = k8s.watch.Watch()
    stream = watcher.stream(core_api.list_pod_for_all_namespaces, timeout_seconds=0)
    for raw_event in stream:
        yaml_str = yaml.dump(raw_event['raw_object'])
        print(yaml_str)
        yaml_path = "status[phase = Failed]"
        if alert(yaml_str, yaml_path):
            print("ALARM!")

My alert function

def alert(yaml_str: str, yaml_path: str) -> bool:
    class Args:
        debug = True
        verbose = False
        quiet = True

    args = Args()
    log = ConsolePrinter(args)

    try:
        processor = Processor(log, yaml_str)
        yamlPath = YAMLPath(yaml_path)
        print(yamlPath)
        nodes = processor.get_nodes(yamlPath)
        return len(list(nodes)) > 0
    except YAMLPathException as ex:
        print(ex)

The really strange thing is that if I save the output of print(yaml_str) in a file, open that file and pass the content to my alert function, it works. That is doing something like the below

def get_yaml_str():
    yaml_file = "pod-status3.yaml"
    with open(yaml_file) as stream:
        try:
            yaml_data = yaml.safe_load(stream)
            return yaml_data
        except yaml.YAMLError as exc:
            print(exc)

Any clue what I'm doing wrong?

The only way to delete nodes now is with yaml-set command line. Would be really nice to have the functionality in a yamlpath function somehow. Maybe from the node itself, so we could write:

for node in proc.get_nodes("/some/path"):
   node.delete()

Or maybe there is already a simple to do it and I missed it?

Note to self: Users are able to create null values but once a node is null, that value cannot be changed to anything else. Example:

---
scalar_string: value
scalar_null:
scalar_int: 1028
scalar_bool: true

Actual Result

$ cat scalars.yaml | yaml-set --change=/scalar_null --value=new
---
scalar_string: value
scalar_null:
scalar_int: 1028
scalar_bool: true

Expected Result

$ cat scalars.yaml | yaml-set --change=/scalar_null --value=new
---
scalar_string: value
scalar_null: new
scalar_int: 1028
scalar_bool: true

Is your feature request related to a problem? Please describe.
In the case where the file to be changed via yaml-set or yaml-merge is either non-existent, empty, or not empty but without any values, the behavior and proper handling is unclear to me.

It seems that some dummy data must be fed in first in these cases, but I may be misunderstanding, because the docs for yaml-merge say:

With the --mergeat (-m) argument, you can direct all RHS content to one or more LHS destinations indicated via YAML Path. This enables users to merge together data fragments or even arbitrary data structure rather than premade, otherwise complete documents. Specifying a YAML Path which matches zero nodes will result in the missing structure being created on-the-fly, if possible.

Describe the solution you'd like
I'd appreciate a note from the author on best merge practices when the original document may be empty, nonexistent, or valueless.

Describe alternatives you've considered

$ rm -f vars.yml
$ yaml-set -g .newkey -a 5 -F literal vars.yml
ERROR:  File not found:  vars.yml
Please try --help for more information.
$ yaml-merge vars.yml - <<<'newkey: "5"'
ERROR:  File not found:  vars.yml
Please try --help for more information.
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ yaml-merge -m . vars.yml - <<<'newkey: "5"'
ERROR:  File not found:  vars.yml
Please try --help for more information.
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ touch vars.yml
$ yaml-set -g .newkey -a 5 -F literal vars.yml
$ # no stdout/err, but retcode 1
$ yaml-merge vars.yml - <<<'newkey: "5"'
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ yaml-merge -m . vars.yml - <<<'newkey: "5"'
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ printf '%s\n' '---' >vars.yml
$ yaml-set -g .newkey -a 5 -F literal vars.yml
$ # no stdout/err, but retcode 1
$ yaml-merge vars.yml - <<<'newkey: "5"'
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ yaml-merge -m . vars.yml - <<<'newkey: "5"'
CRITICAL:  The first input file, vars.yml, has nothing to merge into.
$ printf '%s\n' 'oldkey: "4"' >>vars.yml
$ yaml-merge vars.yml - <<<'newkey: "5"'
---
oldkey: "4"
newkey: "5"
$ # success!
$ yaml-merge -m . vars.yml - <<<'newkey: "5"'
---
oldkey: "4"
newkey: "5"
$ # success!
$ yaml-set -g .newkey -a 5 -F literal vars.yml
$ # success!