Giter Club home page Giter Club logo

yandex_market_language's People

Contributors

divaltor avatar pyup-bot avatar stefanitsky avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

yandex_market_language's Issues

Huge memory usage when reading large files

  • Yandex Market Language (YML) for Python version: 0.6.0
  • Python version: 3.8.3
  • Operating System: Windows 10 2004 (for testing) and Debian 10 (for production)

What I Did

I tried to process a 100 MB file and the processing machine exceeded the memory limit (~2 gb). Is there any way to optimize file reading or processing so that it doesn't go out of memory?

image

Error validate date

  • Yandex Market Language (YML) for Python version: 0.6.2
  • Python version: 3.8
  • Operating System: Windows 10 develop and Ubuntu 18.1 server

Description

When parsing the price list, I get an exception. The price list is generated by the official Yandex module for Bitrix. As you can see, the timezone is used in the price list.

What I Did

Run command

feed = parse(file_path) 

I get an exception

yandex_market_language.exceptions.ValidationError: time data '2021-08-05T02:13:01+00:00' does not match format '%Y-%m-%d %H:%M'

001
002

CI/CD

Need to replace Travis CI with Github actions.

Error parsing seller_warranty tag

  • Yandex Market Language (YML) for Python version: 0.6.0
  • Python version: 3.8.3
  • Operating System: Windows 10 v. 1909 (build num 18363.836)

Description

When I try to parse a file that has a tag called <seller_warranty> I get an error. I attach the log below.

What I Did

Code

import argparse
from typing import Optional

import yandex_market_language as yml


class YMLParser:

    def __init__(self, file: str, count: Optional[int] = None):
        self.feed = yml.parse(file)
        self.data = self.feed.to_dict()


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('file', help='.yml or .xml file to parse')
    parser.add_argument('-c', '--count', type=int, help='minimum count of items to add them to separate file')

    args = parser.parse_args()

    if not args.file:
        raise ValueError('You need to select file for parse')

    yml_parser = YMLParser(args.file, args.count)
    print(yml_parser.data)

File template

<?xml version="1.0" encoding="windows-1251"?>
<!DOCTYPE yml_catalog SYSTEM "shops.dtd">
<yml_catalog date="2020-04-14 12:00">
	<shop>
		<name>...</name>
		<company>...</company>
		<url>...</url>		
		<categories>There is some <category> tags </categories>
                <offers>
                             <offer>
                             // Some tags with information about offer

                             // Empty tag <seller_warranty>
                             <seller_warranty></seller_warranty>
                             </offer>
                 </offers>
        </shop>
</yml_catalog>

Log error

Traceback (most recent call last):
  File "C:/Users/god/PycharmProjects/yml_parser/app/main.py", line 24, in <module>
    yml_parser = YMLParser(args.file, args.count)
  File "C:/Users/god/PycharmProjects/yml_parser/app/main.py", line 10, in __init__
    self.feed = yml.parse(file)
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\yml.py", line 44, in parse
    return YML(file_or_path).parse()
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\yml.py", line 30, in parse
    return Feed.from_xml(root)
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\models\feed.py", line 44, in from_xml
    shop = Shop.from_xml(el[0])
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\models\shop.py", line 183, in from_xml
    offer = models.ArbitraryOffer.from_xml(offer_el)
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\models\offers.py", line 536, in from_xml
    return ArbitraryOffer(**kwargs)
  File "C:\Users\god\.virtualenvs\virtualenvs\yml-parser-GzhhnpNv-py3.8\lib\site-packages\yandex_market_language\models\offers.py", line 506, in __init__
    super().__init__(vendor=vendor, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'seller_warranty'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.