neuralegion / har Goto Github PK

View Code? Open in Web Editor NEW

22.0 18.0 7.0 344 KB

HAR (HTTP Archive) parser in Crystal

License: MIT License

Crystal 100.00%

har http-archive http-archive-format json crystal parser

har's Introduction

har

HAR parser for version 1.2

Installation

Add this to your application's shard.yml:

dependencies:
  har:
    github: NeuraLegion/har

Usage

json = HAR.from_file(file)
# or from string using HAR.from_string(String)
pp json
json.version
# => 1.2
json.entries
# etc..

Development

Pretty much done. If there is a needed feature please open an issue.

Testing

To run specs with debugging output you will have to set the LOG_LEVEL to debug.

LOG_LEVEL=debug crystal spec

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

Contributors

bararchy Bar Hofesh - creator, maintainer
Sija Sijawusz Pur Rahnama - contributor, maintainer

har's People

Contributors

Stargazers

Watchers

Forkers

tawawhite kedit sdogruyol nlomerowitz bcardiff jkthorne

har's Issues

Lazily yield entries from IO instead of a string

While parsing big HAR files, I may occasionally run into the following error:

2021-01-09T12:11:24.521623Z   INFO - NeuraLegion::HARParsingService: Processing "incoming/some.har.zip"...
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Invalid memory access (signal 11) at address 0x0
[0x557abc4e9396] *Exception::CallStack::print_backtrace:Int32 +118
[0x557abc49402e] __crystal_sigfault_handler +398
[0x7fef41040890] ???
[0x7fef402058f0] abort +560
[0x557abcc8e0f2] ???
[0x557abcc8e39a] GC_expand_hp_inner +410
[0x557abcc8e5fd] GC_collect_or_expand +349
[0x557abcc9244b] GC_alloc_large +235
[0x557abcc92b43] GC_generic_malloc +307
[0x557abcc92cc1] GC_malloc_kind_global +225
[0x557abcc931f1] GC_realloc +209
[0x557abc55be61] *GC::realloc<Pointer(Void), UInt64>:Pointer(Void) +49
[0x557abc47ebe4] __crystal_realloc64 +68
[0x557abc4df23e] *Pointer(UInt8) +94
[0x557abc5604d2] *String::Builder#resize_to_capacity<Int32>:Pointer(UInt8) +50
[0x557abc5603e8] *String::Builder#write<Slice(UInt8)>:Nil +184
[0x557abc671db2] *Compress::Zip::ChecksumReader +370
[0x557abcaaeafe] *NeuraLegion::HARParsingService#parse<Compress::Zip::ChecksumReader>:NeuraLegion::HARParsingService::Observation +302
[0x557abcaa0d89] *NeuraLegion::HARParsingService#run:Bool +70601
[0x557abc4c4925] ~procProc(Nil) +197
[0x557abc590120] *Fiber#run:(IO::FileDescriptor | Nil) +208
[0x557abc49351d] ~proc2Proc(Fiber, (IO::FileDescriptor | Nil)) +29
[0x0] ???

I suspect that this is the failing piece of code:

    private def parse(io : IO) : Observation
      observation = Observation.new
      parser = Parser.new

      entries = HAR.from_string(io.gets_to_end).entries # This
      entries.each do |entry|

I.e. the io.gets_to_end part.

I think it would be useful if there was an initializer accepting an IO instead of a String which would yield entries lazily, one-by-one. For example, HAR.from_io(io) do |entry|.

However, this would imply an entirely different approach compared to how the shard is implemented now, so this is a long-shot... 🤔

Request `to_http` method lost multipart/form-data bodies

require "json"
require "har"

r = %q<{
                    "url": "https://api.example.com/v2/file",
                    "method": "POST",
                    "headers": [
                        {
                            "name": "Authorization",
                            "value": "RELEVANT_API_KEY"
                        }
                    ],
                    "queryString": [],
                    "cookies": [],
                    "postData": {
                        "mimeType": "multipart/form-data",
                        "params": [
                            {
                                "name": "query",
                                "value": "mutation add_file($file: File!) {add_file_to_column (item_id: 1234567890, column_id:\"files\" file: $file) {id}}\n"
                            },
                            {
                                "name": "map",
                                "value": "{\"image\":\"variables.file\"}\n"
                            },
                            {
                                "name": "image",
                                "value": ""
                            }
                        ]
                    },
                    "headersSize": -1,
                    "bodySize": -1,
                    "httpVersion": "HTTP/1.1"
                }>

req = HAR::Request.from_json(r)
http = req.to_http_request
x = String.build do |io|  
  http.to_io(io)
end

puts x.inspect

"POST https://api.example.com/v2/file HTTP/1.1\r\n" +
"Authorization: RELEVANT_API_KEY\r\n" +
"Content-Length: 0\r\n" +
"\r\n"

Possibly bad transformation from HAR::Headers to HTTP::Headers

http_headers = HTTP::Headers{ "Foo" => ["bar", "baz"] }
String.build { |io| HTTP.serialize_headers(io, http_headers) } #  => "\"Foo: bar\\r\\nFoo: baz\\r\\n\\r\\n\""

but using code

har/src/har/request.cr

Lines 98 to 104 in 152ff00

 def http_headers : HTTP::Headers 

 HTTP::Headers.new.tap do |http_headers| 

 headers.each do |header| 

 http_headers[header.name] = header.value 

 end 

 end 

 end

For example in case if har request headers contains:

 "headers": [
    {
      "name": "Referer",
      "value": "https://secret.dvwa.k3s.qa.nexploit.app"
    },
    {
      "name": "Referer",
      "value": "https://secret.dvwa.k3s.qa.nexploit.app/vulnerabilities/xss_d/"
    }
  ],

we got only second value using har_request.http_headers because already existed header value will be rewritten

Seems we need to use something like that to process headers correct:

    def http_headers : HTTP::Headers
      HTTP::Headers.new.tap do |http_headers|
        headers.each do |header|
          http_headers.add(header.name, header.value)
        end
      end
    end

neuralegion / har Goto Github PK

har's Introduction

har

Installation

Usage

Development

Testing

Contributing

Contributors

har's People

Contributors

Stargazers

Watchers

Forkers

har's Issues

Lazily yield entries from IO instead of a string

Request `to_http` method lost multipart/form-data bodies

Possibly bad transformation from HAR::Headers to HTTP::Headers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def http_headers : HTTP::Headers
	HTTP::Headers.new.tap do \|http_headers\|
	headers.each do \|header\|
	http_headers[header.name] = header.value
	end
	end
	end