Subscribing to market data with the latest TWS return error "Please use 'Key=Value' format for Misc options" about ib-api HOT 27 CLOSED

OK.
Just checked: My code was executed with V 10.12. sorry.

By comparing to the python code, the only difference is "XYZ" for not supported mktDataOptions.
You can try to change it to "" in messages/outgoing/request_marketdata.rb:99

First impression is: it works

Will test it next week.

from ib-api.

Hi,
which tws-version do you use?

from ib-api.

I'm using Build 10.16.1k. I couldn't find the version

from ib-api.

I was running on the "latest" channel for updates in IBKR.

Switching to the stable channel fixed the issue.

from ib-api.

Thanks for mention this incident anyway.
I will update the application for the 10.xx release during the summer.

from ib-api.

Ping me if you need testing done, would be happy to help.

from ib-api.

I'm using the latest stable build (10.19.1j, Jan 11, 2023), and have exactly this issue.

Where could I start to look for the problem? I'm willing to patch it myself, but no idea where to begin.

I assume it must be somewhere in the construction of the message for the contract details.

from ib-api.

Hi casper,
thats surprising.

Just did

> s =  Stock.new symbol: :amd   => #<IB::Stock:0x00007fead2535bc8                                                    
> s.market_price
 => 71.1 

using the master branch of ib-api and ib-extensions

to reproduce:

• start a paper-trade or a »real« TWS / IbGateway
• clone ib-api and ib-extensions
• modify the Gemfile of ib-extensions:
  • gemspec
  • gem "ib-api" , path: "../ib-api/"
  • gem 'ox'
• adjust bin/console.yml to your port
• start the gateway: ./gateway t (or g)
• enter : s= Stock.new( symbol: :amd); s.market_price

market_price uses the code you mentioned above.

I am using the stable version of the ib-gateway (Version 10.19)

from ib-api.

Short addition: Am working on integrating ib-ruby, arcadedb and bridgetown to a platform to develop, run and publish trading-solutions. Thus, the gems will be updated soon. (ib-symbols are already done).
If you want to participate, your input would be greatly appreciated. hartmut

from ib-api.

I followed your instructions to the T. Even tried a couple different Ruby versions to rule out some Ruby version fluke. No go :(

I am using TWS, not Gateway.
Live and Demo account, both have the same issue.

from ib-api.

Btw. this was working fine before I upgraded my TWS to the latest stable. My previous TWS version was quite old, but no problems there.

from ib-api.

BTW: This works with the experimental ver_10 branch, too.

from ib-api.

Ok that seems to fix it! :)
It takes a couple of seconds to get the quote though, but I assume this is normal?

Thanks a lot for the help.

Btw. from the (little bit) I have used this library so far the API seems very pleasant to work with. Nice job on it. Thanks to everyone that's working on it.

from ib-api.

Slightly related question (perhaps I need to open a new issue?), but correct me if I'm wrong but I believe there is a race condition in market_price.rb?

https://github.com/ib-ruby/ib-extensions/blob/060d4a9b80df0d49e7ce8697f2349c6161c7dd05/lib/ib/market-price.rb#L67

Line 67:

   s_id = tws.subscribe(:TickSnapshotEnd){|x| q.push(true) if x.ticker_id == the_id }

It runs q.push, but elsewhere you call q.close. I've had a couple of very hard to reproduce crashes, where that line creates a push: queue closed (ClosedQueueError) error.

Which means someone closed the queue before TickSnapshotEnd arrived.
I think the subscription runs in a separate thread, which means any close call runs the risk of closing the queue on an incoming Snapshot message.

Not familiar with Queue concurrency, but perhaps you need to check for q.closed? before calling push? Or unsubscribe all messages before calling q.close.

It happens perhaps once in 100 market_price calls. Sometimes I can't get it to happen whatever I try. Sometimes it happens 10 times in a row. Totally random.

from ib-api.

Interesting. Never happened in my setup.

The idea of the utility is to use the queue to serialize concurrent events.
Under normal circumstances, the subscriptions should have been removed prior to a possible next incoming data-event. The problem with race conditions is, they appear if circumstances are not normal.
We can check if the queue is still open before each writing or respond to the CloseQueueError and let the core library take care of the circumstances. I prefer to use the second approach.

edit:
I think, I found the cause of the issue.
There is a while !q.closed loop. The centerpiece there: if results are present populate contract.misc & contract.bars.
This might be time consuming and happens before the TickSnapshotEnd Event is received. This calculation should be run after anything is unsubscribed.

from ib-api.

pushed the changes in market-price to a new branch in ib-extensions.
Can you please check if the algo is more robust now?

from ib-api.

Ok awesome that you figured it out. But I still got the ClosedQueueError exit. Can you really rescue an exception like that from inside a block, with a rescue clause outside a block?

I think it doesn't work, right? Try this:

def save &block
  @blk = block
end

def call
  @blk.call
end

begin
  save { 1/0 }
rescue
  puts "BANG"
end

# Zero division exception will still be raised
call

from ib-api.

@casper as I see it, there are three options.
a) use the TickDataEnd message as trigger to fill the results into the contract-object
b) test the state of the queue prior to its access
c) use a fixed time to complete the data-gathering

The reason to use the present »dirty« approach: sometimes a huge time gap before the TickDataEnd-Message appears.
Perhaps you can test this in your environment. I have to do that blindly.
(If you want, you can get the right to push to ib-ruby)

from ib-api.

Ok. It's nice that it tries to acquire the data as soon as possible I think.

But I noticed another thing, which might explain why you cannot reproduce it. Running with Ruby 3.x the error never happens. At least I am not able to make the problem trigger with 3.x. Running with an older Ruby, especially 2.5.1 (which happens to be my Ubuntu 18.04 system Ruby), it happens much more frequently.

For now this problem is not a show stopper. I can switch Ruby version no problem. I tried testing the state prior to access on 2.5.1, and it seems to work. However, if we really nitpick, in reality I would probably need a mutex for it, because I think (?) Ruby could still context switch between the test and the access (small chance though). Since the test + access is not really an atomic operation.

I'm going to switch to Ruby 3.x version for now. If the problem starts cropping up on 3.x also, maybe I can think about a fix. I'm not really that experienced on Ruby multi-threading, so I would need to read up on it a bit.

I mean if we just want to do it "correctly" and not dirty. The dirty 99% fix is just to test prior to access. OR perhaps even just create something like a monkey-patched push-safe method, that just catches the exception and ignores the error? But in any case, Ruby 3.x seems to work for now.

from ib-api.

Do not use the system-ruby, ubuntu provides. Either install it via snap or rvm.
Best approach: run the ib-container, where everything runs in isolation. Thats my standard environment.

from ib-api.

Ok. Got it. I'm making a command-line tool for used during manual trading in conjunction TWS (so no gw), so the container is not so good for this use case. But rvm works. It's curious. 3.x must have some difference in its threading, since the issue does not pop up there. Seems a subtle difference in threading code is enough to not trigger the problem.

from ib-api.

Now I am getting this error when requesting historical data.

  ib.send_message IB::Messages::Outgoing::RequestHistoricalData.new(
    :request_id     => 100,
    :contract       => contract,

    :end_date_time  => Time.now.to_ib,
    :duration       => '30 D',
    :bar_size       => :min15, 

    :what_to_show   =>:trades,
    :use_rth        => 0,
    :keep_up_todate => 0)

I have patched [:mkt_data_options, ""] in request_marketdata.rb, but no go :(

Is the historical data request using some additional fields which are not in request_marketdata? I don't get why the patch does not work in this case.

from ib-api.

I figured it out. I grepped the code for strings with XYZ, and replaced them all by an empty string. It looks like the historical request is in lib/ib/messages/outgoing/bar_requests.rb, where several XYZ strings must be patched.

from ib-api.

Which version of ib-api are you using?
(xyz should be eliminated in the recent versions)

from ib-api.

Yeah you're right. My bad. It looks like it was an old checkout. I see you put this fix in bar_requests.rb also earlier. Sorry. Just tested and it works.

from ib-api.

Recently I had problems fetching market- and historical data with multiple threads. Am investigating that. So, use concurrent requests with caution.
(I guess, max of 50 threads is ok, so my »solution« would be, to introduce some batching-facility to limit the amount of concurrent threads.

from ib-api.

Note: there is a macro "contract.get_bar". It is used by contract.eod (ib-extensions). This can be used to fetch other then daily data. It returns polar dataframes for further data-processing.

from ib-api.