Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Market data, Trading, Technical support, Features requests, etc
ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Tue Dec 31, 2019 3:29 pm

Bitmex connections for XBTUSD@MEX and ETH_USD@MEX are disconnected at
15:00 December 31, 2019 UTC.
After reconnecting,
"No data received. Usually this means that market is currently closed."
Bitmex exchange has large transaction at that particular moment(OVERLOAD).

Time and Sales are shown however but with several minutes delay in trade data.

In my own C# connection to bitmex
BitMEXApi bitMEXApi=new BitMEXApi(bitmexKey,bitmexSecret);

time stamp of order book from Bitmex and UTC time is compared.
If there is delay, the connection is reset to resolve delay problem.

I think Bookmap adapter for Bitmex has similar problem of delay / or frozen state at the OVERLOAD.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Tue Dec 31, 2019 3:35 pm

In BookmapAPI/bitmex-adapter,
bitmex-adapter/bitmex-adapter/src/main/java/com/bookmap/plugins/layer0/bitmex/

BmConnector.java has websocket connector
WebSocketClient client = new WebSocketClient(ssl);

I cannot find codes for "checking possible delay" by examining timestamp and UTC time to
reconnect if there are OVERLOAD in bitmex.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Tue Dec 31, 2019 3:49 pm

Bookmap's diagnostics_logs and Error Reports>threads
termination_0 lock and termination_0 are attached in email: support@bookmap titled as
"Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC."

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Tue Dec 31, 2019 4:00 pm

One possibility is:
If reconnecting attempts are more than 60 times/min, Bitmex locks the
connection for one hour
.  Thus in reconnecting, sleep should be inserted to
prevent possible locking (Http 403 Forbidden error) from Bitmex.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Wed Jan 01, 2020 5:39 am

Bitmex XBDUSD is properly restored with time sales delay now correctly working.
How Bookmap detects and diagnosises the problem, and correct?

I run market making using order book and trade data in bitmex. The overload and unexpected problems from Bitmex exchange require
human intervention
. I would want to learn how the logic of Bookmap to identify
frozen state, and to restore the program.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Wed Jan 01, 2020 12:21 pm

Freezing of XBTUSD@MEX at 12:01 PM Jan 2,2020. Operation of time sales resumed with 3 mins delay.
After 10 mins., bookmap screens disappear(flash) and refresh by itself with normal operation of zero delay.


Question 1:
What is cause of freezing and delay?
Perhaps same as my C# market making codes for websocket of Bitmex in overload

Question 2:
How bitmex detects the freezing and refresh by itself?
I want to learn this logic.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Thu Jan 02, 2020 4:34 am

Subscribe/Receive has drawback in delay as in current bookmap trade(time and sales)
for several minutes in overload condition.


In practical purpose, I choose to subsample trade data and orderbook by switching polling
methods in overload.
Overload condition is triggered by longer delay computed from
Bitmex timestamp and UTC now, than the threshhold value(in my C# market making, 5 seconds.


Bookmap provides extremely valuable information. In the delay from overload condition,
practical solution, suboptimal choice, may be experimented. If there is better logical solution,
I want to learn.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Thu Jan 02, 2020 4:52 am

Time delay is detected by Bookmap after 15 minutes(4:45 am Jan.2, 2020). Heatmap is redrawn correctly.
But from practical viewpoints of traders, suboptimal solution showing polling is better than frozen and
redrawn after 15 minutes.
Profit and loss is far more important than optimality. However if you find logical solution to the delay
in overload, I would be extremely happy and want to learn.

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Thu Jan 02, 2020 8:31 am

Is time delay in Bitmex due to websocket?
In receving of future and option data, data is received through UDP in multicasting
with end point. In DMA(direct market access), I have no experience of time delay or
missing events.


In polling, cumulative volume data is sacrificed through sampling.

In cryptocurrency, widespread use of websocket instead of multicasting UDP is the main reason for delay?

ingyukoh
Posts: 32
Joined: Tue Nov 26, 2019 7:06 am
Been thanked: 1 time

Re: Error Report: Bitmex XBTUSD frozen at 15:00 December 31, 2019 UTC.

Post by ingyukoh » Thu Jan 02, 2020 1:28 pm

Bookmap error:
Full report content:
java.lang.NullPointerException
at com.bookmap.plugins.layer0.bimex.Provider.listenForEcecutionTask(Provider.java:903)


Bookmap run options
External data/Trading simulated by Bookmap

Load previously opened workspace
Record Live data

Since I do NOT trade through brokerage account(real or simulated), listenForExecution has no responsibility
in running. However listenForExecution examines "new",client Order ID,trailing stop(Not triggered, Trade,canceled.



In overload condition, threadworker gets delayed and out of synch, or wait until fellow worker in sync.
Synchronization of thread workers seem to be reason of delay or frozen state in Bitmex overload state(where there are a lot of trading at the given time).
In overload condition, ignore synchronization, just provide APPROXIMATE heatmap and volume dots.
Since we lose some of trade data in listenForTrade, cannot compute cumulative volume delta precisely.
Since we lose some of order book in listenForOrderBookL2, COB, CQC are approximate.

But from trader viewpoint, real time APPROXIMATE heatmap and volume dots are MUCH BETTER than frozen
heatmap or volume dots.

Of course, if one figures out to solve overload state threadworker synchronization and provide coherent heatmap and volume dots, it would be really good.

Post Reply