Running a Windows app headless in a debian/node container
I had a problem. I have been building a data analysis/visualisation app in Node/React for a client and have been sourcing my data from the undocumented/deprecated Yahoo Finance API. It turns out that (surprisingly) the undocumented/deprecated Yahoo Finance API doesn’t provide complete or consistent data for UK stocks (we’re interested in the fundamental and technical data points). So we had to find an alternative datasource, and boy was this hard.
We evaluated the following sources:
About half didn’t provide real-time or delayed prices (they only had historical) and/or didn’t provide enough fundamental/technical signals. Also some were far too expensive ($15k+ pa). And one didn’t respond (Reuters, disappointingly, I’m looking at you).
The stand out winner was DTN (although subsequently we found out their fundamental/technical data was incomplete for UK stocks and we had to pivot to US stocks to complete the MVP — they might resolve this issue).
Data source problem solved… but not quite. DTN don’t provide an internet API, they only provide a proxy agent that runs on a host machine, which you can communicate with locally via a TCP/IP socket…
So I had a new host of problems… i). write a streaming socket client in js and ii). workout how to run this IQFeed agent in “production”.
Writing the client was relatively easy. i). write a pretty wrapper around
net.Socket, ii). connect to a few ports, iii). do an auth handshake and configure their client (using their propriety streaming API) and iv). choose the stocks that I want to watch, and hey presto, I now have a streaming source of data. I even wrapped the calls in promises, so you can do: (the example below batches and tracks 6
Event: 'data' events)
I would love to open source my iqfeed client, but their developer agreement prevents me from sharing info about their propriety API. Also this is why there are no screenshots :-(
Anyway, now to move onto the bigger problem. How to get the IQFeed agent in production. I noticed that their macOS version was identical to their Windows version but packaged using WineBottler, so Wine was my starting point.
I then spun up an Ubuntu VM, installed Wine and again, hey presto, it worked out of the box. Perfect. Now how to I squeeze it into my node Docker container that the app actually runs in…
So here we go:
I’m like oh, win32, old school… let’s try again:
I didn’t really know what X server but I was aware that running a windows GUI installer on a linux non-GUI OS was going to be difficult.
After a bit of Googling I found this excellent post by Fredrik Averpil (credit for getting XQuartz and Docker working all his). It basically tells me how I can run X server locally on macOS and if I enable networking, I can share a display with my container. I was like WTF? This is magic. Let’s do it.
On my mac:
In “XQuartz Preferences” enable networking “Preferences > Security > [x] Allow connections from network clients” and then add local IP to ACL:
Then the moment of truth do (repeat the above install):
I was like OMG. A Windows Installer running in a debian/node container on Docker has just appeared on my Mac desktop. Mind blown.
Then halfway through the install it hanged… just hanged…: “Microsoft Visual C++ 2014 Update 4 Redistributable not found!”. Microsoft LOL.
After trawling IQFeed forums and elsewhere on the web I found that the solution was to use the official WineHQ Debian packages https://wiki.winehq.org/Debian instead.
So here we go again:
This time it downloaded the dependences and worked!! Now for the critical test, does the installed program work?
And yes! It worked… I cannot describe how elated I was at this point. Now is it portable and repeatable?
Yes it was. Now is it possible to run it headless?
After much Googling I found a program Xvfb (X Virtual Frame Buffer)… could this work?
And then I got those magic words: “connected to server!”.
Well that was 5 hours of fun!
Finally, can I turn all of this into a Docker image?
This is now running successfully in production. I hope it opens up opportunities to people who face similar problems.
Originally published at https://msyea.github.io on July 7, 2017.