Live Webinar and Q&A: ChatGPT is fun, but the future is fully autonomous AI for code (Live Webinar Mar 21, 2023) Save Your Seat
Facilitating the Spread of Knowledge and Innovation in Professional Software Development
The practice of test && commit || revert teaches how to write code in smaller chunks, further reducing batch size. TCR yields high coverage by design, which smooths the downstream testing pipeline.
This article describes a framework for making architectural decisions using three building blocks: The company’s own Technology Radar; Technology Standards; and Architecture Decision Records (ADRs). The framework clarifies decision-making, team involvement, and information on already made decisions and aligns with the company’s needs and culture.
Omar Sanseviero discusses the trends in the ML ecosystem for Open Science and Open Source, the power of creating interactive demos using Open Source libraries and BigScience.
In this podcast Shane Hastie, Lead Editor for Culture & Methods spoke to James Stanier, Director of Engineering at Shopify and Track Host at QCon San Francisco for the Remote and Hybrid Work: What’s Next track, about what makes a great engineering manager, nurturing culture with 14000 remote workers and deliberately balancing synchronous and asynchronous work.
Istio’s new ambient mesh allows for a sidecar-less data plane. This article discusses the advantages of that approach and the work that went into making it a reality.
Adopt the right emerging trends to solve your complex engineering challenges. Attend in-person or online.
Find real-world practical inspiration from the world’s most innovative software leaders. Attend in-person or online.
Learn what’s next in software from world-class leaders pushing the boundaries. Attend in-person or online.
Your monthly guide to all the topics, technologies and techniques that every professional needs to know about. Subscribe for free.
InfoQ Homepage Articles InfoQ Case Study: NASDAQ Market Replay
This item in japanese
Jun 09, 2008 16 min read
by
NASDAQ Market Replay provides a NASDAQ-validated replay and analysis of the activity in the stock market. The application is built using the Adobe Flex and AIR platform, and utilizes the Amazon Simple Storage Service (S3) for persisting historical market data. The combination of S3 and AIR offers a powerful deployment model with little internal infrastructure required. The simple, yet robust, deployment is possible because the AIR runtime runs on the client machine. Amazon S3 removes the need for a traditional middle-tier server, as the data is accessed in from the Amazon "cloud".
The Market Replay application enables users to view the best bids and offers at any point in time, replay the market in simulated real-time, and zoom to view events at the millisecond level. Investors can validate best-execution and Reg NMS compliance. Brokers and traders can review events at the time when their trades occurred to determine whether there was a problem or a missed opportunity. Brokers can send clients a NASDAQ-validated replay of the moment a trade occurred to validate their performance.
Figure 1. Market Replay Case Study
Presented by: Mathew Lodge – CEO, Diffblue
Save your seat
A universal experience among investors and traders, both professional and non-professional, is the question "what happened?" Did the broker get best execution and comply with Reg NMS? Did the trader miss an opportunity? Why did a retail investor get a different price than expected? What investors, traders, and compliance officers need to answer these questions is a way to rewind and replay the market, slow it down, and zoom in to the second and even millisecond level to see exactly what happened.
The NASDAQ Market Replay application does this using "NASDAQ Official" validated data. There are a number of use cases where Market Replay is useful. For example:
NASDAQ Market Replay provides both casual and professional investors the information they need to understand what has happened in the market.
The Market Replay application was launched in February, 2008, at the same time as Adobe officially released the Adobe AIR 1.0 platform for production use. With the power of the AIR and S3 platforms, it only took about 6 months to take the application from concept to a working production implementation. A team of 10 people worked on the application at different points throughout the implementation, working on both the backend data offloads and the user interface.
The client desktop application was built using Adobe Flex and runs in the AIR runtime. Adobe Flex is an application development framework that is used by developers to build applications that run in the Flash Player. Adobe AIR allows developers to create applications for the desktop using Web technologies such as HTML/CSS, Ajax, Flash, and Flex. In addition, AIR provides offline support and a simple deployment paradigm for delivering the client applications.
S3 provides a robust solution for storing the high volumes of data required to provide the replays. This allows for a unique deployment as the AIR application runs on the user’s computer and data is stored in S3. Thus, no heavy server infrastructure was required to deploy the application into production.
The Market Replay implementation improves on similar solutions in a number of ways. For example:
The pairing of AIR and S3 makes it possible to retrieve and visualize the data quickly for the users. One of the application’s main functions is allowing users to view the state of the consolidated order book at any point in time. The application is able to quickly load new order book files from S3, and the AIR application uses the processing power of the user’s desktop computer to update the user interface. This enables users to move easily from one instance to another, without any delays for a server to recalculate and disseminate the display of the next state of the order book.
An essential feature of the application is that it provides users a realistic replay of the market activity in the same way a trader would have seen it at their workstation in real time. This functionality requires sorting and aggregating the order update messages to create the consolidated order book at any point in time. The Flex interface provides two main displays to accomplish this:
It is key that the application be able to display every detail in the data. This requires zooming down to quote updates that could be as short as a fraction of a millisecond. The application must allow users to zoom into smaller and smaller time frames to view each element in the data. In addition, the replays must be slowed down so that those events can be understood by the human eye. These types of animations / visualizations of the data made Adobe Flex a natural choice.
One of the major advantages of deploying to the Flash Runtime is the native animation support, which provides the foundation for the powerful visualizations in the Replay application. The Adobe Flex framework, which is built on top of the Flash API’s, provides a full set of charting components for displaying common visualizations of data. The Flex charting components, along with the other out-of-the-box Flex components, are well implemented to provide rich functionality but still be extensible. Thus, the standard Flex charting components were extended and customized for the Market Replay application. By beginning with the off-the-shelf Flex data visualization components, NASDAQ was able to develop their custom components more quickly than would have been possible starting from scratch.
Using the out-of-the-box Flex charting components is fairly straightforward. In the following example, fictitious data is hard coded into the MXML source file. MXML is a declarative XML markup provided as part of the Flex framework for coding parts of Flex applications. It is an abstraction on top of the core Flash Player programming language ActionScript.
In this example, data is hard coded in the source to allow for a self-contained example. In the actual Market Replay implementation, real data is loaded from the Amazon S3 data store. The data then uses the HLOCChart (High Low Open Close) Flex component to render a visualization of the data. See the following resulting screen shot.
Figure 2. Sample Visualization
Adobe AIR provides an ideal runtime for the client application, as the local resources allow for doing calculations on large amounts of data quickly. The application allows users to select a time range to calculate the minimum and maximum best bids and offers for each exchange, and the national best bid and offer. These calculations typically require a large data size and heavy processing to compute the results. Processing user requests on the server would have required a powerful web server, and resulted in delays as customers awaited a round-trip request for each calculation request. Adobe AIR allowed for minimal server infrastructure by pushing most of the work to the user’s desktop.
Another way AIR improved the user experience was by enabling replay and analysis functionality that does not depend on uninterrupted network access. Once the data file is retrieved from S3, the replay and calculations can be used with or without an internet connection. This is useful not only for NASDAQ’s clients but also for NASDAQ’s sales force when they’re visiting customers and cannot count on having internet access when they demonstrate the application.
The power of Adobe AIR and Flex were a key factor in launching development efforts for the application. They provided the necessary tools to quickly show quality results, as the work was begun with off-the-shelf Flex components. The components were then customized as the application evolved, eventually leading to the completed application.
Amazon S3 was selected because Market Replay requires that historical market data be stored in a way that is inexpensive and extremely scalable. Stock markets generate many gigabytes of trading data each day. The Market Replay application requires that every detail be stored and rapidly available to enable the system to answer user’s requests quickly.
S3 was also desirable because NASDAQ wanted to offer a data service that is able to keep many years of data online and immediately available at a reasonable cost to all involved. With S3, Market Replay can support billions of files for countless numbers of users without sacrificing performance.
Market Replay supports users who have regulatory, legal, and customer inquiries about trades that may have occurred many months or years ago. It is a key requirement that all historical replay data be immediately available. S3 has proven to be able to maintain consistently fast access speeds.
Before loading data into S3, NASDAQ uses staging servers to convert the data from the format that is distributed over real-time feeds, to a format that is optimized for replaying. This proprietary conversion process results in very simple and efficient text files that are optimized for quick upload and download. The files contain all the quote information required for the AIR desktop client application to build replays and analyses of the market in extreme detail. In full production, this means that Market Replay loads hundreds of thousands of files each day to S3.
Market Replay utilizes a simple and small comma delimited flat file format for the data. S3 is designed to store and retrieve such huge numbers of files quickly and reliably. The files have a simple-human readable format as seen in the example below. The filename provides the stock ticker symbol, date, and start time of the period covered by the data in the file. The first few records provide the initial state of each exchange’s bid and offer. Subsequent records show all the changes. The include fields are: Exchange, sequence number, shares at bid, shares at offer, price of bid, price of offer, start time (milliseconds since midnight), end time (milliseconds since midnight)
Organizing and managing millions of files would seem problematic, but the Replay application fits well with the flat file model, as there is a limited amount of data needed for any particular replay or analysis. That makes for an amount of data that can be stored in a manageable text file on the server. The application knows how to convert the user’s replay request into a filename, request it from Amazon S3, and then parse it once it arrives. The application is not currently designed for open ended queries across many stocks or across long time periods. It is designed to provide views and analysis for a single stock in extreme detail.
Amazon provides both REST and SOAP interfaces for accessing data stored in S3. The Flex framework provides support for working with both REST and SOAP interfaces. In addition, there is an Open Source API, called as3awss3lib, hosted on Google Code that provides comprehensive support for interacting with S3 in AIR runtime using Action Script.
An example of accessing files using the as3awss3lib follows:
In this example, the "getFile" method creates the AWSS3 class provided by as3awss3lib and invokes the "getObject" method for retrieving the desired file. Flex accesses all remote services asynchronously. Thus, the getObject call happens asynchronously and uses the handler declared for the AWSS3Event.OBJECT_RETRIEVED event to process results when the call is returned. In this sample code, the file is saved and an Alert is shown to the user.
The data preparation process breaks the data into individual files. One file covers a single stock symbol, on a single day, for a single 10-minute time period (all time periods are standardized 9:25-9:35, 9:35-9:45, etc). The filename identifies the stock, date, and time-period of the data in the file. It is entirely sequential by date, stock symbol, and time. Users begin a replay by entering a stock symbol, date, and time. The client application translates that symbol, date, time information into a filename. The client then reaches out to S3 and asks for the file that it needs. S3 is very efficient at quickly retrieving files and returning the contents.
Along with the technical benefits of Amazon S3, it was selected because of its pricing model. Its pricing is transparent and predictable. It allows for accurately forecasting cost of operations, and monitoring spending in real time.
Careful preparation of the files uploaded to S3 and the S3 cost profile are critical to enabling NASDAQ to commit to never offloading data. S3 is inexpensive and most of the expense occurs when data is uploaded or downloaded. The cost of maintaining the large number of files that are not touched in a typical month is very low.
S3 charges only for the capacity Market Replay uses. This greatly reduced the cost of developing and launching Market Replay because it did not require NASDAQ to purchase hardware that would take months or years to use at full capacity. S3’s scalability and pricing model enables for scaling up as it is needed, without having to buy extra "headroom" in anticipation of a possible acceleration in user base growth.
NASDAQ Market Replay is nearing the release of a new version that will include trades as well as quotes. It required major changes to the application, including: synchronizing the quotes data and trades data, a complete rewrite of the chart interface, a new panel that includes a list of trades and their characteristics, and new calculations done on the trades data. The quality and power of the Market Replay application architecture and the AIR and S3 platforms have been demonstrated through the enhancement process, as these major updates were implemented in only about a month of development time. In addition, AIR makes it easy to rollout new versions of the application because the application checks and installs updates each time it is launched.
It is an exciting time for the software industry, as the NASDAQ Market Replay implementation demonstrates that a powerful data driven application can be brought to the market quickly and deployed within a limited budget. Much of this is due to the improvements in the platforms, with both the emergence of cloud computing and strong client side runtimes.
Readers can get free trial access to NASDAQ’s Market Replay application at NASDAQ’s DataStore (https://data.nasdaq.com/MR.aspx). Those who would like full access should request the application from their regular stock market data provider such as a broker, financial web portal, or financial information vendor.
Writing for InfoQ has opened many doors and increased career opportunities for me. I was able to deeply engage with experts and thought leaders to learn more about the topics I covered. And I can also disseminate my learnings to the wider tech community and understand how the technologies are used in the real world.
I discovered InfoQ’s contributor program earlier this year and have enjoyed it since then! In addition to providing me with a platform to share learning with a global community of software developers, InfoQ’s peer-to-peer review system has significantly improved my writing. If you’re searching for a place to share your software expertise, start contributing to InfoQ.
I started writing news for the InfoQ .NET queue as a way of keeping up to date with technology, but I got so much more out of it. I met knowledgeable people, got global visibility, and improved my writing skills.
Becoming an editor for InfoQ was one of the best decisions of my career. It has challenged me and helped me grow in so many ways. We'd love to have more people join our team.
A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example
We protect your privacy.
You need to Register an InfoQ account or Login or login to post comments. But there’s so much more behind being registered.
Get the most out of the InfoQ experience.
Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p
by sunil d’monte,
by sunil d’monte,
Your message is awaiting moderation. Thank you for participating in the discussion.
Really interesting case study, thanks for posting it. I wonder if you could elaborate more on your decision not to have a server application sitting between the client and S3? E.g. could you share some performance numbers if you did any prototyping? Also, are you storing any user state, and if so, where? E.g. what if you had a “Save this search as…” feature, which users could re-use at a later point of time…
Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p
Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p
A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example
We protect your privacy.
Real-world technical talks. No product pitches.
Practical ideas to inspire you and your team.
March 27-29, 2023. Attend in-person or online.
QCon London brings together the world’s most innovative senior software engineers across multiple domains to share their real-world implementation of emerging trends and practices.
Level-up on 15 major software and leadership topics including Modern Frontend Development and Architecture, Enhancing Developer Productivity and Experience, Remote and Hybrid Work, Debugging Production, AI/ML Trends, Data Engineering Innovations, Architecture in 2025, and more.
SAVE YOUR SPOT NOW
InfoQ.com and all content copyright © 2006-2023 C4Media Inc.
Privacy Notice, Terms And Conditions, Cookie Policy