Home Our Success Stories Realtime Kibana Dashboard for a financial tech firm

Realtime Kibana Dashboard for a financial tech firm

Ajay Bidyarthy

August 8, 2023

4826

Client Background

Client: A leading fintech firm in the USA

Industry Type: Finance

Services: Financial services

Organization Size: 100+

Project Objective

Create a real-time Kibana dashboard to monitor the real-time movement and activities related to company/stock on the AWS to analyse data and get insights through dashboards to prevent due diligence. Dashboard should include visualizations of sentiments, FOIA requests, stock prices, volume, borrow rate, etc.

Project Description

Create real-time dashboards to get insights about the data and to analyse the relative change in different activities. Someone filing FOIA SEC request or FOIA FDA request and/or registering for conference calls might also have posted some negative tweets on tweeter to influence the market. Dashboard should display data of requests, sentiments, stock prices, etc on the same timeline, so that we will be able to observe the changes and relative changes with respect to time. Make separate dashboard for 2 stock symbols to analyse the activities and changes specific to that and a dashboard for all the data, eg. stocks, requests, etc. Change in sentiments effecting the price of the stock, borrow rate, trading volume, etc. should be noticeable. There is a list of names, make alert on the dashboard when the requests are filed by them on the same timeline used for other data. Also include the candlestick chart to view the stock details like open, close, high, low, volume with respect to time.

Our Solution

For FOIA SEC and FDA requests, made a metric chart representing the total number of requests and requesters, created a date histogram to view the frequency of requests and requesters with respect to time, bar chart to view the top requester name, organization, category, pie chart to view the proportion of final disposition of requests and tag cloud for the description of the requests for the entries present in the selected time range and a search table that contains the selected columns (only relevant ones) for both SEC filings and FDA filings.

Similarly, for citation data, created a date histogram to view the frequency of citations and names of firms who posted with respect to time and bar chart to view number of citations by firm in the selected time range and a search table that contains the selected columns (only relevant ones). Index containing fail to deliver data is used to plot the date histogram in which volume failed is represented by the bar along the line representing the price at that time, bar chart where bars represents the total volume failed to deliver with respect to stock symbol and average price of the stock symbol in the selected time range by a dot size add on and tag cloud of the stock symbol as per fail to delivers.

For twitter data (short seller’s data), made a pie chart to show the proportion of polarity, metric table to show the highest 10 average retweets with respect to user name, made a date histogram to show the frequency of tweets as per time and another date histogram representing the amount of positive and negative sentiments with the help of bars as per time to leverage us to observe if change in amount of sentiments is affecting price of stock, volume in trade and fail to deliver, etc., bar chart to show the total posts and number of posts in the selected time range and another bar chart to show the count of followers and friends in the index in selected time range. A search table is made with columns like polarity, follower counts, retweets and post with timestamp to get precise info of what we have in visualizations.

For the list of names to be tracked on requests made and to make alert for them, added a annotation on the TSVB graph and added all of these along with the above visualizations on the dashboard on Kibana to make it a real-time dashboard and we can use this dashboard to do relative analysis.

For the dedicated dashboards to the stock, created and added following visualizations:

Metric to show number of requests and requesters in FOIA SEC and FDA indexes where description contains terms related to that stock symbol or product of the company.
TSVB of FOIA SEC and FDA and added annotation where the request against the stock or company is filed.
Fail to deliver and price on the same timeline to notice the relative change.
Sentiment and stock details is to be added in these but the data isn’t ready yet from the client’s end.

Project Deliverables

3 dashboards- 1 dashboard for complete data and 2 dashboards dedicatedly for one stock each.

Tools used

Kibana and Elasticsearch

Skills used

Visualizations and analytical skills were used

Databases used

Following databases are used to:

FOIA SEC filings
FOIA FDA filings
Citations
Fail to deliver
Tweeter Short seller data
Stock price

Web Cloud Servers used

AWS Management Console

What are the technical Challenges Faced during Project Execution

As I was using Kibana and studying the stock data for the first time, I faced challenges in making complex visualizations and understanding the terms related to stock data. Using filters while making Vega Charts to make candlestick chart with inconsistent data was displeasing.

How the Technical Challenges were Solved

Challenges related to the creation of complex visualization was solved exploring options on the Kibana and getting reference from the online sources. In order to understand the stock information and how things work, I got immense amount of knowledge from the client and from my project manager. For filtering of data in Vega charts I took help from the online sources.