Monitoring

Currently in production

  • Automated e-mails instances level alerts for excessive memory use

  • Daily automated e-mail report on instance disk usage

  • Daily e-mails reports on successful models run for each concept and each model, including latest data points for data integrity review

  • Models recalibration reports

  • Data integrity reports

  • Automated e-mail alerts if any exception on models and models re-calibration run

  • Unit testing

  • Instances connections with static IP and GCP security group

  • GCP 2FA

  • All running software continuously logs information

Planned for version 2

  • Automated CI/CD with integrity testing

  • Daily data visualization reports at multiple levels

  • Hard back-ups

  • Disk encryption

  • Slack status reports on quant data pipeline and quant infrastructure

  • Slack and phone alerts for servers and models run failures and critical exceptions

  • Servers and data back-up strategy

  • DevOps to design security policy (intrusion/hacking, firewalls, etc)

  • The production will be split across multiple servers (e.g.: 3 servers with 33% loading each), some of them will be production and some will be beta. When there is a code change, the beta server will be turnover first. Conditional on successful testing, the production servers will be turned over in turn

PreviousTech stackNextPerformance monitoring system

Last updated