Intelligent Pipelines for the Speed-of-Money Era
DOI:
https://doi.org/10.5281/zenodo.20578055Keywords:
Generative Artificial Intelligence, GenAI Enabled DevOps, Real Time Transaction Processing, Financial Services Pipelines, Black Swan Event Handling, Enterprise Architecture Theory, Cloud Native Architectures, Transaction Pipeline Scalability, DevOps Automation Practices, Real Time Data Processing, Operational Resilience Engineering, Non Functional Requirements Analysis, Intelligent Pipeline Monitoring, Adaptive System Design, Financial Infrastructure Modernization, Event Driven Architectures, High Volume Transaction Systems, AI Assisted Software Operations, Architectural Patterns And Trade Offs, Enterprise Grade Transaction Systems.Abstract
The emergence of Generative AI (GenAI) unlocks new capabilities in software development and operations. A related trend calls for the adoption of DevOps principles and practices. Furthermore, organizations across industries seek to process data in real time. In financial services, that imperative applies to transaction pipelines, where volumes peak at Black Swan events, yet the average workload is relatively modest. Ideas from GenAI-enabled DevOps can accelerate the design, implementation, and monitoring of such pipelines, but tensions remain. The landscape of existing work is therefore reviewed, gaps identified, Generalized Enterprise Architecture Theory summoned, and research questions posed. Addressing them will yield candidate solutions that reconcile GenAI-enabled DevOps principles with the requirements of real-time transaction processing. A foundation for that exploration is laid by: specifying the functional and nonfunctional requirements for such pipelines; providing an overview of GenAI-driven DevOps concepts, architectures, and operational paradigms amenable to financial Pipelines; and articulating suitable patterns, together with the associated challenges and trade-offs.
Real-time transaction processing pipelines in financial services must support Black Swan event volumes while meeting operational requirements at lower loads. Ideas from GenAI-enabled DevOps can accelerate the design, implementation, and monitoring of such pipelines, but tensions remain. Existing work is reviewed, gaps identified, Generalized Enterprise Architecture Theory summoned, and research questions posed. Addressing them will yield candidate solutions reconciling GenAI-enabled DevOps principles with the requirements of real-time financial transaction processing. A foundation for that exploration is laid by specifying the functional and nonfunctional requirements for such pipelines; providing an overview of GenAI-driven DevOps concepts, architectures, and operational paradigms amenable to financial pipelines; and articulating suitable patterns, together with the associated challenges and trade-offs.
References
[1] Ambler, S. W., & Lines, M. (2012). Disciplined agile delivery. IBM Press.
[2] Arner, D. W., Barberis, J., & Buckley, R. P. (2017). FinTech and RegTech in a nutshell. Northwestern Journal of International Law & Business, 37(3), 371–413.
[3] Bass, L., Weber, I., & Zhu, L. (2015). DevOps: A software architect’s perspective. Addison-Wesley.
[4] Brewer, E. A. (2012). CAP twelve years later. Computer, 45(2), 23–29.
[5] Buchak, G., Matvos, G., Piskorski, T., & Seru, A. (2018). Fintech, regulatory arbitrage, and the rise of shadow banks. Journal of Financial Economics, 130(3), 453–483.
[6] Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile Networks and Applications, 19(2), 171–209.
[7] Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58.
[8] Duffie, D. (2020). Intermediation of payments in the digital age. Journal of Economic Perspectives, 34(4), 119–140.
[9] Fan, W., & Bifet, A. (2013). Mining big data. ACM SIGKDD Explorations Newsletter, 14(2), 1–5.
[10] Frost, J., Gambacorta, L., Huang, Y., Shin, H. S., & Zbinden, P. (2019). BigTech and the changing structure of financial intermediation. Economic Policy, 34(100), 761–799.
[11] Gomber, P., Koch, J. A., & Siering, M. (2017). Digital finance and fintech. Journal of Business Economics, 87(5), 537–580.
[12] Hohpe, G., & Woolf, B. (2004). Enterprise integration patterns. Addison-Wesley.
[13] Humble, J., & Farley, D. (2010). Continuous delivery. Addison-Wesley.
[14] Janssen, M., & Kuk, G. (2016). Big data algorithms in public sector decision-making. Government Information Quarterly, 33(3), 371–377.
[15] Kleppmann, M. (2017). Designing data-intensive applications. O’Reilly Media.
[16] Kreps, J., Narkhede, N., & Rao, J. (2019). Kafka: A distributed messaging system. ACM Queue, 17(2), 20–44.
[17] Kuner, C., Bygrave, L. A., & Docksey, C. (2017). The EU GDPR. Oxford University Press.
[18] Lessmann, S., Baesens, B., Seow, H. V., & Thomas, L. C. (2015). Benchmarking classification models for credit scoring. European Journal of Operational Research, 247(1), 124–136.
[19] Margetts, H., & Dorobantu, C. (2019). Rethink government with AI. Nature, 568(7751), 163–165.
[20] Milne, A., & Parboteeah, P. (2016). The business models of peer-to-peer lending. European Credit Research Institute.
[21] Newman, S. (2015). Building microservices. O’Reilly Media.
[22] OECD. (2019). Regulatory effectiveness in the era of digitalisation. OECD Publishing.
[23] Pasquale, F. (2015). The black box society. Harvard University Press.
[24] Philippon, T. (2016). The fintech opportunity. BIS Working Papers, 655, 1–33.
[25] Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018). Data management challenges in production machine learning. ACM SIGMOD Record, 47(3), 34–43.
[26] Ruff, L., Kauffmann, J. R., Vandermeulen, R., et al. (2021). A unifying review of deep anomaly detection. Proceedings of the IEEE, 109(5), 756–795.
[27] Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 2494–2502.
[28] Shneiderman, B. (2020). Human-centered artificial intelligence. International Journal of Human–Computer Interaction, 36(6), 495–504.
[29] Stonebraker, M., & Cetintemel, U. (2005). One size fits all. Proceedings of the International Conference on Data Engineering, 2–11.
[30] Tanenbaum, A. S., & Van Steen, M. (2017). Distributed systems (3rd ed.). Pearson.
[31] Van der Aalst, W. (2021). Process mining and real-time analytics. Communications of the ACM, 64(8), 76–83.
[32] Wamba, S. F., Gunasekaran, A., Akter, S., Ren, S. J. F., Dubey, R., & Childe, S. J. (2020). Big data analytics and firm performance. Journal of Business Research, 70, 356–365.
[33] Zaharia, M., Das, T., Li, H., et al. (2016). Apache Spark. Communications of the ACM, 59(11), 56–65.
[34] Zetzsche, D. A., Buckley, R. P., Arner, D. W., & Barberis, J. (2017). From fintech to techfin. Journal of Financial Regulation, 3(2), 223–254.
[35] Zuiderwijk, A., Chen, Y. C., & Salem, F. (2021). Implications of big data analytics for public governance. Government Information Quarterly, 38(4), 101577.
Additional Files
Published
Data Availability Statement
None
