How FICO modernizes file transfers with ETL automation using AWS Transfer Family

FICO powers decisions that help people and businesses around the world prosper. Using FICO solutions, businesses in more than 80 countries do everything from protecting four billion payment cards from fraud, to improving financial inclusion, and increasing supply chain resiliency. As a global leader in credit scoring and analytics, FICO processes massive volumes of sensitive financial data through Managed File Transfers (MFT), which makes secure and efficient file transfer capabilities crucial to their operations.

True MFT solutions extend beyond basic file upload and download functionality, serving as essential components for data processing across applications. However, traditional MFT implementations demand substantial infrastructure. This includes SFTP servers, databases, proxy instances, and Business Logic execution nodes, which lead to high fixed operational costs regardless of actual usage. Like many enterprises facing these challenges, FICO’s journey with MFT has evolved from conventional commercial solutions and custom-built systems toward a modern, cloud-native approach. This transformation led FICO to develop a new cloud-based MFT solution with AWS Transfer Family that removes scaling limitations and licensing costs while supporting FICO Platform’s efficiency and scalability.

In this post, we explore how FICO addressed challenges in external transfer security, data governance, and resource scaling through a serverless architecture that eliminates traditional MFT infrastructure constraints. The solution demonstrates how Transfer Family, supported by other serverless architecture components, can transform legacy file transfer systems into a modern, cost-effective platform that automatically processes files based on configurable business rules, while providing the flexibility to handle both direct and complex workflows across global operations.

Legacy MFT solution challenges

Secure and efficient file transfer capabilities are crucial for enterprise operations. However, traditional MFT solutions are increasingly becoming a bottleneck rather than an enabler. FICO’s journey mirrors what many organizations discover: legacy MFT infrastructure, despite years of reliable service, struggles to meet the demands of modern digital operations. The following challenges are associated with the Legacy MFT solution:

• High fixed operational costs:
  • Traditional MFT solutions needed substantial always-running infrastructure such as SFTP servers, databases, proxy instances, and execution nodes.
  • These systems incurred large, fixed operating costs regardless of actual usage, making them financially inefficient during periods of low usage.
• Limited scalability and licensing constraints:
  • Expanding the traditional infrastructure necessitated further licensing and manual provisioning of resources, thereby increasing costs and operational risks.
  • Scaling infrastructure resources and system modifications was time consuming, and the infrastructure couldn’t efficiently adapt to varying workload demands.
• Complex infrastructure management:
  • The legacy MFT system demanded extensive maintenance of multiple components and systems, necessitating dedicated resources for ongoing management.
  • Manual interventions were frequently needed for routine operations, making the system labor-intensive and prone to human error.

Further requirements

Service integration: FICO needed a robust solution to bridge the integration gap between AWS Transfer Family Connectors and AWS Step Functions, because there was no direct integration available between these services at the time of writing. The system needed to maintain workflow states and track transfers reliably while making sure of seamless communication between components. This necessitated a mechanism to store task tokens, initiate transfers, and monitor events to update workflow status effectively. AWS recently announced file transfer status, the capability to query the transfer status directly from Step Functions. In the future, service integration between AWS connectors and Step Functions can be streamlined by directly polling the transfer status for communication between components. The MFT workshop provides guidance on building this kind of solution.

Configuration management standardization: A standardized approach for managing MFT configurations across the enterprise was essential. The requirement called for a REST API implementation that could handle both user and configuration management through AWS Lambda proxy integration. The system needed to store configurations reliably while providing seamless integration with FICO’s existing Continuous Integration and Continuous Delivery/Deployment (CI/CD) tools, making sure of consistent deployment and management processes.

Global deployment automation: To address the complexity of global deployments, FICO needed an infrastructure automation solution capable of managing over 145+ AWS resources through code. The requirement specified the ability to deploy to new regions with minimal configuration changes while maintaining infrastructure consistency. The solution needed to support component-level updates and make sure of reliable state management across multiple regions.

Usage tracking and cost attribution: In the multi-tenant MFT environment, FICO needed precise customer resource usage tracking, distinct applications per MFT instance, and comprehensive resource tagging. The system needed to track detailed metrics such as bandwidth usage, Step Functions transitions, Lambda executions, and storage consumption, with integration to both HAQM DynamoDB audit tables and Splunk through HAQM CloudWatch subscriptions

Solution overview

FICO developed an MFT solution with Transfer Family that addresses key challenges in external transfer security, data governance, ETL automation, and resource scaling. The architecture uses Transfer Family for managing SFTP endpoints, while using HAQM S3 and HAQM Elastic Block Store (HAQM EBS) for efficient storage solutions. The solution orchestrates business logic and ETL processes through Step Functions, with Lambda handling short-running processes and AWS Fargate, managing long-running tasks. Transfer Family connectors enable seamless external file operations, while the infrastructure is supported by DynamoDB for configuration and logging, HAQM API Gateway for RESTful interfaces, and CloudWatch for comprehensive monitoring. Security is maintained through AWS Secrets Manager for secure credential management. Integrating these serverless technologies into a unified architecture allows FICO to create a secure, cost efficient, highly available, and scalable cloud-native MFT solution that effectively addresses traditional MFT challenges while meeting modern operational demands.

The MFT solution architecture consists of three layers:

1. File transfer layer
   • Transfer Family manages SFTP endpoints
   • HAQM S3 provides durable file and object storage
   • Makes sure of secure data transfer and storage
   • External transfers through Transfer Family Connectors

2. Processing orchestration layer

• • HAQM S3 events trigger file detection
  • HAQM Simple Queue Service (HAQM SQS) queues manage processing requests
  • Step Functions coordinate ETL workflows
  • DynamoDB stores configuration data

3. Execution layer
   • Lambda processes time-sensitive tasks
   • Fargate handles compute-intensive operations such as:
     • PGP encryption/decryption
     • File compression

The following diagram shows the logical design and overview.

File transfer and processing workflow

External users securely connect through Transfer Family, which provides SFTP endpoints mapped to designated S3 buckets. This mapping makes sure that users only access their authorized directories.

When files are uploaded to HAQM S3, the service automatically generates events that are captured and routed to regional SQS queues. This event-driven architecture makes sure of reliable message handling and enables asynchronous processing of uploaded files across different regions. Step Functions read the event messages and consult DynamoDB tables to determine the appropriate ETL workflow defined for the file and HAQM S3 prefix (landing location).

If there is an ETL tasks defined, then the State Machine starts looping through the defined tasks in the specified order, and employs either Lambda for quick operations (under lambda time limit) or Fargate for longer-running tasks. This dual approach optimizes cost and performance while handling various operations such as PGP encryption/decryption, compression, and other actions/tasks.

When the ETL task is initiated, State Machine waits for feedback from the Lambda function or Fargate tasks to indicate success before removing the completed tasks from the list of pending actions and continuing to the next sequence.

In the event where an external SFTP push is defined, the Lambda function triggers a Transfer Family Connector to initiate a transfer using the passed/current file, the defined SFTP server and port, and the authentication defined in the Connector.

The following diagram shows the solution design diagram with processing flow using AWS services.

Results and benefits

As a result of building this MFT solution, FICO experienced the following business impact.

Enhanced service integration: The implementation of a dual-Lambda solution with DynamoDB state management successfully bridged the integration gap between AWS Connectors and Step Functions. This approach makes sure of reliable transfer tracking while maintaining workflow states. The solution’s ability to store TaskTokens, initiate transfers, and monitor CloudWatch events for workflow updates has created a robust and dependable file transfer system that maintains consistent workflow states throughout the process. There’s an opportunity to further streamline the architecture by directly tracking the transfer status from Step Functions using the new status querying capability offered by SFTP connectors.

Streamlined configuration management: The REST API implementation using API Gateway and Lambda proxy integration has transformed MFT configuration management. This standardized approach delivers a unified administrative interface that integrates seamlessly with FICO’s internal CI/CD tools. The use of DynamoDB as the configuration store makes sure of reliable and consistent configuration management across all deployments, significantly reducing administrative complexity and improving system reliability.

Optimized global infrastructure management: The solution’s implementation of Terraform for infrastructure automation has enabled successful management of over 145+ AWS resources through code. This automation facilitates rapid deployment of new regions with minimal configuration changes, making sure of infrastructure consistency while supporting component-level updates. The state-based approach has significantly reduced operational overhead in multi-region deployments and improved overall infrastructure management efficiency.

Precise cost attribution system: The implementation of AWS myApplications has enabled distinct application separation per MFT instance, facilitating baseline cost segregation through AWS resource tagging. The solution provides comprehensive tracking of customer-specific metrics through DynamoDB audit tables and Splunk integration through CloudWatch subscriptions. This detailed tracking system encompasses bandwidth usage, Step Functions transitions, Lambda executions, and storage consumption, enabling precise cost allocation in the multi-tenant environment.

Improved operational efficiency: The modern MFT solution has achieved remarkable improvements in operational efficiency, reducing the new environment deployment time from weeks to mere minutes. The serverless architecture has eliminated traditional infrastructure maintenance needs while enhancing scalability and reducing operational costs. The solution’s automated administration and operations tasks in AWS have resulted in significant efficiency gains in file integration processes, supporting hundreds of external business partners and internal data sources.

Conclusion

FICO’s transformation of their legacy MFT infrastructure into a modern, cloud-native solution represents a significant achievement in enterprise file transfer modernization. Using AWS Transfer Family as the cornerstone, combined with serverless components including AWS Lambda, AWS Step Functions, and AWS Fargate, allows FICO to successfully create a robust, scalable solution that handles complex file transfer workflows across global operations.

The implementation has delivered substantial benefits across multiple dimensions, most notably reducing the deployment time from weeks to minutes and eliminating fixed infrastructure costs. The solution now successfully manages over 145+ AWS resources through code, facilitating seamless integration with external business partners while maintaining precise cost allocation in their multi-tenant environment.

To learn more, visit AWS Transfer Family Managed File Transfer.