This diagram illustrates a secure, event-driven data ingestion pipeline using AWS Transfer Family with a custom identity provider (IDP) and tight access control mechanisms. The solution is designed to accept SFTP and AS2 connections from external servers and route incoming files into an Amazon S3 bucket after enforcing authentication, IP validation, and logging.
🔄 Overview of Flow:
Connection Initiation:
An external server initiates an SFTP or AS2 session with AWS Transfer Family.
Authentication Process:
AWS Transfer invokes a Custom IDP Lambda Function to authenticate the user.
This Lambda fetches credentials/configuration from AWS Secrets Manager.
Security Check:
Lambda communicates with Amazon API Gateway, which acts as an access control layer.
API Gateway calls AWS WAF to check the IP address.
If the IP is blocked, WAF returns an event and terminates the connection.
Successful Login:
If login succeeds, the file transfer continues.
Uploaded files are stored in an Amazon S3 Landing Bucket.
Monitoring and Auditing:
All events (logins, IP checks, file uploads) are monitored via Amazon CloudWatch.
Events (success/failure) are logged at each stage (e.g., login result, IP block, object upload).
✅ Key Goals Achieved by This Architecture:
Objective and How It’s Achieved
Secure Authentication Custom Lambda IDP using secrets stored in AWS Secrets Manager
Granular Access ControlIP address validation via API Gateway + AWS WAF
Protocol SupportSupports both SFTP and AS2 via AWS Transfer
ScalabilityServerless components (Lambda, API Gateway, S3) make it scalable
ObservabilityIntegrated CloudWatch logging and event monitoring
Isolation & AuditingIdentity and IP logic decoupled from AWS Transfer itself
💡 Typical Use Case:
This setup is ideal for enterprise file transfer workflows, especially where:
Regulatory compliance requires strict access control
Integration with legacy systems using SFTP or AS2 is necessary
Per-customer authentication and isolation is required (e.g., in B2B data exchanges)