This diagram illustrates a secure, event-driven data ingestion pipeline using AWS Transfer Family with a custom identity provider (IDP) and tight access control mechanisms. The solution is designed to accept SFTP and AS2 connections from external servers and route incoming files into an Amazon S3 bucket after enforcing authentication, IP validation, and logging.

πŸ”„ Overview of Flow:

  1. Connection Initiation:

    • An external server initiates an SFTP or AS2 session with AWS Transfer Family.

  2. Authentication Process:

    • AWS Transfer invokes a Custom IDP Lambda Function to authenticate the user.

    • This Lambda fetches credentials/configuration from AWS Secrets Manager.

  3. Security Check:

    • Lambda communicates with Amazon API Gateway, which acts as an access control layer.

    • API Gateway calls AWS WAF to check the IP address.

    • If the IP is blocked, WAF returns an event and terminates the connection.

  4. Successful Login:

    • If login succeeds, the file transfer continues.

    • Uploaded files are stored in an Amazon S3 Landing Bucket.

  5. Monitoring and Auditing:

    • All events (logins, IP checks, file uploads) are monitored via Amazon CloudWatch.

    • Events (success/failure) are logged at each stage (e.g., login result, IP block, object upload).

βœ… Key Goals Achieved by This Architecture:

Objective and How It’s Achieved

Secure Authentication Custom Lambda IDP using secrets stored in AWS Secrets Manager

Granular Access ControlIP address validation via API Gateway + AWS WAF

Protocol SupportSupports both SFTP and AS2 via AWS Transfer

ScalabilityServerless components (Lambda, API Gateway, S3) make it scalable

ObservabilityIntegrated CloudWatch logging and event monitoring

Isolation & AuditingIdentity and IP logic decoupled from AWS Transfer itself

πŸ’‘ Typical Use Case:

This setup is ideal for enterprise file transfer workflows, especially where:

  • Regulatory compliance requires strict access control

  • Integration with legacy systems using SFTP or AS2 is necessary

  • Per-customer authentication and isolation is required (e.g., in B2B data exchanges)

AWS Transfer Family Custom Identity Provider (IDP)

Next
Next

Customer Data Import Gateway