SR

Work

Amazon Textract async processing

I designed and built the core document processing service behind Textract's asynchronous APIs, as part of a re-architecture of how those jobs run.

Role
Software Development Engineer
Team
Amazon Textract, AWS
When
2021 to 2023
Scale
100M+ documents a month, 15 regions

The product

Amazon Textract reads documents such as forms, invoices and multipage PDFs and returns the text and structure it finds. Large and multipage documents go through its asynchronous APIs.

A customer puts a document in Amazon S3 and calls a start operation like StartDocumentAnalysis. Textract returns a job ID straight away, does the work in the background, and publishes a completion notice to an Amazon SNS topic. The customer then fetches the results with the matching get operation, such as GetDocumentAnalysis.

That simple contract hides a lot. Every document a customer sends has to be picked up, processed and accounted for, whatever the traffic looks like.

How an asynchronous Textract job flows Your app starts a job through the Textract async API. The job goes through a queue to the core processing service, running in Docker on EC2, which uses DynamoDB. When it finishes, a completion notice goes to an SNS topic, and your app fetches the results through the API. Your app Textract async API Queue Core processing Docker on EC2 the part I designed DynamoDB SNS topic start a job get results job finished
Simplified. The request flow comes from the public Textract documentation. Inside, the diagram shows only the building blocks named on this page and leaves out internal detail.

What I did

I was part of the async re-architecture project. My piece was the service that does the core processing of every document customers send to the asynchronous APIs.

  • Owned the system design for the service, then built it.
  • Hosted it as Docker containers on Amazon EC2.
  • Used queues and databases such as DynamoDB to move work through the system reliably.
  • Ran it for customers in 15 AWS regions, at a scale of more than 100 million documents a month.

Built with

  • Distributed systems
  • Python
  • Docker
  • Amazon EC2
  • Queues
  • Amazon DynamoDB