Back to portfolio

Policy Change Monitor: a Serverless Diff of Public Terms Pages

A Python job on AWS Lambda that watches about 28 public terms, privacy and cookie pages, finds exactly what changed, keeps every version in DynamoDB and emails the diff for review.

pythonaws lambdadynamodbsesbeautifulsoupweb scraping
Policy Change Monitor: a Serverless Diff of Public Terms Pages
The diff in the image is sample text. The page names in the watch list were public pages; the client is not named.

The problem

The client had to know when any of dozens of public policy pages changed. Checking them by hand meant changes were found late or not at all.

What I built

  • Watch list of pages, each with the CSS selector for its main text, so menus and footers never show up as changes.
  • Fetch and extract with requests and BeautifulSoup inside Lambda.
  • Compare against the last stored version with a unified diff, and skip pages that did not change.
  • Report: each change is stored as a new version and an SES email with the diff goes to every subscriber.
  • Review API: API Gateway endpoints mark a change as reviewed, lock it while someone is on it, and log who reviewed it and when.

Stack

Python, AWS Lambda, API Gateway, DynamoDB, SES, BeautifulSoup, difflib

Message me on the platform where you found this and I will answer there.