The output of this work fed S&P Global Market Intelligence data products used by financial institutions. I wasn't client-facing. My job was to turn research that needed rooms full of people into crawlers and bots that ran on their own.
Finding social handles for 3.4 million companies
Researchers were collecting companies' official social media handles, on Twitter and other networks, by visiting company websites one at a time. Across S&P's company universe that meant millions of visits.
I wrote a Python and Selenium crawler. It ran across 3.4M+ company websites. Two things made it usable rather than just big. Domain resolution made sure it hit the right site for each company. De-duplication left each company with one clean answer.
It replaced about 160 FTE of manual research and fed a data product with $1.1M of potential revenue. It was the largest of my S&P automations, which together came to about 168 FTE of capacity.
A filings crawler across 40+ state portals
Financial filings were spread across U.S. state portals, each with its own search, its own layout and its own PDFs. I built a crawler that covered 40+ of them: it found the filings, downloaded the PDFs, pulled the text out with pypdf and PDFMiner, and translated regional filings. It saved close to 3 FTE.
It's also where I learned the lesson behind my later work. Each portal had its own script, and every redesign on any one of them meant a code fix, a test and a deployment. That's the problem the config-driven RPA engine solves, and I wrote about the shift in why I stopped writing one bot per website.
The rest of the toolkit
- ESG document scraper: collected ESG documents for data products. About 1.2 FTE and $600K of potential revenue.
- Search automation: $400-800K of potential revenue.
- Website change watcher on Blue Prism and Appian: RPA that tracked changes on monitored sites. About 1.2 FTE.
- MySQL reporting automation: cut reporting effort by about half.
Working with Blue Prism and Appian next to plain Python gave me a practical view of platform RPA versus code-first automation. I compare the two in Playwright vs Blue Prism and UiPath.
Recognition
- Essential Excellence Data Award, Q3 2020
- Breakthrough Achievement Award, 2019
- Rising Star, Q2 2018
- EssentialTECH Foundations badge, 2021
Stack
Python, Selenium, pypdf, PDFMiner, MySQL, Blue Prism, Appian.
Role: Senior Automation Specialist, S&P Global Market Intelligence, Hyderabad, September 2017 to July 2021. Revenue figures are potential revenue of the data products the automations fed.