Optimizing a Machine Learning Pipeline to Reduce Run Time

The discovery phase of this project required an in-depth analysis of the firm’s existing monolithic process which included reviewing the code and technology stack to ensure the appropriate tooling was in place. In this situation, the pipeline included Python and several popular machine learning libraries, but relied heavily on NoSQL and Amazon Relational Database Service (RDS) accessed with long running EC2 instances as opposed to AWS’ Lambda service.

Parallelize the machine learning data pipeline’s processes using Amazon Web Services (AWS) Lambda

After the discovery phase, Six Feet Up’s expert team of developers hit the ground running to execute an optimization strategy that would leverage the physical server aspect of serverless computing. Specifically, the team:

• separated the code base into individual elements,
• spun up parallel machines with the same code base elements,
• hid cached data in Lambdas virtual file systems,
• allowed Python to start off from Lambda’s “disks” of cached data, and
• customized large machine learning libraries to reduce their sizes.

Deploy machine learning models with appropriate monitoring and alerting services to reduce troubleshooting turn-around time

By parallelizing and componentizing the process behind the pipeline, Six Feet Up produced results faster than the systems development life cycle (SDLC), which allows for quicker reconfigurations and optimizations to the process. The cost to keep multiple machines busy with no room for system errors versus parallelizing the process is negligible, but the time and effort saved account for considerable cost savings.