DevOps and CodeOps Tools
DevOps and CodeOps tools integrated for cloud deployment and automation.
Overview
The Codeops and DevOps tool started to become essential as the environment started to grow. Over the course of about 4-6 weeks due to the code generation, the platform balooned from about 30K lines to almost 90K lines of code as the data model was fully flushed out. At the same time a lot of the hand written code was refactored into a cohesive set of libraries that were reusable and Django was transitioned into a more sophisticated front and backend with multiple sub projects (common_features, owner and tenant). This combined with the introduction of the new laptop basically resulted in having to stop and clean up everything. Admittedly, the stop and clean up everything happened about 6-7 times over the course of the project , approximately every 2-3 weeks as the scope grew and I could ‘see’ more of the platform taking shape it would become clear that a new abstraction or better organization was required. Every time I ended up doing the cleanup, I would reflect that in an enterprise, so much of this would be given to me on day 1 (project organization, devops, codeops, processes). In reality, I would probably have considered it a pain in the neck. Having had to abstract on the go and re-organize on the go , I defintiely think of it differently now.
Key Goals:
Make it easy to recover when I move between machines or delete something
Make it easy to deploy
Make it easy to use the platform and not have so many stickie notes and steps to get going
Complexity: Medium
Components
GitHub
Implemented version control, branching strategies, and project tracking.
SOARL Summary
Required best-practice code management across multiple machines and services.
Keeping environments in sync** was occasionally problematic due to initial poor folder structuring.
Ensured no sensitive information (API keys, credentials) leaked into GitHub.
Set up .gitignore properly to exclude sensitive files.
Refactored folder structures to support multi-service development.
Now seamlessly moves between machines without syncing issues.
Proper Git hygiene matters—especially when managing multiple services (Django, FAISS, Redis, NLP utilities, etc.**).
Situation:
Obstacle:
Action:
Result:
Learning:
Docker
Containerized Django, FAISS, and supporting services for streamlined deployment.
SOARL Summary
Introduced Docker after managing too many terminal sessions manually—services needed consolidation.
Docker setup required careful structuring—ensuring correct **directory paths, variables, and dependencies.
Refactored hardcoded configurations to prepare for GCP deployment.
Created optimized Docker images, reducing bloated builds from 9GB to ~1GB.
Separated environments into distinct virtual environments, improving modularity.
Clean, lightweight Docker containers** make deployments faster and easier to manage.
Breaking services into optimized environments** makes Docker builds leaner and faster.
Situation:
Obstacle:
Action:
Result:
Learning:
Terraform
Automating cloud infrastructure provisioning for consistent, cost-efficient deployments.
SOARL Summary
Needed a repeatable way to deploy cloud resources without manual configuration.
Ensuring Terraform correctly managed** cloud permissions and resource dependencies.
Set up Terraform scripts to define GCP configurations.
Deployment consistency improved, and cost tracking became easier.
Infrastructure as Code (IaC) reduces cloud misconfigurations and prevents unnecessary cost spikes**.
Situation:
Obstacle:
Action:
Result:
Learning:
Kubernetes
Managing Django, FAISS, and Redis as orchestrated services within a Kubernetes cluster.
SOARL Summary
Required a way to scale and manage containerized services in production.
Learning Kubernetes configurations, deployments, and networking.
Set up a Google Kubernetes Engine (GKE) cluster to manage workloads efficiently.
Services run seamlessly in an orchestrated environment, improving scalability.
Kubernetes simplifies multi-service management** but requires careful setup to optimize scaling.
Situation:
Obstacle:
Action:
Result:
Learning:
Key Learnings
- In the 0-1 scenario, you dont know what you dont know. Especially when you are creating on the fly. I certainly had no idea I was going to have a multi-tenant platform on day 1 when I created my initial python libraries and scripts. - Each time I had to go back and cleanup the environment (ENV Setup ; Directory Structures; Virtual Environments, Multiple ENV + Docker; Hard Coding directories + Directory Structure; DB and Redis Connections; Logging and Error Handling) it became easier to do and also things made more sense.
Demos
Final Thoughts
As soon as you go beyond a few scripts having an efficiently organized project folder is essential. Even though I had brainstormed on many of these items before I started with an initial guess, I ultimately needed to have the platform take shape before I was able to optimize for it 🚀