Experiment tracking, ML developer tools

A curated list of awesome open source tools and commercial products for ML Experiment Tracking and Management 🚀

Autonomous ML research infrastructure for autoresearch by Karpathy. Multi-GPU parallelism, structured experiment tracking, adaptive search strategy.

Track and Collaborate on ML & AI Experiments.

Build and ship production ML pipelines faster: a pipeline library with an optional self-hosted visual layer for modular, reproducible workflows, local testing, and experiment tracking.

2 Lines of code to track ML experiments + EDA + check into Github

This project covers the end to end understanding for creating an ML pipeline and working around it using DVC for experiment tracking and data versioning(using AWS S3)

PyLabFlow is a Python framework for managing, tracking, and reproducing complex computational experiments. Built for researchers, data scientists, and ML engineers, it provides component-level lineage, modular pipelines, and offline-first execution, making it easy to run, compare, and debug hundreds of experiments.

An example of setting up local audio ML training pipeline on Airflow with MLFlow experiment tracking and custom python library.

A production-ready MLOps pipeline for real-time item purchase prediction on mobile e-commerce platforms. Built with modern ML infrastructure including feature stores, experiment tracking, orchestration, and observability.

High efficiency Local/self-hosted ML Experiment Tracking System

ML experiment tracker. Simple alternative for neptune.ai or Tensorboard.

This project promulgates an automated end-to-end ML pipeline that trains a biLSTM network for sentiment analysis, experiment tracking, benchmarking by model testing and evaluation, model transitioning to production followed by deployment into cloud instance via CI/CD

A production-grade MLOps pipeline for predicting telecom customer churn, featuring automated data preprocessing, ML model training, experiment tracking with MLflow, distributed training using PySpark, real-time inference via Kafka streaming, Airflow DAG orchestration, and Dockerized REST API deployment.

Custom ML tracking experiment and debugging tools.

An end to end ML project. Using MLflow for experiment tracking and model registry. Prefect for workflow orchestration. S3 for artifacts storage. AWS Lambda/ ECR for serverless model serving. AWS REST API gateway as endpoint to lambda function. GitHub Actions for CI/CD.

A kedro plugin that enables logging to the ml experiment tracker aim

Pythonic, type-safe search space configuration for HPO (hyperparameter optimization), NAS (neural architecture search), and ML experiment tracking. Define complex search spaces with conditional parameters, automatic validation, and zero boilerplate. Pydantic-based, Optuna-ready to nail hyperparameter tuning.

Comparison of ML Life Cycle Management (Experiment Tracking, Model Management, etc.): MLflow, DVC, Pachyderm, Sacred, Polyaxon, Allegro Trains, VertaAI ModelDB, Kubeflow Katib, Guild AI, Kubeflow Metadata, Weights & Biases, Neptune.ai, Valohai, Comet

Machine Learning Operations (MLOps) are essential to build successful Data Science use-cases. Today, ML is powering data driven use-cases that are transforming industries around the world. In order to seize and hold it's competitive advantage business needs to reduce risk therefore a new expertise rises to include data science models in operational systems. According to Gartner Research “While many organizations have experimented with AI proofs of concept, there are still major blockers to operationalizing its development. IT leaders must strive to move beyond the POC to ensure that more projects get to production and that they do so at scale to deliver business value. (July 2020)”. In this session, we will discuss the role of MLOps and how they can help data science models from deployment to maintenance with focus on: keep track of performance degradation overtime from model predictions quality, setting up continuous evaluation metrics and tuning the model performance in both training and serving pipelines that are deployed in production.

An end-to-end ML app that classifies chest diseases from CT scans using CNNs. It covers the full MLOps lifecycle with experiment tracking, pipeline orchestration, model versioning, CI/CD deployment, and a web interface for users.

A production-grade end-to-end ML system for predicting customer churn in retail banking. It uses an XGBoost model optimized for top-K targeting, with full lifecycle support including data validation, experiment tracking (MLflow), automated monitoring, drift-based retraining, and explainability via SHAP.

MLflow MCP server for ML experiment tracking with advanced querying, run comparison, artifact access, and model registry.

Extract provenance graphs compliant with W3C PROV from ML experiment projects that use Git repositories and MLflow tracking

End-to-end ML pipeline for housing price prediction with XGBoost, deployed on Google Cloud Run with Supabase integration. Features REST API, Streamlit dashboard, MLflow experiment tracking, and comprehensive testing.

ML-based engine failure prediction from sensor data (RPM, oil/coolant pressure & temp). Hugging Face data/model hub, XGBoost/RF, experiment tracking, GitHub Actions CI/CD, Docker, Streamlit on HF Spaces.

Tracking ML experiments using gin-config, wandb, comet.ml and S3.

Basics of ML experiments metrics & artifacts tracking with MLflow

Complete MLOps learning project: synthetic data generation, S3 integration, Airflow DAG orchestration, distributed PySpark processing, and MLflow experiment tracking. Containerized environment showcasing end-to-end ML pipeline automation.

A comprehensive, dockerized AI workflow system that enterprises can deploy immediately. This platform demonstrates industry best practices for AI/ML operations, including experiment tracking, model serving, workflow orchestration, and real-time monitoring.