Anomaly Detection System: Spark & ML

You are a ML engineer with expertise in advanced analytics. Design and implement a complete real-time anomaly detection system for analyzing supply chain optimization using Apache Spark for big data, handling small dataset (<1GB).

ANALYSIS REQUIREMENTS:
1. Data Collection Strategy: Sources, APIs, ETL pipelines
2. Data Preprocessing: Cleaning, transformation, feature engineering
3. Exploratory Data Analysis: Statistical summaries, visualizations, correlations
4. Model Development: Algorithm selection, training, validation, hyperparameter tuning
5. Model Evaluation: Metrics (accuracy, precision, recall, F1, ROC-AUC), cross-validation
6. Deployment: Production pipeline, monitoring, retraining strategy
7. Visualization: Interactive dashboards, reports, alerts
8. Documentation: Methodology, assumptions, limitations, recommendations

DELIVERABLES:
- Complete analysis code (Python/R/SQL scripts)
- Jupyter notebooks with explanations
- Data preprocessing pipeline
- Trained model files with evaluation metrics
- Interactive dashboard (Tableau/Power BI/Plotly)
- Statistical analysis report
- Model documentation
- Deployment guide
- Performance monitoring setup

Include data preprocessing steps, feature engineering techniques, model selection rationale with comparisons, interpretation guidelines, and actionable business insights. Make it production-ready with proper error handling and monitoring.

BONUS: Add troubleshooting section and common pitfalls to avoid. [Ref: 8cc02487]