CaffeOnSpark brings deep learning onto Hadoop/Spark clusters. By combining salient features from deep learning framework Caffe and big-data framework Apache Spark, CaffeOnSpark enables distributed deep learning on a cluster of GPU and CPU servers.
As a distributed extension of Caffe, CaffeOnSpark supports neural network model training, testing and feature extraction. Caffe users could now perform distributed learning using their existing LMDB data files other format) and minorly adjusted network configuration (as illustrated).
CaffeOnSpark is a Spark package for deep learning. It is complementary to non-deep learning libraries MLlib and and Spark SQL. CaffeOnSpark's Scala API provides Spark applications with an easy mechanism to invoke deep learning (see sample) over distributed datasets.
CaffeOnSpark was developed by Yahoo for large-scale distributed deep learning on our Hadoop clusters in Yahoo's private cloud. It's been used by Yahoo for photo search, content classification and so on.
CaffeOnSpark provides some important benefits over alternative deep learning solutions.
- It enables model training, test and feature extraction directly upon Hadoop datasets.
- It turns your Hadoop or Spark clusters into a powerful platform for deep learning, without setting up new clusters.
- Server-to-server direct communication (ethernet or infiniband) achieves speedy learning, and eliminates scalability bottleneck.
- Caffe users' existing datasets (ex. LMDB) and configurations could be applied for distributed learning without conversions.
- High-level API empowers Spark applications to easily conduct deep learnings.
- Incremental learning is supported to leverage previously trained model or snapshots.
- Additional data formats and network interfaces could be easily added.
- It's easily deployabe at public clouds (ex. AWS EC2) and private cloud.
Please check CaffeOnSpark wiki site for detailed documentations including buiding instruction, and started guides for standalone cluster and EC2 cluster.
Please note:
- Batch sizes specified in prototxt files are per device.
- Memory layers should not be shared among GPUs, and thus "share_in_parallel: false" is required for layer configuration.
Please join CaffeOnSpark user group for discussions and questions.
The use and distribution terms for this software are covered by the Apache 2.0 license. See LICENSE file for terms.