Add Ingestion Job management API for Feast Core - #548

Merged

feast-ci-bot merged 66 commits into

feast-dev:masterfrom

mrzzy:job-management-api

Apr 7, 2020

Commits on Apr 6, 2020

Added protobuf definitions for a Job Management API
Zhu Zhanyan
committed
1e1abb1
Added query methods to JobRepository to query jobs by store and featureset
Zhu Zhanyan
committed
bd04076
Added hashCode() & equals() to Job model compare and hash jobs
Zhu Zhanyan
committed
643f042
Added toIngestionProto() to Job object to convert Job model to ingestion job proto
Zhu Zhanyan
committed
f4b417e
Added query methods to FeatureSetRepository to query by exact (name, project) or (name,version)
Zhu Zhanyan
committed
9d62a38
Added listJobs() to JobService to handle request list ingestion jobs requests
Zhu Zhanyan
committed
32a0bb5
Added missing filter field in ListIngestionJobsRequest protobuf
Zhu Zhanyan
committed
4f55012
Added code to setup mockups for testing JobService
Zhu Zhanyan
committed
b3aa72a
Revert "Added hashCode() & equals() to Job model compare and hash jobs"
Zhu Zhanyan
committed
06b84df
Added JobServiceTest unit tests for JobService's listJobs()
Zhu Zhanyan
committed
c4b901a
Added stopJobs() to JobService to handle requests to stop jobs
Zhu Zhanyan
committed
cd0bde2
Added getTransitionalStates() to JobStatus to return collection of transitional states
Zhu Zhanyan
committed
c59035f
Changed stopJobs() to throw unsupported error on transitional job statuses
Zhu Zhanyan
committed
898045c
Moved conversion of JobStatus to IngestionJobStatus proto to JobStatus.
Zhu Zhanyan
committed
f4711cd
Make findFeatureSet() match only one featureset.
Zhu Zhanyan
committed
fe744ea
Added matchFeatureSets() to SpecService to match Feature Sets from References
Zhu Zhanyan
committed
a79f8b4
Refactor JobService listJobs() to use SpecService to provide featureset matching
Zhu Zhanyan
committed
5b25929
Revert findBy methods added to FeatureSetRepository as no longer used.
Zhu Zhanyan
committed
1997bb1
Added restartJob() to JobManagers to restart ingestion/import jobs
Zhu Zhanyan
committed
f320e38
Added restartJob() to JobService to restart ingestion jobs
Zhu Zhanyan
committed
0ec5afb
Update job model (due to new extId) when restartJob() in JobService
Zhu Zhanyan
committed
83cc4c7
Use assertThat() & equalTo() instead of assertEquals() in JobService
Zhu Zhanyan
committed
cf6a9c4
Revert "Use assertThat() & equalTo() instead of assertEquals() in JobService" due to failed tests
Zhu Zhanyan
committed
8088b33
Use hamcrest’s assertThat() and equalTo() instead of Junit's assertEquals
Zhu Zhanyan
committed
bdc2803
Hook up JobService list, stop, restart job methods to CoreServiceImpl.
Zhu Zhanyan
committed
fafcd11
Throw InvalidArgumentException instead when calling listJobs() with invalid FeatureSetReference
Zhu Zhanyan
committed
7a97e1f
Fixed typos in javadocs: InvalidArgumentException should be IllegalArgumentException
Zhu Zhanyan
committed
d277f93
Fixed in findByFeatureSetsIn() query in JobRepository
Zhu Zhanyan
committed
255946b
Renamed toIngestionProto() methods to toProto() to follow code convention
Zhu Zhanyan
committed
09e9ddc
Make JobService's listJobs() a transaction to prevent DB data race conditions
Zhu Zhanyan
committed
c85da17
Moved matchFeatureSets() in SpecService to private method in JobService
Zhu Zhanyan
committed
c876e40
Make the map that maps between JobStatus and IngestionJobStatus static
Zhu Zhanyan
committed
1799c3a
Make JobService's listJobs() to return all ingestion jobs on empty filter
Zhu Zhanyan
committed
1bead61
Fixed issue where the jobManager map that JobService built used wrong keys
Zhu Zhanyan
committed
19994ea
Fix issue where actual Job Status is not synced with database.
Zhu Zhanyan
committed
ff73b5f
Log stopJob() & restartJob() operations to make debugging easier
Zhu Zhanyan
committed
c207f1a
Use Runner.name() instead of runner.toString() to build JobManager map
Zhu Zhanyan
committed
c64039a
Move documentation on JobService operations to CoreService protobuf definition
Zhu Zhanyan
committed
ed4dae9
Added IngestJob to python sdk as native representation of IngestionJob proto
Zhu Zhanyan
committed
24c09be
Make empty filter on JobService's listJobs() select all ingestion jobs
Zhu Zhanyan
committed
2930388
Added bindings for Job management API to python sdk client.
Zhu Zhanyan
committed
5cc8abe
Fixed __connect_core() to connect to Feast CoreService when calling on Job API calls
Zhu Zhanyan
committed
d86cb66
Auto reload IngestJob.status and IngestJob.external_id on get property.
Zhu Zhanyan
committed
d9874f2
Added IngestJob.wait() to wait for job status to transtion
Zhu Zhanyan
committed
19620b2
Added basic job api e2e test to exercise job api
Zhu Zhanyan
committed
6b47ecb
Reorder the operations e2etest to make sure that jobs are running after test
Zhu Zhanyan
committed
f3471ac
Added e2e test for all types exercising job api
Zhu Zhanyan
committed
1707401
Fixed typo in function arguments
Zhu Zhanyan
committed
9999db5
Added unit tests for Ingestion Job API additions in python sdk
Zhu Zhanyan
committed
f012623
Rename "ingestion" to "ingest" for more consistent naming
Zhu Zhanyan
committed
2e819b2
Disable support for restarting Job in a terminal state due to possible race conditions
Zhu Zhanyan
committed
ba08c8a
Added __str__ and __repr__ to IngestJob to render ingestjob in human readable string
Zhu Zhanyan
committed
5fb774c
Added FeatureSetRef to represent references to featursets
Zhu Zhanyan
committed
80ef3c9
Admend client's list_ingest_jobs() to accept feature references directly
Zhu Zhanyan
committed
d3c9a3d
Fixed typo in IngestJob.store property
Zhu Zhanyan
committed
0cfc32a
Fixed issue with FeatureSetRef.from_str not converting version to int
Zhu Zhanyan
committed
a540157
Make the grpc error message more apparent on stop_ingest_job() and restart_ingest_job()
Zhu Zhanyan
committed
355b571
Added feast ingest-job list, describe, stop, restart to CLI
Zhu Zhanyan
committed
05b33a6
Rename Job to RetrievalJob to prevent confusion with IngestJob
Zhu Zhanyan
committed
3b8198a
Updated e2e tests to use FeatureSetRef in list_ingest_jobs()
Zhu Zhanyan
committed
19bd141
Fixed due e2e tests to cater to new limitations on stop_ingest_job()
Zhu Zhanyan
committed
9f6cf6a
Increase timeout on test_all_types_ingest_jobs() e2e test.
Zhu Zhanyan
committed
87054b9

Commits on Apr 7, 2020

Configure IngestJob.wait() to backoff with a exponentially larger wait duration
Zhu Zhanyan
committed
1848dd4
Added print statements to debug e2e test failure.
Zhu Zhanyan
committed
146fb2b
Revert "Added print statements to debug e2e test failure."
Zhu Zhanyan
committed
0fa6b93
Fixed issue of test waiting for aborted job to become running causing timeout.
Zhu Zhanyan
committed
66d3f77