-
New Feature
-
Resolution: Fixed
-
Normal
-
None
Currently there is no way to do incremental imports. In order to support incremental imports, the scraper needs to know the last listen that was imported. To accomplish this, we need to:
- create a new table that tracks the user id and the last imported timestamp
- the cassandra writer can then update the last imported timestamp for a user at the end of a batch of imports.
- expose this date as a new API endpoint
- update the scraper to utilize this new endpoint.
- is a dependency of
-
LB-20 Incremental imports
- Closed