GitLab

Scheduled execution, payload is blank

• If startTimestamp is not set, query statefile lastSuccessfulLoad
• Implement Statefile Query Function (not sure where it will be)
• If statefile lastSuccessfulLoad is not set: [optional] if env == DEV : five minutes ago // if env != DEV : error;
• Be sure of both topics names to implement one "publishing" function
• endTimestamp is calculated from now()

Self-execution (paginating through a result set):

• if hasMore: true, trigger self topic
• All payload fields required, if not valid : error

{
   "startTimestamp": t,
   "endTimestamp": t,
   "offset": uuid,
}

Backfill:

Triggered with:

{
   "startTimestamp": t,
   "endTimestamp": t
}

• only start and end fields required Statefile schema:

{
  "lastSuccessfulLoad": t
}

State persistence:

After pagination is complete (hasTrue: false), and events published to PubSub, write endTimestamp to state as lastSuccessfulRun

Note: timestamps must be unix format, including milliseconds. Python doesn't do this by default so much be *1000 sadfase