Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
Hubspot Extract to PubSub
Hubspot Extract to PubSub
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 1
    • Issues 1
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 0
    • Merge requests 0
  • Requirements
    • Requirements
    • List
  • Operations
    • Operations
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
  • Analytics
    • Analytics
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Cosmos
  • Modules
  • Hubspot Extract to PubSubHubspot Extract to PubSub
  • Issues
  • #2

Closed
Open
Created Oct 21, 2020 by Ray Walker@rawalkerOwner

Fix async GCS writes concurrency

To handle hundreds of events per polling interval we need to write multiple events to GCS concurrently, as they appear to take approximately 100ms per write.

Find a way to execute all GCS writes concurrently without blowing memory budget - possibly requires semaphor.

https://console.cloud.google.com/logs/viewer?pinnedLogId=000001-653228db-05b2-491c-ba0d-c91b6eadefdb&pinnedLogTimestamp=2020-10-21T11:03:37.703Z&project=cosmos-dev-286703&minLogLevel=0&expandAll=false&timestamp=2020-10-26T11:02:28.026000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeEnd=2020-10-26T11:02:44.640Z&interval=P7D&resource=cloud_function&filters=text:hubspot&scrollTimestamp=2020-10-21T11:08:32.471000000Z&advancedFilter=resource.type%3D%22cloud_function%22%0Ahs-extract-gcs-cosmos-dev&dateRangeStart=2020-10-19T11:02:44.640Z

Alternate solution would be to write to PubSub topics and execute all writes to GCS in separate function if faster / more scalable.

image

Edited Oct 26, 2020 by Ray Walker
Assignee
Assign to
Sprint 1
Milestone
Sprint 1 (Past due)
Assign milestone
Time tracking