Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[cards] Changing card get API route to ensure faster retrieval. #442

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[cards] Changing card get API route to ensure faster retrieval. #442

wants to merge 2 commits into from

Conversation

valayDave
Copy link
Contributor

@valayDave valayDave commented Sep 19, 2024 •
edited
Loading

  • this change will ensure ds-root is available via metadata so that cards can be directly accessed from the API endpoint .
  • We extract ds-root from the metadata table to retrieve cards.
  • shedding dead code in light of new changes.
  • We also change the API route like
  • Instead of having a cache, it directly asks S3 when it gets requested a card-id (which consists of hash,type,user-id).
  • This works very nicely for the data-update / HTML retrieval route.
  • The cache is useful for the list card method
"/flows/{flow_id}/runs/{run_number}/steps/{step_name}/tasks/{task_id}/cards/{type}/{uuid}?{user_set_id}="

@valayDave
[card-cache] optimizations to avoid deadlocks
f939277 
- deadlocks happened when cleanups coincided with heavy load
- tests simulating a cleanup (disk + shared-objects) and heavy load together were successfully able to reproduce the deadlock situation
- Locking at the `context` level:
    - Each time we want to clean up we lock to create a new context
    - All directories/processes are written within that new context.
    - Switching this context which ensure that all new processes get created differently and the cleanup process can safely remove everything.
    - Context also sets the read/write directory for the cache object used in the API endpoint.
    - All locking on the API side now always time-bound. The code time's out if it can't acquire a lock.
    - The method ensure that all operations won't in-definately hold the lock.
- Changed defaults for minimum amount to time to wait for cards in the cache process to 20 seconds (helps make things snappier)
- Added `timings` dict in card cache to optimize loading cycles (Ensured that it is set based on a per-card basis)
@valayDave
[cards] Changing card get API route to ensure faster retrieval.
24aa7c4 
- this change will ensure ds-root is available via metadata so that cards can be directly accessed from the API endpoint .
- We extract ds-root from the metadata table to retrieve cards.
- shedding dead code in light of new changes.
@valayDave valayDave mentioned this pull request Sep 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@valayDave