First draft of explanation_performance guide - #2705
Merged
erikvansebille merged 16 commits intoAug 13, 2026
Merged
Conversation
| | | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. | | ||
|
|
||
| ```{note} | ||
| In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. However, we are working on an upstream fix in to make caching compressed zarr files faster, so this may change in the future. |
Contributor
There was a problem hiding this comment.
Not sure what you're referring to here by 'working on an upstream fix'?
Member
Author
There was a problem hiding this comment.
Oh, I thought you mentioned that the issue was a repeated/inefficient uncompressing logic in zarr. And that we could try patch/fix this issue in the zarr library itself?
If that's not in the planning; then better to remove this statement
Contributor
|
Looks good to far! Some comments above |
Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com>
Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com>
for more information, see https://pre-commit.ci
…ebille/Parcels into explanation_performance
erikvansebille
commented
Aug 13, 2026
| | | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. | | ||
|
|
||
| ```{note} | ||
| In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. However, we are working on an upstream fix in to make caching compressed zarr files faster, so this may change in the future. |
| | Works out-of-the-box | Only performs well for very small ParticleSets | | ||
|
|
||
| ```{note} | ||
| The long-term plan for Parcels development is to make this Option 4 work well for all cases. However, this will require significant work on Dask indexing. |
Member
Author
There was a problem hiding this comment.
This is correct, right?
Comment on lines
+32
to
+37
| source_store = zarr.storage.LocalStore(filenames) | ||
| cache_store = zarr.storage.MemoryStore() | ||
|
|
||
| store = CacheStore( | ||
| store=source_store, cache_store=cache_store, max_size=MAX_CACHE_SIZE | ||
| ) |
Member
Author
There was a problem hiding this comment.
Should/can we put some of this under the hood in the parcels.open_raw_zarr() method?
| | | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. | | ||
|
|
||
| ```{note} | ||
| In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. However, we are working on an upstream fix in to make caching compressed zarr files faster, so this may change in the future. |
Member
Author
There was a problem hiding this comment.
Oh, I thought you mentioned that the issue was a repeated/inefficient uncompressing logic in zarr. And that we could try patch/fix this issue in the zarr library itself?
If that's not in the planning; then better to remove this statement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an explanation_performance.md guide to the documentation, detailing the four options for loading fieldsets
Note that this PR is blocked by the implementation of
windowed_arrays(#2656)Checklist
mainfor normal development,v3-supportfor v3 support)AI Disclosure