Read NeuroExplorer files above 2 GB and fix spike waveform offsets - #1898
Open
h-mayorquin wants to merge 1 commit into
Open
Read NeuroExplorer files above 2 GB and fix spike waveform offsets#1898h-mayorquin wants to merge 1 commit into
h-mayorquin wants to merge 1 commit into
Conversation
Two defects in neuroexplorerrawio, both returning wrong data without raising. Waveform data starts after the variable's timestamps, which the specification stores as 4 bytes each. _get_spike_raw_waveforms skipped only n * 2 bytes, so it began reading halfway through the timestamp array. On the gin file File_neuroexplorer_2.nex, sig01i_wf started [-26086, 22, -24757, 22] where the file holds [-60, -13, 37, 138]. DataOffset is declared signed in the specification, but NeuroExplorer keeps writing past 2 GB and stores the low 32 bits, so a variable beyond that point reads back negative and the reader indexes its memmap from the wrong end of the file. Reading the field unsigned recovers the true offset exactly for any file below 4 GB; above that the information is genuinely lost and the file has to be re-exported as .nex5, which uses 64-bit offsets. Verified on a 2.5 GB recording with 196 variables, 69 of them past the boundary: every array then lands where the headers say, and the first samples of a continuous variable read -208, -241, -282 instead of 193, 215, 224. The counts come out of the header as numpy int32, so mixing them with a python int in the offset arithmetic raises OverflowError under numpy 2. Every value entering that arithmetic is now a python int.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rodrigo Paz from the Nelson lab can't convert their NeuroExplorer sessions as most of their files are above 2 GB.
This PR reads
DataOffsetas unsigned and that recovers the true offset for any file below 4 GB. Above that the low bits are really gone and the file has to be re-exported as.nex5, which uses 64 bit offsets. I also fixed the spike waveforms which were read fromoffset + n * 2when the timestamps that precede them are 4 bytes each.I tested both on a 2.5 GB recording of theirs with 196 variables, 69 of them past the boundary. Every array lands where the headers say and the first samples of
AD18read-208, -241, -282instead of193, 215, 224.