From bert@bic.mni.mcgill.ca Mon Nov 11 19:39:04 2002 From: bert@bic.mni.mcgill.ca (Robert VINCENT) Date: Mon, 11 Nov 2002 14:39:04 -0500 Subject: [MINC-development] Welcome Message-ID: Hi everyone, Welcome to the list. I thought I'd start off by posting a few guesses about the goals for MINC development. To test my understanding of the current state of things, I'd like to see if there is consensus about the following claims: 1. MINC could be better documented. 2. MINC would benefit from a more complete set of automated tests. 3. MINC requires extensions to support files larger than core. 4. MINC requires extensions to support files larger than 2 GB. 5. MINC requires extensions to run on 64-bit architectures. 6. Given #3, block-structuring MINC data will improve file access speed. 7. MINC must remain backward compatible with existing data. I've intentionally used MINC as a blanket term for the file format, libraries, and dependant applications. Claims #3, #4, and #5 are related but not necessarily completely overlapping. There are probably people running on 32-bit architectures who need to process large files, and there are people with 64-bit systems that still don't have enough core to process huge files in memory. My undertanding is that Jason is currently spending some time on #5, making MINC tools run in a 64 bit world on SGI. The The MICe group seems to be focused on #3 and #6. Has anyone invested much time in these other things? Is anyone aware of work by the NetCDF group to address any of these issues? What other major or minor tasks are obvious to people? Thanks! -bert From jason@bic.mni.mcgill.ca Mon Nov 11 20:03:19 2002 From: jason@bic.mni.mcgill.ca (Jason Lerch) Date: Mon, 11 Nov 2002 15:03:19 -0500 Subject: [MINC-development] Welcome In-Reply-To: ; from bert@bic.mni.mcgill.ca on Mon, Nov 11, 2002 at 02:39:04PM -0500 References: Message-ID: <20021111150318.A1245801@bic.mni.mcgill.ca> Hi all, (firstly: it appears that the default reply to goes to the sender of the message and not the list - can that be easily changed?) I have some minor comments to add to that list - but thanks for getting the discussion started! One item to add to the wishlist: * better binary distribution of the MINC toolkit. I've just started on creating rpms, tardists, etc., but the whole process should be more automated (modificiation of the makefiles/autoconiscation structure) and extended to everything that we are giving away. Some comments about the other bits you've listed: > 3. MINC requires extensions to support files larger than core. > 4. MINC requires extensions to support files larger than 2 GB. There are really two issues here: extensions to the core libraries which allow for just that, and a check of the code-base of the existing tools to make sure that they are using those libraries correctly. For example, the various algorithms within INSECT rely on their own library which is a) a pain in the arse to build and b) not currently capable of dealing with large volumes. So even if the prerequisite changes were made to MINC and volume_io, you'd still have to go into at least those bits of code to ensure that they rely on MINC and not just pull everything into memory in a different array structure. > 5. MINC requires extensions to run on 64-bit architectures. I'm not quite sure what you mean by extensions - the only work to be done is to check the code for incorrect assumptions about the sizes of pointers, ints, etc. And while I have started that, I have not been proceeding in any sort of rigorous fashion, but rather just trying to patch problems as I run into them. So I suppose this will integrate nicely with a thorough test suite. Cheers, Jason From minc-development@bic.mni.mcgill.ca Mon Nov 11 20:21:56 2002 From: minc-development@bic.mni.mcgill.ca (Robert VINCENT) Date: Mon, 11 Nov 2002 15:21:56 -0500 Subject: [MINC-development] Welcome In-Reply-To: <20021111150318.A1245801@bic.mni.mcgill.ca> Message-ID: Jason, On Mon, 11 Nov 2002, Jason Lerch wrote: > (firstly: it appears that the default reply to goes to the sender of > the message and not the list - can that be easily changed?) > I think I've changed this behavior as of this message. > > 5. MINC requires extensions to run on 64-bit architectures. > > I'm not quite sure what you mean by extensions - the only work to be > done is to check the code for incorrect assumptions about the sizes of > pointers, ints, etc. And while I have started that, I have not been > proceeding in any sort of rigorous fashion, but rather just trying to > patch problems as I run into them. So I suppose this will integrate > nicely with a thorough test suite. Perhaps "extension" is too strong a term. Substitute "testing and bugfixing" if that's more accurate. -bert From minc-development@bic.mni.mcgill.ca Mon Nov 11 21:04:42 2002 From: minc-development@bic.mni.mcgill.ca (John G. Sled) Date: Mon, 11 Nov 2002 16:04:42 -0500 Subject: [MINC-development] Welcome In-Reply-To: ; from bert@bic.mni.mcgill.ca on Mon, Nov 11, 2002 at 02:39:04PM -0500 References: Message-ID: <20021111160442.J7615@sickkids.ca> Hi Bert, I'm back in Toronto and have had a chance to talk to my colleagues at MICe about the MINC meeting. We are pleased that the BIC is committed to improving the MINC format and we are keen to collaborate with you on this project. As discussed at the meeting, I think this mailing list should be a good way to collect the ideas of interested parties so as to put together a design proposal (MINC 2.0?). I had suggest we meet in Montreal in December to go over the details; however, that doesn't seem feasible at this point. Can we set a meeting for mid January instead? I have a few additions to your list of MINC wishes. #8 MINC should support a multiresolution representation of the data that would allow one to efficiently work with data at reduced resolutions. This has obvious implications for visualization tools, but may also be an asset in implementing segmentation and registration schemes that are multiscale. One proposal for handling this is to use a Harr wavelet decomposition. #9 MINC should internally handle compression in either a lossy or lossless manner. #10 An interesting suggestion that came up at the meeting is that the MINC API should also address our interest in remote visualization by providing network access to data. In a sense the problems of working with out of core data and working with remote data are the same. In principle the API could be set up to make little distinction between file:/myfile.mnc and minc://myserver.ca/myfile.mnc #11 MINC should have an expanded set of languages that it supports. There is much enthusiasm here for a C++ API that hides some of the complexity. At the meeting it was also suggested that a JAVA API would be a good strategic choice. #12 The handling of numeric formats should be reexamined. In the present scheme when using volume IO there is one number type for the original file, another for the in memory volume and a third, either floating point or integer, that has meaning to the user. If this is combined with a caching mechanism then it would seem more sensible to cache this latter type rather than either of the former two. On a related note, perhaps it would also be helpful to distinguish between files that are meant to be interpreted as integers and those that are not. A minor comment, on point #4: when MINC is compiled on a 32bit architecture with the large file operating system flags then it does work with files larger than 2GB. I haven't tested this extensively, but the minc* tools appear to work. cheers, John Hospital for Sick Children Mouse Imaging Centre 555 University Ave. Toronto, Ontario M5G 1X8 Canada Phone: 416 813-7654 x1438 Fax: 416 813-2208 On Mon, Nov 11, 2002 at 02:39:04PM -0500, Robert VINCENT wrote: > Hi everyone, > > Welcome to the list. I thought I'd start off by posting a few guesses > about the goals for MINC development. > > To test my understanding of the current state of things, I'd like to see > if there is consensus about the following claims: > > 1. MINC could be better documented. > 2. MINC would benefit from a more complete set of automated tests. > 3. MINC requires extensions to support files larger than core. > 4. MINC requires extensions to support files larger than 2 GB. > 5. MINC requires extensions to run on 64-bit architectures. > 6. Given #3, block-structuring MINC data will improve file access speed. > 7. MINC must remain backward compatible with existing data. > > I've intentionally used MINC as a blanket term for the file format, > libraries, and dependant applications. > > Claims #3, #4, and #5 are related but not necessarily completely > overlapping. There are probably people running on 32-bit architectures > who need to process large files, and there are people with 64-bit systems > that still don't have enough core to process huge files in memory. > > My undertanding is that Jason is currently spending some time on #5, > making MINC tools run in a 64 bit world on SGI. The The MICe group seems > to be focused on #3 and #6. Has anyone invested much time in these other > things? Is anyone aware of work by the NetCDF group to address any of > these issues? > > What other major or minor tasks are obvious to people? > > Thanks! > > -bert > > > > _______________________________________________ > MINC-development mailing list > MINC-development@bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-development > From minc-development@bic.mni.mcgill.ca Mon Nov 11 21:20:11 2002 From: minc-development@bic.mni.mcgill.ca (Jason Lerch) Date: Mon, 11 Nov 2002 16:20:11 -0500 Subject: [MINC-development] Welcome In-Reply-To: <20021111160442.J7615@sickkids.ca>; from jgsled@sickkids.ca on Mon, Nov 11, 2002 at 04:04:42PM -0500 References: <20021111160442.J7615@sickkids.ca> Message-ID: <20021111162011.B1249883@bic.mni.mcgill.ca> > #12 The handling of numeric formats should be reexamined. In the present > scheme when using volume IO there is one number type for the original file, > another for the in memory volume and a third, either floating point or > integer, that has meaning to the user. If this is combined with a > caching mechanism then it would seem more sensible to cache this > latter type rather than either of the former two. On a related note, > perhaps it would also be helpful to distinguish between files that > are meant to be interpreted as integers and those that are not. > A little aside - that functionality exists to a limited extent in the bicpl with the label volume concept. That was also one of the main reasons for me to start on my (badly named) oobicpl library, which is a C++ wrapper to volume_io and bicpl for handling volumes, where there is a subclass each for regular (floating point) and label (integer) volumes. But I do like all of your additions to the wishlist for MINC 2.0 As another aside, I've created a web pagehere: http://www.bic.mni.mcgill.ca/software/minc2 wich has absolutely no content, but is owned by Bert (this is news to him too) and might serve as a good place to place various design documents as ideas begin to cement on this list. Jason From minc-development@bic.mni.mcgill.ca Mon Nov 11 23:27:35 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Tue, 12 Nov 2002 09:27:35 +1000 Subject: [MINC-development] Welcome In-Reply-To: <20021111160442.J7615@sickkids.ca> Message-ID: On Mon, 11 Nov 2002, John G. Sled wrote: > so as to put together a design proposal (MINC 2.0?). I had suggest we > meet in Montreal in December to go over the details; however, that > doesn't seem feasible at this point. Can we set a meeting for mid > January instead? How about 1 week after ISMRM or even (perish the though) _at_ ISMRM? Then you might find a lot of other interested parties there as well. Unless of course you wish to keep a bit more 'exclusive'. ;) > #8 MINC should support a multiresolution representation of the data > that would allow one to efficiently work with data at reduced resolutions. > This has obvious implications for visualization tools, but may also > be an asset in implementing segmentation and registration schemes > that are multiscale. One proposal for handling this is to use a > Harr wavelet decomposition. And on this note there has been an attempt made at this quite some time ago. The (nonworking?) code I have is a sparse field interpolator for MINC. > #10 An interesting suggestion that came up at the meeting is that > the MINC API should also address our interest in remote visualization > by providing network access to data. In a sense the problems of > working with out of core data and working with remote data are the > same. In principle the API could be set up to make little distinction > between file:/myfile.mnc and minc://myserver.ca/myfile.mnc netCDF has extensions to do this already. In addittion to extensions for a XML web interface. > A minor comment, on point #4: when MINC is compiled on a 32bit > architecture with the large file operating system flags then it does > work with files larger than 2GB. I haven't tested this extensively, > but the minc* tools appear to work. from OZ on this one as well. -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Mon Nov 11 23:48:06 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Tue, 12 Nov 2002 09:48:06 +1000 Subject: [MINC-development] Welcome In-Reply-To: Message-ID: On Mon, 11 Nov 2002, Robert VINCENT wrote: > To test my understanding of the current state of things, I'd like to see > if there is consensus about the following claims: > > 1. MINC could be better documented. > 2. MINC would benefit from a more complete set of automated tests. > 3. MINC requires extensions to support files larger than core. > 4. MINC requires extensions to support files larger than 2 GB. > 5. MINC requires extensions to run on 64-bit architectures. > 6. Given #3, block-structuring MINC data will improve file access speed. > 7. MINC must remain backward compatible with existing data. I think that should at least keep you busy for a few weeks... I don't quite follow #2 though, you mean more mincxxxxx progs? or something more specific? Take a look in /software/source/minc_dev for a few things I am working on currently. However at some stage we need to stop re-inventing the wheel and use something like PDL (which now has a working netCDF module that works with MINC! -- well as far as I've tested it anyhow) or perish the thought 'Numeric Python' :). I do agree with you about digging around netCDF and seeing what is available first before re-inventing the wheel. netCDF as far as I understand is already network transparent. To this end, netCDF can also read many other formats like HDF so there is also no reason as to why we need limit outselves to the netCDF version of network transparency. As such perhaps we need to make MINC more 'netCDF'ish' :). In order that many of the other freeware netCDF viewers out there can read MINC files without to much munging of the header. ncview come to mind: http://meteora.ucsd.edu/~pierce/ncview_home_page.html re: Compression: I seem to remember reading somewhere about netCDF and compression, the consensus was that netCDF probably shouldn't do this as HDF already does it. (remember, netCDF can read HDF tranparently). > What other major or minor tasks are obvious to people? My biggest wish has been that some functionality can be added to the core MINC libs (voxel-loop more specifically) such that MINC can take advantage of multiple processors (bewoulf!). netCDF supports concurrent access of files (RW) so there is no reason as to why MINC shouldn't do parallell ops when it can. (this would mean most of the minxxxx progs could be sped up to a factor of X). Thanks for the oportunity to get on a soap-box. -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Tue Nov 12 00:07:05 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Tue, 12 Nov 2002 10:07:05 +1000 Subject: [MINC-development] Welcome In-Reply-To: Message-ID: On Tue, 12 Nov 2002, Andrew Janke wrote: > > 1. MINC could be better documented. > > 2. MINC would benefit from a more complete set of automated tests. > > 3. MINC requires extensions to support files larger than core. > > 4. MINC requires extensions to support files larger than 2 GB. > > 5. MINC requires extensions to run on 64-bit architectures. > > 6. Given #3, block-structuring MINC data will improve file access speed. > > 7. MINC must remain backward compatible with existing data. Oh, and while we're at it, can I add: 8. Munging volume_io.h to reduce the amount of name space pollution? ie: #define X 0 #define Y 1 #define Z 2 is a shining example. If you attempt to use GSL within volume_io you will get obsecure errors as GSL uses gsl_matrix * X in some of it's function prototypes..... Perhaps something like this would be better: #define VOLIO_X 0 #deinfe VOLIO_Y 1 #define VOLIO_Z 2 There are plenty more examples in volume_io that also need addressing, this of course may be a backwards compatibility nightmare, although having a volume_io_old.h with wrappers may be a way around this. Others that come to mind are: ALLOC, FREE, REALLOC, FALSE, TRUE, OFF, ON, INTERPOLATE, PI, ABS, MAX, SIGN, MIN, IJ, IJK, FLOOR, ROUND and N_DIMENSIONS. IMHO all #defines in a library header should be prefixed with the libraries short name ala GNU libraries. (such as GSL). -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Tue Nov 12 00:20:11 2002 From: minc-development@bic.mni.mcgill.ca (Robert Vincent) Date: Mon, 11 Nov 2002 19:20:11 -0500 Subject: [MINC-development] Welcome References: Message-ID: <3DD0493B.6060104@world.std.com> Andrew, You make good points about volume_io.h. I've only just glanced at it, but hadn't given it much thought. I completely agree that using some sort of VOLIO_ or similar prefix is preferable in the future. I'm sure we can come up with something that can be kept backward compatible. -bert Andrew Janke wrote: > On Tue, 12 Nov 2002, Andrew Janke wrote: > > >>>1. MINC could be better documented. >>>2. MINC would benefit from a more complete set of automated tests. >>>3. MINC requires extensions to support files larger than core. >>>4. MINC requires extensions to support files larger than 2 GB. >>>5. MINC requires extensions to run on 64-bit architectures. >>>6. Given #3, block-structuring MINC data will improve file access speed. >>>7. MINC must remain backward compatible with existing data. >> > > Oh, and while we're at it, can I add: > > 8. Munging volume_io.h to reduce the amount of name space pollution? > > ie: > #define X 0 > #define Y 1 > #define Z 2 > > is a shining example. If you attempt to use GSL within volume_io you will get > obsecure errors as GSL uses gsl_matrix * X in some of it's function > prototypes..... > > Perhaps something like this would be better: > > #define VOLIO_X 0 > #deinfe VOLIO_Y 1 > #define VOLIO_Z 2 > > There are plenty more examples in volume_io that also need addressing, this of > course may be a backwards compatibility nightmare, although having a > volume_io_old.h with wrappers may be a way around this. > > Others that come to mind are: > > ALLOC, FREE, REALLOC, FALSE, TRUE, OFF, ON, INTERPOLATE, PI, ABS, MAX, SIGN, > MIN, IJ, IJK, FLOOR, ROUND and N_DIMENSIONS. > > IMHO all #defines in a library header should be prefixed with the libraries > short name ala GNU libraries. (such as GSL). > > > -- > Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) > Australia->University of Queensland->Centre for Magnetic Resonance > Work: +61 7 3365 4100 || Home: +61 7 3800 4042 > > _______________________________________________ > MINC-development mailing list > MINC-development@bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-development > From minc-development@bic.mni.mcgill.ca Tue Nov 12 00:33:50 2002 From: minc-development@bic.mni.mcgill.ca (Robert Vincent) Date: Mon, 11 Nov 2002 19:33:50 -0500 Subject: [MINC-development] Welcome References: Message-ID: <3DD04C6E.1010506@world.std.com> Hi All, Thanks for all of the suggestions so far! To clarify what I mean by "automated tests", I'm speaking of code and scripts dedicated to the purpose of verifying the correctness of the library routines and utilities. Also, to clarify about the 2GB file limit - my understanding is that this is a NetCDF issue caused by the use of 32 bit offsets within the NetCDF format. The NetCDF official website mentions this problem and says it will be fixed in 4.0. Jason and Steve mentioned that huge MINC files work fine today if the image is the last data object in the file. I assume that this is because the existing MINC utilities never perform an lseek() within this part of the file. Using a network or virtual file interface would probably change this. -bert From minc-development@bic.mni.mcgill.ca Tue Nov 12 01:09:03 2002 From: minc-development@bic.mni.mcgill.ca (Peter NEELIN) Date: Mon, 11 Nov 2002 20:09:03 -0500 Subject: [MINC-development] Welcome In-Reply-To: <3DD04C6E.1010506@world.std.com> Message-ID: On Mon, 11 Nov 2002, Robert Vincent wrote: > Jason and Steve mentioned that huge MINC files work fine today if the > image is the last data object in the file. I assume that this is > because the existing MINC utilities never perform an lseek() within this > part of the file. Using a network or virtual file interface would > probably change this. To clarify a bit: The 2 GB limitation in NetCDF files arises because the file format only stores 32-bit offsets to the beginning of a variable. Larger files can be used if the beginning of the last variable starts before the 2 GB limit. Thus the image variable must be last (since it is the largest variable). This will only work if minc and NetCDF are compiled in an environment that supports 64-bit seeks (NetCDF is seeking to the beginning of each piece being read in, so it has to be able to seek beyond 2 GB). Peter ---- Peter Neelin (neelin@bic.mni.mcgill.ca) From minc-development@bic.mni.mcgill.ca Tue Nov 12 01:26:02 2002 From: minc-development@bic.mni.mcgill.ca (Peter NEELIN) Date: Mon, 11 Nov 2002 20:26:02 -0500 Subject: [MINC-development] Welcome In-Reply-To: Message-ID: On Tue, 12 Nov 2002, Andrew Janke wrote: > To this end, netCDF can also read many other formats like > HDF so there is also no reason as to why we need limit outselves to the netCDF > version of network transparency. Another clarification: Unless things have changed, NetCDF does not read HDF. You can, however, compile against the HDF library since it has a NetCDF-compatible API that will read NetCDF files, but will only write HDF files. (My information may be out of date - I haven't checked recently.) > As such perhaps we need to make MINC more 'netCDF'ish' :). In order that many of > the other freeware netCDF viewers out there can read MINC files without to much > munging of the header. The one big issue is the rescaling of data. I intentionally departed from the standard since I felt that it was more important to be able to scale slices separately (and so not have to store the whole volume in memory). I had thought about writing a MINC to NetCDF "converter" (really a fixer-upper), but I never found any NetCDF applications that really justified the effort. > ncview come to mind: > http://meteora.ucsd.edu/~pierce/ncview_home_page.html This looks neat. Have you tried it out? It should work, although the slice scaling would be off with some files. Try "mincreshape -normalize" to get all slices onto the same scale, and then use "minc_modify_header -dinsert" to stuff in scale and offset for the image variable and you should have a NetCDF-compliant file. > re: Compression: I seem to remember reading somewhere about netCDF and > compression, the consensus was that netCDF probably shouldn't do this as HDF > already does it. (remember, netCDF can read HDF tranparently). I don't believe that the NetCDF folks are planning on giving up on it yet! HDF just provides a NetCDF-like API and can read NetCDF files. But once gone over to HDF, there is no going back. Someone has made modifications to 3.3 (I think) that makes use of zlib for compression. The NetCDF folks are not big fans of on-the-fly compression since it interferes with random access and fast IO (I think that that is the gist of the argument - have a look at the NetCDF FAQ). Peter ---- Peter Neelin (neelin@bic.mni.mcgill.ca) From minc-development@bic.mni.mcgill.ca Tue Nov 12 03:32:57 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Tue, 12 Nov 2002 13:32:57 +1000 Subject: [MINC-development] Welcome In-Reply-To: <3DD0493B.6060104@world.std.com> Message-ID: On Mon, 11 Nov 2002, Robert Vincent wrote: > You make good points about volume_io.h. I've only just glanced at it, > but hadn't given it much thought. I completely agree that using some > sort of VOLIO_ or similar prefix is preferable in the future. I'm sure > we can come up with something that can be kept backward compatible. On that note I don't see why a lot of it should even be there.... PI, ABS, and SQR for example. PI should just be removed IMHO (what's wrong with M_PI?) The rest that are unrelated to volumes should be moved to something like "funky_macros.h". Just my 2c. a From minc-development@bic.mni.mcgill.ca Tue Nov 12 03:58:51 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Tue, 12 Nov 2002 13:58:51 +1000 Subject: [MINC-development] Welcome In-Reply-To: Message-ID: On Mon, 11 Nov 2002, Peter NEELIN wrote: > > ncview come to mind: > > http://meteora.ucsd.edu/~pierce/ncview_home_page.html > > This looks neat. Have you tried it out? It should work, although the slice > scaling would be off with some files. Try "mincreshape -normalize" to get > all slices onto the same scale, and then use "minc_modify_header -dinsert" > to stuff in scale and offset for the image variable and you should have a > NetCDF-compliant file. It will work without a scale and offset as you get to choose the variable you wish to display (assuming you are not worried about correct slice normalisation) http://www.cmr.uq.edu.au/~rotor/graagh/ncview_snap.jpg :) a From minc-development@bic.mni.mcgill.ca Tue Nov 12 15:05:02 2002 From: minc-development@bic.mni.mcgill.ca (Jason Lerch) Date: Tue, 12 Nov 2002 10:05:02 -0500 Subject: [MINC-development] binary distribution Message-ID: <20021112100502.D1214281@bic.mni.mcgill.ca> Hi all, the binary distribution project is slowly progressing; see here: http://www.bic.mni.mcgill.ca/software/distribution/ A question about dpkg for those debian afficionados: does anyone know of the dpkg tools existing for RedHat? Even though I'm using epm to build all the packages, I still need the native packaging tools to actually create those respective packages. Some other questions I could use some advice on: * should the creation of these binary packages become part of the make dist mechanism, or live as a separate script. I'm leaning toward the latter as I think that it'll be easier to implement, but am willing to be convinced otherwise. * The headers for the packages should arguably be included in the CVS tree for each respective package. Where should the netcdf package header reside? * What version of mni_autoreg is the one that should be distributed? * Should we create packages for the perl libraries as well, or rely on the perl installation mechanism? I have not really tested any of the packages, and am somewhat reluctant to on my machine as I'm currently quite happy with my /usr/local/mni and therefore reluctant to mess it up - so testers are welcome! Cheers, Jason From minc-development@bic.mni.mcgill.ca Tue Nov 12 19:34:30 2002 From: minc-development@bic.mni.mcgill.ca (Jason Lerch) Date: Tue, 12 Nov 2002 14:34:30 -0500 Subject: [MINC-development] Welcome In-Reply-To: ; from rotor@cmr.uq.edu.au on Tue, Nov 12, 2002 at 09:48:06AM +1000 References: Message-ID: <20021112143430.N1214281@bic.mni.mcgill.ca> > My biggest wish has been that some functionality can be added to the core MINC > libs (voxel-loop more specifically) such that MINC can take advantage of > multiple processors (bewoulf!). netCDF supports concurrent access of files (RW) > so there is no reason as to why MINC shouldn't do parallell ops when it can. > (this would mean most of the minxxxx progs could be sped up to a factor of X). I whole-heartedly support that idea. Why else did we get all these multiprocessor machines, eh? Jason From minc-development@bic.mni.mcgill.ca Tue Nov 12 23:09:59 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Wed, 13 Nov 2002 09:09:59 +1000 Subject: [MINC-development] Welcome In-Reply-To: <20021112143430.N1214281@bic.mni.mcgill.ca> Message-ID: On Tue, 12 Nov 2002, Jason Lerch wrote: > > My biggest wish has been that some functionality can be added to the core MINC > > libs (voxel-loop more specifically) such that MINC can take advantage of > > multiple processors (bewoulf!). netCDF supports concurrent access of files (RW) > > so there is no reason as to why MINC shouldn't do parallell ops when it can. > > (this would mean most of the minxxxx progs could be sped up to a factor of X). > > I whole-heartedly support that idea. Why else did we get all these > multiprocessor machines, eh? So that we can do lots of individual jobs and load up NFS to the hilt! (but here we are making minc-development bic-specific again... :) a From minc-development@bic.mni.mcgill.ca Tue Nov 12 23:28:03 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Wed, 13 Nov 2002 09:28:03 +1000 Subject: [MINC-development] binary distribution In-Reply-To: <20021112100502.D1214281@bic.mni.mcgill.ca> Message-ID: On Tue, 12 Nov 2002, Jason Lerch wrote: > the binary distribution project is slowly progressing; see here: > > http://www.bic.mni.mcgill.ca/software/distribution/ good good, I assume at some time you are going to add a link to: http://www.bic.mni.mcgill.ca/software/distribution/packages ie: when you have them stable? > A question about dpkg for those debian afficionados: does anyone know > of the dpkg tools existing for RedHat? Even though I'm using epm to > build all the packages, I still need the native packaging tools to > actually create those respective packages. 3 options (that I know of) -------------------------- * Convert. I agree with steve here, and now after finally getting FAI to work (http://www.informatik.uni-koeln.de/fai/) there is _NO_ turning back. Far FAR better than the redhat alternative. Once you've seen the light rpm is a dkpg... * apparently you can install .deb files using rpm as an 'alien' package. Who knows how intelligent this is though. * Try the experimental releases of dpkg (for other environments) http://some.debian.mirror/debian/project/dpkg > * should the creation of these binary packages become part of the make > dist mechanism, or live as a separate script. Separate script/makefile IMHO or as another target in the make structure ie: 'make bindist'. > * Should we create packages for the perl libraries as well, or rely on > the perl installation mechanism? Leave it up to PERL, far safer. Or better still perhaps have a punt at submitting them to CPAN. > I have not really tested any of the packages, and am somewhat > reluctant to on my machine as I'm currently quite happy with my > /usr/local/mni and therefore reluctant to mess it up - so testers are > welcome! Just keep me informed when they change, I have a little stack of debian boxes (affectionately known as onefish, twofish, redfish, blufish, lorax, sneed, etc etc etc) that are dying to try something bleeding edge. :) -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Wed Nov 13 06:06:36 2002 From: minc-development@bic.mni.mcgill.ca (Steve ROBBINS) Date: Wed, 13 Nov 2002 01:06:36 -0500 Subject: [MINC-development] binary distribution In-Reply-To: <20021112100502.D1214281@bic.mni.mcgill.ca>; from jason@bic.mni.mcgill.ca on Tue, Nov 12, 2002 at 10:05:02AM -0500 References: <20021112100502.D1214281@bic.mni.mcgill.ca> Message-ID: <20021113010636.B1361597@shadow.bic.mni.mcgill.ca> On Tue, Nov 12, 2002 at 10:05:02AM -0500, Jason Lerch wrote: > Hi all, > > the binary distribution project is slowly progressing; see here: > > http://www.bic.mni.mcgill.ca/software/distribution/ > > A question about dpkg for those debian afficionados: does anyone know > of the dpkg tools existing for RedHat? Even though I'm using epm to > build all the packages, I still need the native packaging tools to > actually create those respective packages. I don't know about dpkg on redhat, but it would be easy enough for me to build the packages on a debian system. > Some other questions I could use some advice on: > > * should the creation of these binary packages become part of the make > dist mechanism, or live as a separate script. I'm leaning toward the > latter as I think that it'll be easier to implement, but am willing > to be convinced otherwise. Easier to implement is good. It shouldn't be part of "make dist", in any case: that's for source distributions. You could add rules for "make rpm", etc or a do-it-all "make bindist", perhaps. > * The headers for the packages should arguably be included in the CVS > tree for each respective package. Where should the netcdf package > header reside? Headers? You're talking about the input to epm? Then I agree. For non-MNI stuff (netcdf) I'd make a CVS directory for all foreign stuff and put it there. -S From minc-development@bic.mni.mcgill.ca Mon Nov 18 22:39:24 2002 From: minc-development@bic.mni.mcgill.ca (Leila Baghdadi) Date: Mon, 18 Nov 2002 17:39:24 -0500 (EST) Subject: [MINC-development] Welcome Message-ID: Hi everyone My name is Leila Baghdadi, I work at the MICe center. I had a chance to look the e-mails that have been sent over the past week and I did not see any suggestion about efficiency/performance. It seems to me that minc1.0 is definitely faster than minc0.6 but still I checked the performance of minc1.0 against VTK for warping images and VTK is at least twice faster on my computer which has 2 Xeon 2 GHz cpus. E.g. VTK can apply a thin-plate spline warp to an image around 2 or 3 times faster than minc1.0 can. The main reason for this because VTK is multi-threaded. I thought that this should also be added to "wish-list" for long term plans. Thanks Leila From minc-development@bic.mni.mcgill.ca Tue Nov 19 05:09:15 2002 From: minc-development@bic.mni.mcgill.ca (Steve ROBBINS) Date: Tue, 19 Nov 2002 00:09:15 -0500 Subject: [MINC-development] Welcome In-Reply-To: ; from baghdadi@sickkids.ca on Mon, Nov 18, 2002 at 05:39:24PM -0500 References: Message-ID: <20021119000915.A250493@shadow.bic.mni.mcgill.ca> On Mon, Nov 18, 2002 at 05:39:24PM -0500, Leila Baghdadi wrote: > Hi everyone > > My name is Leila Baghdadi, I work at the MICe center. > > I had a chance to look the e-mails that have been sent over the past week > and I did not see any suggestion about efficiency/performance. > > It seems to me that minc1.0 is definitely faster than minc0.6 but still I > checked the performance of minc1.0 against VTK for warping images and VTK > is at least twice faster on my computer which has 2 Xeon 2 GHz cpus. > E.g. VTK can apply a thin-plate spline warp to an image around 2 or 3 > times faster than minc1.0 can. > > The main reason for this because VTK is multi-threaded. Really? So if you profile them on a single CPU machine, the MINC code is roughly as fast as the VTK code? If so, I'm impressed: I thought it would be a lot slower. Volume_io is riddled with switch statements (based on number type) that I thought would stall the CPU pipelining. -Steve From minc-development@bic.mni.mcgill.ca Tue Nov 19 06:19:55 2002 From: minc-development@bic.mni.mcgill.ca (Steve ROBBINS) Date: Tue, 19 Nov 2002 01:19:55 -0500 Subject: [MINC-development] Welcome In-Reply-To: ; from bert@bic.mni.mcgill.ca on Mon, Nov 11, 2002 at 02:39:04PM -0500 References: Message-ID: <20021119011955.B250493@shadow.bic.mni.mcgill.ca> On Mon, Nov 11, 2002 at 02:39:04PM -0500, Robert VINCENT wrote: > Hi everyone, > > Welcome to the list. I thought I'd start off by posting a few guesses > about the goals for MINC development. > > To test my understanding of the current state of things, I'd like to see > if there is consensus about the following claims: > > 1. MINC could be better documented. I'm sure you'll get no disagreement on this point. Andrew started extending the existing documentation (I've lost the url for this, though). I finished off the missing manpages for all the tools in MINC when I made the Debian packages. I keep meaning to put them in CVS, but have not yet done so. My personal wish about documentation, however, is to better document the actual file structure. Yes, it is based on NetCDF, but there seem to be some corner cases, e.g. "vector" volumes must have the vector dimension last. And it may be just me, but I have a hard time working out exactly how the image scaling works. Like Andrew, I feel that MINC would benefit by moving back towards netcdf standards. One reason this appeals to me is that we could possibly take advantage of the *already existing* bindings that netcdf has for various languages, e.g. C++, Java, Python, and perl. The thinner the "minc layer" on top of bare netcdf, the less code one has to maintain for all these other languages. Peter has raised one big difference between netcdf and MINC, namely that MINC supports independent scaling on each slice of an image, while netcdf "conventions" allow only a single overall scale for packed data. I wonder if Peter has some feeling about whether we could work with the netcdf folks to get the MINC view blessed. I'm not sure this really buys us much in the short run, really, since the netcdf "convention" for packed data is not actually implemented in the core library -- it is up to the application to support such a convention. At least, that's my understanding. However, in the long run, a layer of code supporing minc-style rescaling for packed data may appear in netcdf. Heck, we could supply them such a layer ;-) > 3. MINC requires extensions to support files larger than core. Notwithstanding your later disclaimer that the term "MINC" refers to both the file structure, libraries, and apps, I think this point needs some precision. I don't think there is anything in the file format, nor in netcdf, nor even in MINC that prevents one from processing a file larger than core. I'm not too sure about volume_io, but I think it also allows this (?). Isn't it also true that all the minc programs are carefully written (using voxel_loop) so that processing files larger than core is feasible? So, what's left? ;-) > 4. MINC requires extensions to support files larger than 2 GB. It seems that netcdf itself needs modifications to allow this in full generality. The netcdf FAQ is a little cagey about this, saying that "large files can be written" (with certain restrictions like the last variable may be extend past the 2GB limit, but it must start before that mark, etc). What escapes me, though, is: are there restrictions on how you can USE such a large file? Netcdf stores the offset to the beginning of the variable in a signed int (thus the 2GB restriction). But what about accessing a given slice? Is seeking going on -- and if so, are these offsets correctly calculated using off_t in netcdf? If yes, this is good news for us! > 6. Given #3, block-structuring MINC data will improve file access speed. ... for some access patterns. It may degrade other access patterns, e.g. traditional voxel_loop processing. > What other major or minor tasks are obvious to people? On a much longer time scale / more speculative note: should MINC remain based on netcdf, or is something like HDF more suitable? I don't know, but some of the proposed additions like compression or sparse data storage suggest that the netcdf view of the world (i.e. data is a dense matrix and fast random access is key) is not fitting well. It also doesn't fit well for polyhedral data, of course. That's enough babbling for now, -Steve From minc-development@bic.mni.mcgill.ca Tue Nov 19 14:04:46 2002 From: minc-development@bic.mni.mcgill.ca (Jason Lerch) Date: Tue, 19 Nov 2002 09:04:46 -0500 Subject: [MINC-development] Welcome In-Reply-To: <20021119011955.B250493@shadow.bic.mni.mcgill.ca>; from stever@bic.mni.mcgill.ca on Tue, Nov 19, 2002 at 01:19:55AM -0500 References: <20021119011955.B250493@shadow.bic.mni.mcgill.ca> Message-ID: <20021119090446.C382214@bic.mni.mcgill.ca> > > 6. Given #3, block-structuring MINC data will improve file access speed. > > ... for some access patterns. It may degrade other access patterns, > e.g. traditional voxel_loop processing. I don't see why that has to be the case. voxel_loop by definition does not care about neighbourhood information, so it should be perfectly feasible to rewrite it to read the data block by block and loop through the blocks, thereby making it's disk access as continuous - or nearly as continuous - as before. Jason From minc-development@bic.mni.mcgill.ca Tue Nov 19 14:32:13 2002 From: minc-development@bic.mni.mcgill.ca (John G. Sled) Date: Tue, 19 Nov 2002 09:32:13 -0500 Subject: [MINC-development] Welcome In-Reply-To: <20021119011955.B250493@shadow.bic.mni.mcgill.ca>; from stever@bic.mni.mcgill.ca on Tue, Nov 19, 2002 at 01:19:55AM -0500 References: <20021119011955.B250493@shadow.bic.mni.mcgill.ca> Message-ID: <20021119093213.A2415@sickkids.ca> On Tue, Nov 19, 2002 at 01:19:55AM -0500, Steve ROBBINS wrote: > Peter has raised one big difference between netcdf and MINC, namely > that MINC supports independent scaling on each slice of an image, > while netcdf "conventions" allow only a single overall scale for > packed data. I wonder if Peter has some feeling about whether > we could work with the netcdf folks to get the MINC view blessed. I'm > not sure this really buys us much in the short run, really, since the > netcdf "convention" for packed data is not actually implemented > in the core library -- it is up to the application to support such > a convention. At least, that's my understanding. However, in the > long run, a layer of code supporing minc-style rescaling for packed > data may appear in netcdf. Heck, we could supply them such a layer ;-) Perhaps we should take the easier route and revert to the netcdf scaling convention (i.e. one scale factor for the volume). All of the programs that use volume_io work this way already. Although having separate scale factors for each slice makes good use of the available numerical precision, if one considers digitization to be a source of noise then this adds noise nonuniformly across the slices. Another disadvantage of this scheme is that reformatting the data, for example from sagittal to transverse and back, is not an identity operation. > > 3. MINC requires extensions to support files larger than core. > > Notwithstanding your later disclaimer that the term "MINC" > refers to both the file structure, libraries, and apps, I think > this point needs some precision. I don't think there is anything > in the file format, nor in netcdf, nor even in MINC that prevents > one from processing a file larger than core. I'm not too sure about > volume_io, but I think it also allows this (?). Volome_io does allow one to work with files larger than core; however, the layout of the MINC file and the implementation of caching makes this slow for complicated 3D algorithms and visualization. The algorithms that do work well are mostly the ones that could have been implemented with voxel_loop. > > Isn't it also true that all the minc programs are carefully > written (using voxel_loop) so that processing files larger than > core is feasible? > > So, what's left? ;-) > All of the complicated 3D algorithms and visualization tools that we haven't written yet -- plus a few that we have ;) > > > 4. MINC requires extensions to support files larger than 2 GB. > > It seems that netcdf itself needs modifications to allow this > in full generality. The netcdf FAQ is a little cagey about this, > saying that "large files can be written" (with certain restrictions > like the last variable may be extend past the 2GB limit, but it > must start before that mark, etc). > > What escapes me, though, is: are there restrictions on how you > can USE such a large file? Netcdf stores the offset to the beginning > of the variable in a signed int (thus the 2GB restriction). But > what about accessing a given slice? Is seeking going on -- and > if so, are these offsets correctly calculated using off_t in netcdf? > If yes, this is good news for us! Yes, recent versions of NetCDF can be compiled to use 64 bit versions of seek so that even 32 bit programs can access the whole volume. > > > 6. Given #3, block-structuring MINC data will improve file access speed. > > ... for some access patterns. It may degrade other access patterns, > e.g. traditional voxel_loop processing. I don't think that voxel_loop would be affected by this since the loop could proceed through the voxels in their natural (i.e. file) order for most cases. > > What other major or minor tasks are obvious to people? > > On a much longer time scale / more speculative note: should > MINC remain based on netcdf, or is something like HDF more > suitable? I don't know, but some of the proposed additions > like compression or sparse data storage suggest that the netcdf > view of the world (i.e. data is a dense matrix and fast random > access is key) is not fitting well. It also doesn't fit well > for polyhedral data, of course. I don't know the answer to this, but now would be a good time to make these decisions given that MINC 3.0 is likely a long way off. cheers, John From minc-development@bic.mni.mcgill.ca Tue Nov 19 15:28:14 2002 From: minc-development@bic.mni.mcgill.ca (Steve ROBBINS) Date: Tue, 19 Nov 2002 10:28:14 -0500 Subject: [MINC-development] Welcome In-Reply-To: <20021119093213.A2415@sickkids.ca>; from jgsled@sickkids.ca on Tue, Nov 19, 2002 at 09:32:13AM -0500 References: <20021119011955.B250493@shadow.bic.mni.mcgill.ca> <20021119093213.A2415@sickkids.ca> Message-ID: <20021119102814.F260579@shadow.bic.mni.mcgill.ca> On Tue, Nov 19, 2002 at 09:32:13AM -0500, John G. Sled wrote: > On Tue, Nov 19, 2002 at 01:19:55AM -0500, Steve ROBBINS wrote: > > > 6. Given #3, block-structuring MINC data will improve file access speed. > > > > ... for some access patterns. It may degrade other access patterns, > > e.g. traditional voxel_loop processing. > > I don't think that voxel_loop would be affected by this since the loop > could proceed through the voxels in their natural (i.e. file) order for > most cases. I probably should not have mentioned voxel_loop. What I actually had in mind was something like a convolution operation: you need to drag a 5x5x5 window across the volume, for example. I'm not an expert in these matters, but the naive approach of scanning through the volume with a five-slice cache would be better served with the current disk layout. Is that not the case? Even if that is the case, a different layout may indeed be so much faster in some situations that it is an improvement overall. I don't know. I would be surprised if it is a win in _every_ case, and it would be nice to know a bit about what the tradeoff is. By the by, it turns out that HDF5 implements "chunked" storage and sparse storage already. And they have some kind of caching mechanism in the library. There is a small study on how the cache parameters affect the read and write efficiency: http://hdf.ncsa.uiuc.edu/HDF5/doc/Chunking.html -S From minc-development@bic.mni.mcgill.ca Tue Nov 19 18:17:39 2002 From: minc-development@bic.mni.mcgill.ca (David Gobbi) Date: Tue, 19 Nov 2002 13:17:39 -0500 (EST) Subject: [MINC-development] Welcome In-Reply-To: <20021119000915.A250493@shadow.bic.mni.mcgill.ca> Message-ID: On Tue, 19 Nov 2002, Steve ROBBINS wrote: > On Mon, Nov 18, 2002 at 05:39:24PM -0500, Leila Baghdadi wrote: [snip] > > > > It seems to me that minc1.0 is definitely faster than minc0.6 but still I > > checked the performance of minc1.0 against VTK for warping images and VTK > > is at least twice faster on my computer which has 2 Xeon 2 GHz cpus. > > E.g. VTK can apply a thin-plate spline warp to an image around 2 or 3 > > times faster than minc1.0 can. > > > > The main reason for this because VTK is multi-threaded. > > Really? So if you profile them on a single CPU machine, the > MINC code is roughly as fast as the VTK code? If so, I'm impressed: > I thought it would be a lot slower. Volume_io is riddled with switch > statements (based on number type) that I thought would stall the > CPU pipelining. The comparison depends on interpolation mode, the type of warp that is applied, whether the warp is applied in the forward or reverse direction, CPU architecture, etc. I am getting a tar file ready that will contain a script and a set of data + transformations for comprehensive benchmarking. With luck I'll be able to release it sometime this week. I've seen some cases where minc & VTK run at the same speed, and other cases where VTK is up to 9 times as fast and minc. So VTK is faster, but not miraculously so. It would also be nice to see some ITK vs. minc benchmarks, if there is an ITK analogue of mincresample. - David From minc-development@bic.mni.mcgill.ca Tue Nov 19 20:56:51 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Wed, 20 Nov 2002 06:56:51 +1000 Subject: [MINC-development] Welcome In-Reply-To: <20021119090446.C382214@bic.mni.mcgill.ca> Message-ID: On Tue, 19 Nov 2002, Jason Lerch wrote: > > ... for some access patterns. It may degrade other access patterns, > > e.g. traditional voxel_loop processing. > > I don't see why that has to be the case. voxel_loop by definition does > not care about neighbourhood information, so it should be perfectly > feasible to rewrite it to read the data block by block and loop > through the blocks, thereby making it's disk access as continuous - or > nearly as continuous - as before. If we really are going to get technical and change to HDF, why not just do it in the way the volume_io was originally intended to. ie: support both. You'll note that volume_io's code has been written such that it doesn not require minc, in theory it could support many file formats. a From minc-development@bic.mni.mcgill.ca Tue Nov 19 21:24:38 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Wed, 20 Nov 2002 07:24:38 +1000 Subject: [MINC-development] Welcome In-Reply-To: <20021119000915.A250493@shadow.bic.mni.mcgill.ca> Message-ID: On Tue, 19 Nov 2002, Steve ROBBINS wrote: > > The main reason for this because VTK is multi-threaded. > > Really? So if you profile them on a single CPU machine, the > MINC code is roughly as fast as the VTK code? If so, I'm impressed: > I thought it would be a lot slower. Volume_io is riddled with switch > statements (based on number type) that I thought would stall the > CPU pipelining. Although you can bypass a lot of these and directly access the underlying macros should you wish too. However the speed gained in doing this is often not all that great. :) Well, in the tests I have done anyway. a From minc-development@bic.mni.mcgill.ca Tue Nov 19 21:53:15 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Wed, 20 Nov 2002 07:53:15 +1000 Subject: [MINC-development] Welcome In-Reply-To: <20021119011955.B250493@shadow.bic.mni.mcgill.ca> Message-ID: On Tue, 19 Nov 2002, Steve ROBBINS wrote: > > 1. MINC could be better documented. > > I'm sure you'll get no disagreement on this point. Andrew started > extending the existing documentation (I've lost the url for this, > though). http://www.bic.mni.mcgill.ca/software/MDP/ The main idea here was to make the documentation more accessible (HTML + PDF) as opposed to tex (hurl) and .ps files. Yes, yes I know tex2html exists, but it (IMNSHO) is gawdy/tacky and usully produces copious output with little content, cripes!, that sounds just like latex! :). As such the initial effort was to convert all the existing documentation to SGML such that conversion from there to .info (GNU) .man .pdf .ps .tex .doc .rtf is a simple button click. >From there I hope to write a few minc tutorials to get newbies going. > I finished off the missing manpages for all the tools in MINC when I made the > Debian packages. I keep meaning to put them in CVS, but have not yet done so. Nice, I had never even noticed! On the upside I now know what mincwindow does! Funnily enough louis has written a volume_io proggy of his own called 'fix_range' or something.... and you could do it in minccalc. In the interim what are others thoughts about adding a -v option to all minc progs? (that of course should be automatic from cvs/release number). -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Thu Nov 21 01:58:01 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Thu, 21 Nov 2002 11:58:01 +1000 Subject: [MINC-development] mincfft. was: [geeks] vector_diemsion and rawtominc In-Reply-To: Message-ID: On Sun, 3 Nov 2002, Peter NEELIN wrote: > > Now of course the simple fix might be to modify mincfft to use > > MIvector_dimension but It did seem nice at the time to use complex_vector. > > Hmmm... I had not realized this. Your image dimensions, according to the > minc convention (which is pretty loose, but not on this point), are > "xspace" and "complex_vector". I suspect that you really want them to be > "yspace", "xspace" and "vector_dimension". The magic of three image > dimensions only happens when the last (fastest varying) dimension is > called "vector_dimension". In other words, mincfft is not minc compliant. > Although there is currently no mechanism in minc to identify how vector > data should be interpreted (rgb colour, complex values, displacements, > etc.), it is at the very least guaranteed to be identifiable in compliant > minc files by the existence of a dimension named "vector_dimension". Any > other name means that the data is scalar and the dimension is a parameter- > space dimension. OK, I have bowwed under the pressure and have fixed the version of mincfft in cvs (/s/s/minc_dev/mincfft) to now use MIvector_dimension. I have also updated the 2D fft option to the point that it now should be correct and a whole lot less memory hungry. At some stage I should re-write large slabs of it and remove the dependance on volume_io (there is no real need for it) but that's a job for another day....:) But now I have a question: I know volume_io handles vector volumes differently but is this to the point where it thinks a 3spatial + vector file has 3 dimensions only? try this: $ mincfft -2D -centre <3D_minc_file.mnc> fft.mnc All is fine, fft.mnc is a 4D file. (-centre shifts the fft result to the middle of the volume to make it more easily human grokkable) $ mincfft -2D -inverse fft.mnc -magnitude mag.mnc -phase phase.mnc Error: MINC file has only 3 dims, volume requires 4. Error: MINC file has only 3 dims, volume requires 4. Hrm, the output works though. So do I have to create a vector volume as such with volume_io? char *freq_dimorder[] = {MIzspace,MIyspace,MIxspace,MIvector_dimension}; *out_vol = create_volume(3, freq_dimorder, NC_FLOAT, TRUE, 0.0, 0.0); Currently I have it as such: *out_vol = create_volume(4, freq_dimorder, NC_FLOAT, TRUE, 0.0, 0.0) Hrm... -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042 From minc-development@bic.mni.mcgill.ca Thu Nov 21 02:15:19 2002 From: minc-development@bic.mni.mcgill.ca (Peter NEELIN) Date: Wed, 20 Nov 2002 21:15:19 -0500 Subject: [MINC-development] mincfft. was: [geeks] vector_diemsion and rawtominc In-Reply-To: Message-ID: On Thu, 21 Nov 2002, Andrew Janke wrote: > But now I have a question: I know volume_io handles vector volumes differently > but is this to the point where it thinks a 3spatial + vector file has 3 > dimensions only? Yes. I think that the default behaviour is to use the minc library magic to make everything scalar. There is a volume_io option that allows this to be turned off (I don't have time to look it up right now). Be warned, however, that volume_io uses pointers to pointers to pointers for its arrays (it was faster that way at the time that David benchmarked it). The down side of this is that a complex vector short volume will have as much pointer as data, even on a 32-bit system (one pointer for every 4 bytes of data). Peter ---- Peter Neelin (neelin@bic.mni.mcgill.ca) From minc-development@bic.mni.mcgill.ca Thu Nov 21 03:34:42 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Thu, 21 Nov 2002 13:34:42 +1000 Subject: [MINC-development] mincfft. was: [geeks] vector_diemsion and rawtominc In-Reply-To: Message-ID: On Wed, 20 Nov 2002, Peter NEELIN wrote: > > But now I have a question: I know volume_io handles vector volumes differently > > but is this to the point where it thinks a 3spatial + vector file has 3 > > dimensions only? > > Yes. I think that the default behaviour is to use the minc library magic > to make everything scalar. There is a volume_io option that allows this to > be turned off (I don't have time to look it up right now). Be warned, > however, that volume_io uses pointers to pointers to pointers for its > arrays (it was faster that way at the time that David benchmarked it). The > down side of this is that a complex vector short volume will have as much > pointer as data, even on a 32-bit system (one pointer for every 4 bytes of > data). Hrm, I see. So what you are saying is: rewrite it using ICV's like you should have in the first place you slacker. :) Time time... a From minc-development@bic.mni.mcgill.ca Fri Nov 22 17:40:45 2002 From: minc-development@bic.mni.mcgill.ca (John G. Sled) Date: Fri, 22 Nov 2002 12:40:45 -0500 Subject: [MINC-development] [Fwd: data format] Message-ID: <20021122124045.A2052@sickkids.ca> This is a message that Art Wetzel had sent describing the data layout they are using at the Pittsburgh Supercomputing Centre to serve up the visual human. From what I understand, some of the details are still in flux, but it does give some insight into their approach. cheers, John ############################ The basic element of data going to the client is a cube. Since the client is in control of the entire interaction every cube is delivered by explicit client request. For the Vis Female we currently provide 4 scales of cubes all composed of 8*8*8 = 512 voxels. The scales are just multiples of 2 so we have full resolution, 1/2 res, 1/4 res and 1/8 res cubes. The 1/8 res cubes for example enclose what was originally a full scale 64*64*64 voxel region. Therefore, each of the 512 voxels in a 1/8 scale cube corresponds to the low pass filtering of an 8*8*8 voxel region from the full scale data. In going from a low res cube to the next higher res (by a factor of 2) there are 8 possible octant cubes to choose from. These subcubes nestle into the corners of the larger cubes. These parameters such as number of res levels and cube size can be changed and we will want different choices for some special circumstances. For example, with substantially larger data sets we'll want more levels and possibly larger cubes. One wants larger cubes in order to use compression methods with longer supports. One wants smaller cubes to minimize the amount of unused data in extracting cutting planes. Clearly with the 8*8*8 cubes we actually get 8 times as much data as needed to construct any cutting plane (ignoring interpolation implications). This is offset by 2 things A) taking advantage of 3D compression correlations and B) the fact that we are presuming interactive navigation with rotations or at least a lot of region sweeps which will eventually use the extra data which stays in the client side cache which is flushed by the LRU criteria. The client cache is maintained as an octree which is traversed as needed to construct the cut plane image. The client is responsible for requesting those cubes which intersect the current cutting plane. We always retrieve the coarsest level of cubes first to provide both a quick display and also a basis for some of the hierarchical compression methods. Every client request therefore consists of a scale together with a cube number from the full object bounding box. Now the issue of data organization at the server... I've run compatible servers using several different data organizations not to mention compression methods. For example, when working with new data that one wants to see quickly but for just a single user its convenient to just concatenate all the planar transverse (or what ever base orientation you have) images and let the server extract individual cubes (which we sometimes call chads) from the solid block. In that mode the easiest thing given our 64bit server machine is to just memory map the whole thing and let the server code extract voxels to compose each cube in real time as requested by the client. Of course this mode suffers from poor speed since it is effectively page faulting new data from disk and will produce very bad disk seek patterns for some orientations. Also, it is only compatible with compression methods that can be done on the fly by the server. Nevertheless, this is convenient for a quick check when working with brand new data before going to the trouble of building a good data structure. The better method explicitly reorganizes and maintains the data in cube form on the server. In the current best variant on this strategy the data is collected into contiguous blocks corresponding to the coarse cube size together with all of its finer detail descendents. Again, the data is all memory mapped but when a paging op occurs it applies to all scales of a particular cubic volume at once. The linear layout then is something like 84211111111211111111.....4211111111 etc following a depth first recursive tree. This was probably not very successful in showing it but I hope you get the idea. In the uncompressed 4 scale hierarchy this whole block is mostly devoted to the 512 sale 1 cubes containing a total of 262144 voxels. It also contains 64 scale 2 cubes, 8 scale 4 cubes and the single level 8 parent cube. Therefore, without compression the whole hierarchy is ~1.143 times as big as just the original scale 1 voxels. Another advantage of this orgainzation is that it can use 16 bit offsets to index scale 1 data and only needs a small number of full 64 bit pointers to tie together the high levels. Finally compression ... I have been working a number of variations on compression techniques for both lossless and lossy storage and transmission. Given a fast network link the system (like the release version you've tested) operates pretty well with no explict compression other than the apparent visual compression produced by the hierarchy and the coarse to fine retrieval order. When navigating quickly enough to get just scale 8 data this is really a crude 512:1 compressive effect. Next the are methods which operate by simply compressing the data within each cube without making use of the overall hierarchy. These include include 3D DCT forming a JPEG type blocked transform and also some of the short support wavlets. When using the wavelets in this method its only hierarchy is the 3 octaves to reduce the 8*8*8 cube to a single voxel. At the most reasonable visual quality levels but where lossy methods are acceptable the 3D DCT within block compression works well out to more than 20:1. In the case of the Visible Female data the reversible integer wavelets are only able to get ~2:1 as limited by data SNR. Other than Haar the wavelets also have block bounds issues which limit their effectiveness in this mode. The better methods take advantage of the coarse to fine hierarchy in a multiresolution sense. Again there's DCT and short wavelets. Using DCT's and coding each level as a difference between an interplated expansion of the coarser parent and the true target we get good results in excess of 30:1 compression. In operation the effective rate is much higher yet when navigating quickly enough to stay at the scale 2 or even scale 4 levels. This is the version that we'll put out for the end of November public release upgrade. I'm working on a variation of that to deliver lossless data by using an approximate integer DCT that will give exactly the same result on all machines to be combined with a final precomputed "lossless overlay". This gives the speed advantage of lossy compression for navigation but still can give lossless delivery for analysis or a soon to be provided data save facility so you can export data to other applications on the client end. In the hierarchical modes the wavelet methods extend beyond a single cube resolution on up the full hierarchy. However, the block size does create some awkardness with boundary effects so its hard to do things like the 9/7 filter that is so popular in 2D image compression. Its much more straightforward to use the TS (2-6) transform which is a reversible integer wavelet as used by the Ricoh CREW system. In that case a low res boundary layer that is missing where a higher res cube fits into into the corner of its coarser parent can be sent along with the child coefficients to complete the transform support. This explaination leaves out the reversible color space transforms, both the one from CREW and another that I like better, and also handing noncubic voxels such as the Visible Male and particularly the new 70mm film rescan which is 1mm in z but .144 in x&y. Also I've not done an implementation with embedding methods due to issues presented by the random access requirement. You ask about discrete segmentation data. Eventually I want to put in a set of (probably 16 bit) code numbers that will label the voxels. This will be a 0 terminated octree that fits into the regular data hierarchy. While we're working on segmentation however so that the voxel labeling is changing its much more convenient to store that in a separate data structure what drives a separate server process. That is what is shown as the "Voxel server software" in Figure 3 of the Draft PSC Visible Human Overview that I sent last week. In the current incarnation however, that returns the URL for a mesh model of the appropriate tissue rather than a coded ID number. As you say this whole area is something we can talk about. We have enough flexability to handle a lot of variants on the theme so long as the server and client can coordinate. We're putting in a startup negotiation phase to be able to run a lot of varieties to test the effectiveness of different compressions, bandwidths, application requirements etc. From minc-development@bic.mni.mcgill.ca Sat Nov 23 02:48:54 2002 From: minc-development@bic.mni.mcgill.ca (Peter NEELIN) Date: Fri, 22 Nov 2002 21:48:54 -0500 Subject: [MINC-development] mincfft. was: [geeks] vector_diemsion and rawtominc In-Reply-To: Message-ID: On Thu, 21 Nov 2002, Andrew Janke wrote: > On Wed, 20 Nov 2002, Peter NEELIN wrote: > > > Yes. I think that the default behaviour is to use the minc library magic > > to make everything scalar. There is a volume_io option that allows this to > > be turned off (I don't have time to look it up right now). Be warned, > > however, that volume_io uses pointers to pointers to pointers for its > > arrays (it was faster that way at the time that David benchmarked it). The > > down side of this is that a complex vector short volume will have as much > > pointer as data, even on a 32-bit system (one pointer for every 4 bytes of > > data). > > Hrm, I see. So what you are saying is: rewrite it using ICV's like you should > have in the first place you slacker. :) No, not really. You can still use volume_io. I looked it up - you just need to declare "minc_input_options options;" and then call set_default_minc_input_options( &options ); set_minc_input_vector_to_scalar_flag( &options, FALSE ); if ( input_volume( ..., &options ) != OK ) { } As for the memory issue, you could either ignore it, or try to work around it by loading two separate scalar volumes that do not include vector_dimension (I think that you need to use advance_volume to get to the second volume). I have not tried this, but if you can get volume_io to treat the vector_dimension like any other dimension, it should work fine. I believe that your current volume (with a name other than vector_dimension) should suffer from the same issue if it is the fastest-varying dimension of the file. Given enough memory, you would not have noticed this problem. Peter ---- Peter Neelin (neelin@bic.mni.mcgill.ca) From minc-development@bic.mni.mcgill.ca Thu Nov 28 02:39:05 2002 From: minc-development@bic.mni.mcgill.ca (Andrew Janke) Date: Thu, 28 Nov 2002 12:39:05 +1000 Subject: [MINC-development] beta mincmorph Message-ID: For those who are interested I have updated the version of mincmorph (the eventuall replacement for binop) in: /s/s/minc_dev/mincmorph If you look at the code of the Group option (Connected component labelling -- in kernel_ops.c) you'll note I haven't used the classical approach to resolving equivalences. (Building an array of equivalences as you go with redundacies). Instead I've used an approach that I can't find anywhere else... but I'm sure has been described before. The reason for using it is that i can guarantee memory usage will only grow to a finite size this way, something I can't ensure with the classical version especially in 256^3 volumes. As such, if anyone else has seen this approach anywhere else I'm all ears as to whether I have missed something important. For all the test cases I've fed at it, it appears to be correct anyhow. On another somewhat related note, this version appears to be able to generate files that will crash register once it attempts to load the colourmap (after loading the file). I assume this is some range problem but I'm currently stumped as to where it is... Typical invocation to find groups in a file for data between the range 300 and 1000 in the file foo.mnc: mincmorph -verbose -successive 'B[300:1000]G' foo.mnc res.mnc This should produce a unsigned short volume as output (use -byte for byte output but this means that internally the group operator is going to have trouble if during the grouping it requires more tha 256 labels.) This bit I am unsure as to how to handle (internal precision) so again I'm interested in others comments. To then view the output file something like this is probably needed: mincmath -clamp -const2 0 100 res.mnc res_ranged.mnc or mincwindow .... Enough from me for now.... :) -- Andrew Janke ( rotor@cmr.uq.edu.au || www.cmr.uq.edu.au/~rotor ) Australia->University of Queensland->Centre for Magnetic Resonance Work: +61 7 3365 4100 || Home: +61 7 3800 4042