From Ryan.Lim at nki.rfmh.org Fri May 24 12:11:41 2019 From: Ryan.Lim at nki.rfmh.org (Lim, Ryan (NKI)) Date: Fri, 24 May 2019 16:11:41 +0000 Subject: [Loris-dev] Bulk Upload existing study Message-ID: Hi All, I'm currently evaluating using Loris for use as simple public interface for phenotypical data, and I was wondering how to best go about doing this. We currently have ~1500 subjects and >100,000 assessments, so entering information by hand will not be an option. The main thing that I will need users to access is the Data Query Tool, so I was wondering if it were possible to simply bulk upload data to the CouchDB instance and have the Data Query Tool work? If not, I suppose another option would be to bulk upload data to the MySQL database and use the scripts to populate the Data Query Tool, but I know there are >100 tables in MySQL, and very few would be relevant to me. Has anyone tried this before? Thanks, Ryan ________________________________ IMPORTANT NOTICE: This e-mail is meant only for the use of the intended recipient. It may contain confidential information which is legally privileged or otherwise protected by law. If you received this e-mail in error or from someone who was not authorized to send it to you, you are strictly prohibited from reviewing, using, disseminating, distributing or copying the e-mail. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND DELETE THIS MESSAGE FROM YOUR SYSTEM. Thank you for your cooperation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rida.abou-haidar at mcin.ca Tue May 28 13:43:44 2019 From: rida.abou-haidar at mcin.ca (Rida Abou-Haidar) Date: Tue, 28 May 2019 13:43:44 -0400 Subject: [Loris-dev] Bulk Upload existing study In-Reply-To: References: Message-ID: Hi Ryan, Let me start by asking you a couple questions: - Is your data retrospective only or are you still collecting new data? - how is you data currently stored ? how is it organised ? Depending on your answers to the above questions, you might wanna go through LORIS to the DQT vs jumping directly into the DQT. Both of these options have been done by other projects and I wouldn't say one is easier/harder then the other. As far as the 100+ tables that exist in LORIS are concerned, you would not need to worry about them if you decide to go down that road. Let me explain the considerations for each of these options in more details. 1. Importing data into MySQL then exporting to CouchDB: If you choose this option you would first start by installing loris and configuring it to load in the browser. Then you would have to set up your study, deciding what your Projects would be, your Subprojects (cohorts), your Sites, the different timepoints (visit labels) a candidate can participate in and the test battery for each timepoint... The next step would be to create the instruments you have in each session. the easiest way to do that is to use the instrument builder to generate LINST files which are very basic instruments that you can use to generate the necessary SQL tables. Once these variables are configured you need to start importing the data using the API, you can create your candidates and all the sessions (assessments) by parsing your data and issuing API calls. You can then import all the instrument data directly into their dedicated instrument tables using the API again. Finally you would generate the data dictionary and export the data to couchDB. this whole process usually takes a couple of weeks. 2. Importing data directly into CouchDB: With this option you still need to go through the install and configuration of the website however you don't necessarily need to setup all your variables. you can just add the list of possible timepoints (visit labels) a candidate can participate in and then export the data to couchDB. However the challenge here is to properly format your data during the import. the best way of doing that is to look at how the Couchdb exporter tools format the data and mimic these scripts for your own data. I've seen this method work for a couple projects but it can get more or less complicated depending on you setup. Best, Rida Abou-Haidar CCNA Software Developer Montreal Neurological Institute McGill University rida.abou-haidar at mcin.ca On Fri, May 24, 2019 at 1:11 PM Lim, Ryan (NKI) wrote: > Hi All, > > > I'm currently evaluating using Loris for use as simple public interface > for phenotypical data, and I was wondering how to best go about doing this. > We currently have ~1500 subjects and >100,000 assessments, so entering > information by hand will not be an option. > > > The main thing that I will need users to access is the Data Query Tool, so > I was wondering if it were possible to simply bulk upload data to the > CouchDB instance and have the Data Query Tool work? > > > If not, I suppose another option would be to bulk upload data to the MySQL > database and use the scripts to populate the Data Query Tool, but I know > there are >100 tables in MySQL, and very few would be relevant to me. Has > anyone tried this before? > > > Thanks, > > Ryan > > ------------------------------ > > IMPORTANT NOTICE: This e-mail is meant only for the use of the intended > recipient. It may contain confidential information which is legally > privileged or otherwise protected by law. If you received this e-mail in > error or from someone who was not authorized to send it to you, you are > strictly prohibited from reviewing, using, disseminating, distributing or > copying the e-mail. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN > E-MAIL AND DELETE THIS MESSAGE FROM YOUR SYSTEM. Thank you for your > cooperation. > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: