[Loris-dev] Bulk Upload existing study

[Rida Abou-Haidar]

Hi Ryan,

Let me start by asking you a couple questions:
 - Is your data retrospective only or are you still collecting new data?
 - how is you data currently stored ? how is it organised ?

Depending on your answers to the above questions, you might wanna go
through LORIS to the DQT vs jumping directly into the DQT. Both of these
options have been done by other projects and I wouldn't say one is
easier/harder then the other. As far as the 100+ tables that exist in LORIS
are concerned, you would not need to worry about them if you decide to go
down that road.

Let me explain the considerations for each of these options in more details.

1. Importing data into MySQL then exporting to CouchDB: If you choose this
option you would first start by installing loris and configuring it to load
in the browser. Then you would have to set up your study, deciding what
your Projects would be, your Subprojects (cohorts), your
Sites, the different timepoints (visit labels) a candidate can participate
in and the test battery for each timepoint... The next step would be to
create the instruments you have in each session. the easiest way to do that
is to use the instrument builder to generate LINST files which are very
basic instruments that you can use to generate the necessary SQL tables.
Once these variables are configured you need to start importing the data
using the API, you can create your candidates and all the sessions
(assessments) by parsing your data and issuing API calls. You can then
import all the instrument data directly into their dedicated instrument
tables using the API again. Finally you would generate the data dictionary
and export the data to couchDB. this whole process usually takes a couple
of weeks.

2. Importing data directly into CouchDB: With this option you still need to
go through the install and configuration of the website however you don't
necessarily need to setup all your variables. you can just add the list of
possible timepoints (visit labels) a candidate can participate in and then
export the data to couchDB. However the challenge here is to properly
format your data during the import. the best way of doing that is to look
at how the Couchdb exporter tools format the data and mimic these scripts
for your own data. I've seen this method work for a couple projects but it
can get more or less complicated depending on you setup.

Best,
Rida Abou-Haidar
CCNA Software Developer
Montreal Neurological Institute
McGill University
rida.abou-haidar at mcin.ca


On Fri, May 24, 2019 at 1:11 PM Lim, Ryan (NKI) <Ryan.Lim at nki.rfmh.org>
wrote:

> Hi All,
>
>
> I'm currently evaluating using Loris for use as simple public interface
> for phenotypical data, and I was wondering how to best go about doing this.
> We currently have ~1500 subjects and >100,000 assessments, so entering
> information by hand will not be an option.
>
>
> The main thing that I will need users to access is the Data Query Tool, so
> I was wondering if it were possible to simply bulk upload data to the
> CouchDB instance and have the Data Query Tool work?
>
>
> If not, I suppose another option would be to bulk upload data to the MySQL
> database and use the scripts to populate the Data Query Tool, but I know
> there are >100 tables in MySQL, and very few would be relevant to me. Has
> anyone tried this before?
>
>
> Thanks,
>
> Ryan
>
> ------------------------------
>
> IMPORTANT NOTICE: This e-mail is meant only for the use of the intended
> recipient. It may contain confidential information which is legally
> privileged or otherwise protected by law. If you received this e-mail in
> error or from someone who was not authorized to send it to you, you are
> strictly prohibited from reviewing, using, disseminating, distributing or
> copying the e-mail. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN
> E-MAIL AND DELETE THIS MESSAGE FROM YOUR SYSTEM. Thank you for your
> cooperation.
> _______________________________________________
> Loris-dev mailing list
> Loris-dev at bic.mni.mcgill.ca
> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190528/f98635fd/attachment.html>