[Loris-dev] BIds Insertion

Fri Dec 6 14:29:55 EST 2019

Hi all,

I have problem with the insertion of the bids dataset in my server with loris 22.0.
I am testing with a short dataset and passing this parameters:

Python bids_import -d /dataset_bids -p database_config.py  -c  -s  -v

Trace and Error:

Connecting to:
        database: CBMCUBA
        username: lorisuser
        hostname: localhost
        port    : 3306

Executing query:
        SELECT Value FROM Config WHERE ConfigID = (SELECT ID FROM ConfigSettings WHERE Name = %s);

With arguments:
        ('default_bids_vl',)

Executing query:
        SELECT Value FROM Config WHERE ConfigID = (SELECT ID FROM ConfigSettings WHERE Name = %s);

With arguments:
        ('dataDirBasepath',)

Loading the BIDS dataset with BIDS layout library...

        => BIDS dataset loaded with BIDS layout

Grepping candidates from the BIDS layout...
Validating the list of participants...
        => Passed validation of the list of participants

        => List of participants found:
                MC0002270
                MC0002275
                MC0002339
                MC0002347
                MC0002352
                MC0002362

Grepping list of sessions from the BIDS layout...
        => List of sessions found:

                MC0002270: 1
                MC0002339: 1
                MC0002275: 1
                MC0002362: 1
                MC0002352: 1
                MC0002347: 1

Grepping the different modalities from the BIDS layout...
        => Done grepping the different modalities from the BIDS layout

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/
Copying file /home/lorisadmin/dataset_test/dataset_description.json to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/dataset_description.json
Copying file /home/lorisadmin/dataset_test/README to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/README
Copying file /home/lorisadmin/dataset_test/participants.tsv to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/participants.tsv

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002270',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('586022', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002275',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002275

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('531254', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002275/ses-1

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002339',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002339

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('854503', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002339/ses-1

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002347',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002347

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('615335', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002347/ses-1

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002352',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002352

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('606661', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002352/ses-1

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002362',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002362

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('827956', '1')

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002362/ses-1
Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/anat/

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002270',)

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('586022', '1')

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/anat/sub-MC0002270_T1w.json to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/anat/sub-MC0002270_T1w.json

Executing query:
        SELECT type FROM ImagingFileTypes

Executing query:
        SELECT f.FileID, f.File FROM files AS f JOIN parameter_file USING (FileID) JOIN parameter_type USING (ParameterTypeID) WHERE Value=%s

With arguments:
        ('2bc67d1f0ffa5f5860f0c71f31cc517adf364d5a9503ab402758fa0e0836e14e956354fd992be4a0dbaf3f0454ead82b824a718853b09ab3ed83503ead37e7ab',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s

With arguments:
        ('MC0002270',)

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('586022', '1')

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.json to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.json

Executing query:
        SELECT type FROM ImagingFileTypes

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.bvec to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.bvec
Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.bval to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-01_dwi.bval

Executing query:
        SELECT f.FileID, f.File FROM files AS f JOIN parameter_file USING (FileID) JOIN parameter_type USING (ParameterTypeID) WHERE Value=%s

With arguments:
        ('1a075f0c10b856888e728c5421791cf6d18e363ca45255df46f4161460173f8153907173ae3840ac383e68d5f271eadc54bf03ea52fae737f758f376819295b3',)

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.json to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.json

Executing query:
        SELECT type FROM ImagingFileTypes

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.bvec to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.bvec
Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.bval to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/dwi/sub-MC0002270_run-02_dwi.bval

Executing query:
        SELECT f.FileID, f.File FROM files AS f JOIN parameter_file USING (FileID) JOIN parameter_type USING (ParameterTypeID) WHERE Value=%s

With arguments:
        ('bef678b4383bde1df93411e05bf75d8bccb5adf985e610ee7029c74056397ca43449dc20c187fa553ac0487a5adff67dc9e49827a8c227382768968c0444c9e7',)

Creating directory /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/eeg/

Executing query:
        SELECT * FROM candidate WHERE PSCID = %s
/data/loris/bin/mri/python_virtualenvs/loris-mri-python/lib/python3.5/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
/data/loris/bin/mri/python_virtualenvs/loris-mri-python/lib/python3.5/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.utils.testing module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.utils. Anything that cannot be imported from sklearn.utils is now part of the private API.
  warnings.warn(message, FutureWarning)
/data/loris/bin/mri/python_virtualenvs/loris-mri-python/lib/python3.5/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.datasets.base module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.datasets. Anything that cannot be imported from sklearn.datasets is now part of the private API.
  warnings.warn(message, FutureWarning)

With arguments:
        ('MC0002270',)

Executing query:
        SELECT * FROM session WHERE CandID = %s AND Visit_label = %s

With arguments:
        ('586022', '1')

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.json to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.json

Executing query:
        SELECT type FROM ImagingFileTypes

Executing query:
        SELECT  PhysiologicalOutputTypeID FROM    physiological_output_type WHERE   OutputTypeName = %s

With arguments:
        ('raw',)

Executing query:
        SELECT pf.PhysiologicalFileID, pf.FilePath FROM physiological_file AS pf JOIN physiological_parameter_file USING (PhysiologicalFileID) JOIN parameter_type USING (ParameterTypeID) WHERE Value=%s

With arguments:
        ('2962f54555ebd038297d3056cbad12379a3a293cb2128930f76a1da8e033987e7f6440632b00e585c22826879108c00f535e5e2f205ed5107c04ee7ae51c3282',)

Executing query:
        SELECT  PhysiologicalModalityID FROM    physiological_modality WHERE   PhysiologicalModality = %s

With arguments:
        ('eeg',)

Copying file /home/lorisadmin/dataset_test/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.edf to /data/loris/data/bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.edf

Executing query:
        INSERT INTO physiological_file (AcquisitionTime, SessionID, PhysiologicalOutputTypeID, FileType, InsertedByUser, PhysiologicalModalityID, FilePath) VALUES (%s,%s,%s,%s,%s,%s,%s)
With arguments:
        [(None, 1, 1, 'set', 'lorisadmin', 1, 'bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.edf')]

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('json_file',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 109, 'bids_imports/Dataset_containing_Cuban_Brain_Mapping_database_BIDSVersion_1.2.1/sub-MC0002270/ses-1/eeg/sub-MC0002270_task-protmap_eeg.json')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('EMGChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 101, 0)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('PowerLineFrequency',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 108, 60)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('EEGPlacementScheme',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 116, '10-10')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('ECGChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 112, 0)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('physiological_file_blake2b_hash',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 107, '2962f54555ebd038297d3056cbad12379a3a293cb2128930f76a1da8e033987e7f6440632b00e585c22826879108c00f535e5e2f205ed5107c04ee7ae51c3282')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('TriggerChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 115, 0)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('physiological_json_file_blake2b_hash',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 105, 'b3c94060b316453cfc2816eb57cc9012377bcfe345dab476a2b6507651d1fd0b79b5987635659439c91ac44156d49ec191c9faf694e788d7919be79e8d87e3ad')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('TaskDescription',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 104, 'Protocol used in Cuban Brain Mapping Project')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('SoftwareFilters',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 100, 'n/a')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('EOGChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 106, 0)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('SamplingFrequency',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 102, 200)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('MiscChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 110, 0)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('RecordingType',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 114, 'continuous')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('EEGChannelCount',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 99, 62)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('RecordingDuration',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 111, 1801.42)

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('EEGReference',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 113, 'common')

Executing query:
        SELECT ParameterTypeID FROM parameter_type WHERE Name = %s AND SourceFrom='physiological_parameter_file'

With arguments:
        ('TaskName',)

Executing query:
        INSERT INTO physiological_parameter_file (PhysiologicalFileID, ParameterTypeID, Value) VALUES (%s,%s,%s)
With arguments:
        (3, 103, 'protmap')

Traceback (most recent call last):
  File "bids_import.py", line 431, in <module>
    main()
  File "bids_import.py", line 75, in main
    read_and_insert_bids(bids_dir, config_file, verbose, createcand, createvisit)
  File "bids_import.py", line 216, in read_and_insert_bids
    loris_bids_root_dir    = loris_bids_root_dir
  File "/data/loris/bin/mri/python/lib/eeg.py", line 143, in __init__
    self.register_raw_data()
  File "/data/loris/bin/mri/python/lib/eeg.py", line 264, in register_raw_data
    inserted_eeg  = self.fetch_and_insert_eeg_file()
  File "/data/loris/bin/mri/python/lib/eeg.py", line 473, in fetch_and_insert_eeg_file
    fdt_full_path = eeg_file_data['fdt_file']
KeyError: 'fdt_file'

But with this error I can see that the candidates was inserted in the database, so which I can't see in the formulary?

[cid:image003.jpg at 01D5AC41.A11DBCB0]

How I can solve this problem, any idea?,

Thanks,
Iris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20191206/6ebf3c15/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 11106 bytes
Desc: image003.jpg
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20191206/6ebf3c15/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: candidate-table.png
Type: image/png
Size: 32356 bytes
Desc: candidate-table.png
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20191206/6ebf3c15/attachment-0001.png>