Cannot download files for ATLAS OpenData 2015

[Moelf]

trying to download this record: https://opendata.cern.ch/record/80000

[jiling@boslogin06 data2015]$ cernopendata-client download-files --recid 5500 --dry-run
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/BuildFile.xml
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/HiggsDemoAnalyzer.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/List_indexfile.txt
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall_lvl3.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3MC.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3data.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4MC.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4data.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.pdf
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png

[jiling@boslogin06 data2015]$ cernopendata-client download-files --recid 80000 --dry-run
# empty

[Moelf]

the problem is that the record_json["metadata"]["files"] is empty.

meanwhile, the record_json["metadata"] is populated:

{'$schema': 'http://opendata.cern.ch/schema/records/record-v1.0.0.json', '_bucket': '4417a2b6-6f8d-44b6-bdfe-a94666e893de', '_file_indices': [{'files': [{'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:09760a13', 'file_id': '6a704c04-dfa7-47c5-978e-804f6fe3a90e', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000001.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_0', 'size': 28222933, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000001.pool.root.1', 'version_id': '87e89653-2773-48af-8803-ec12c5eec047'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:4a131f9c', 'file_id': '56938512-50bd-4de7-8b20-28a17bf7d4a0', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000002.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_1', 'size': 45262824, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000002.pool.root.1', 'version_id': '6e19ff5e-696d-47c1-9df8-0c8032ef603c'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:54337636', 'file_id': 'afde9433-80f0-45f3-8722-807986ace1ed', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000003.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_2', 'size': 29636466, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000003.pool.root.1', 'version_id': 'cbc2201b-78bf-4a81-b74b-df7ba5bf24bf'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:d1ce9a8a', 'file_id': 'f2e75bf2-abdc-48c7-b9fe-4a3b0ff34b94', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000004.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_3', 'size': 33727180, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000004.pool.root.1', 'version_id': 'f1250de4-8d29-4535-beae-0a5ae91ef643'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:9ae15927', 'file_id': '97ae3138-76bb-4743-81fc-16fb61dcdf93', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000005.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_4', 'size': 24730470, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000005.pool.root.1', 'version_id': 'd91dadea-5ba3-4cfc-bfdf-5a3cc89a8141'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:4a0886ff', 'file_id': '1c763514-88d8-4b22-b1b7-c7ed8fc25bdf', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000006.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_5', 'size': 32379812, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000006.pool.root.1', 'version_id': '5c3a7094-358a-47d1-a546-1457f8975501'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:d87b23bc', 'file_id': '105ed3f0-214f-44ba-acc3-451c4c6a10df', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000007.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_6', 'size': 33903261, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000007.pool.root.1', 'version_id': '120a14a4-3b15-411d-aab3-1cef573b6c43'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:5e36a70b', 'file_id': '1c067d1b-b569-4a0d-867e-d6b0c9fe88de', 'filename': 'data15_13TeV:DAOD_PHYSLITE.37001626._000008.pool.root.1', 'key': 'data15_13TeV_Run_00266904_file_index.json_7', 'size': 33306512, 'uri': 'root://eospublic.cern.ch//eos/opendata/atlas/rucio/data15_13TeV/DAOD_PHYSLITE.37001626._000008.pool.root.1', 'version_id': '58b7d314-ec4e-41e8-8d93-94562e84ff8b'}, {'bucket': '8072ea11-6b5d-49ee-9af8-22eae14e4233', 'checksum': 'adler32:8d59df4e', 'file_id': '453dc613-fead-4e5d-a2d1-07281094606e', 'filename': 
...

[Moelf]

cc. @zlmarshall

[tiborsimko]

The problem is related to the recent deployment of the CERN Open Data portal which brought some changes as to how files attached via file indexes are being treated.

The problem has been ticketised in the portal repository already: cernopendata/cernopendata-portal#111

There is a tentative PR but it does not pass the unit tests yet: #147 (CC @psaiz)

Sorry for the troubles!

[zlmarshall]

Hi @Moelf ,

Just to add that since you're an ATLAS member you can temporarily work around this issue with pure rucio commands:

rucio list-dids "opendata:*PHYSLITE*"

and

rucio list-file-replicas opendata:data15_13TeV.00279764.physics_Main.deriv.DAOD_PHYSLITE.r13286_p4910_p6026

for example.

Cheers,
Zach

[gollumben]

Hi, is there any progress on this?

[tiborsimko]

Hi, is there any progress on this?

The problem has just been fixed in the master branch. You can install cernopendata-client from the master as follows, or just wait a few days for the new official release on PyPI. (Should happen later this week.)

mkvirtualenv test
pip install git+https://github.com/cernopendata/cernopendata-client@master
cernopendata-client get-file-locations --recid 80000

[gollumben]

Thanks!!