Downloading data from command line

[calledbymountains]

Could a method for downloading the data from command line be provided ? I need to download the data to a cluster.

[Qiang-Xu]

Well, technically you can:

• Click the link to start download on browser.
• Find downloads page, copy the link(expires in several days) and cancel the download, for example on chrome
  
• Run wget "PASTE THE LINK YOU JUST COPIED HERE", for example: wget "https://s3.amazonaws.com/data.nuscenes.org/public/v1.0/v1.0-trainval03_blobs.tgz?AWSAccessKeyId=XXXXXXXXXXXX&Signature=YYYYYYYYYYYYYYYYYYY%3D&Expires=ZZZZZZZZZ", remember double quotes are necessary here.

One thing that's annoying here is wget saves the link with name like:

v1.0-trainval01_blobs.tgz?AWSAccessKeyId=XXXXXXXXXXX&Signature=YYYYYYYYYY%3D&Expires=ZZZZZZ

After all the downloads are done, you can rename them all:

ls v1.0-*tgz?* |while read line; do mv $line ${line%%\?*}; done

[eskjorg]

One thing that's annoying here is wget saves the link with name like:

v1.0-trainval01_blobs.tgz?AWSAccessKeyId=XXXXXXXXXXX&Signature=YYYYYYYYYY%3D&Expires=ZZZZZZ

You can use the -O flag:

wget -O new_filename.zip "PASTE THE LINK YOU JUST COPIED HERE"

[nlgranger]

Hi all. I suggest you use the cliget extension for firefox which generates the curl or wget command lines from the previous downloads. You just start the download regularly, cancel, and copy the command into your terminal/script.

[Syzygianinfern0]

https://github.com/Syzygianinfern0/NuPlan-Download-CLI
I've just hacked together a script to automate the download in the terminal entirely. Feel free to use it if you would like to.

[yiakwy-xpu-ml-framework-team]

In chrome, Instead of copy the "copy link address", I suggest you to copy the response :

{"url": "https://s3.amazonaws.com/data.nuscenes.org/public/v1.0/v1.0-mini.tgz?AWSAccessKeyId=ASIA6RIK4RRMPWMKBHKA&Signature=H%2B22siEjZyE4uEmQdNH6xRHFO7Q%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEK3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIHdSTXYUCCVtnH7yh5C0I%2BhUJkKLbzC5iHJgbA2%2BzOgnAiEAyRJjNCyw9BulhssJf1ukI%2F01tisAw79BuFz4wb9Q8lgq9AIIFhADGgw5OTkxMzk2MDk2ODgiDEe%2BKzW0gxtIQZhM3yrRAllwaC0zT%2B9OJsPjnT4jJpIzKM5%2BkJsXImFY%2BSZGYOsqf9emIqW3PAR1ASL8nwIGj4dwb%2Fg%2FxieaxyEZjoUEdRnV7DmkTvkPNa8GoV3D4Q2zyInmaIkkoF%2Fl1s9RnnzNAQDJ8gDdrszi%2FlqEOgJ7HoEl2VLK4viZLhBSft8kjEEe8xWAvmWV0EJL6eUvmWxxNS9lcWzAD7ZnGv2fkDzzG9D2EqcwPafyHGRb9jSA5iut5d2FYb0TpWFIrZT0wurM0olU3iNexyvBcFx0%2BsHU8vLlJ2eCYi0zXxAbmBSjK0%2B%2FjAf7qt4mYzr5YhjaIHzLYz6VN%2BvNW5m%2Bayxj6bwkBnQZ2AM0KzIYytgY8dA%2FcbyQl2ZJhdQDlAVOtKDAiYn0AaQKOdajqH25h5KBAWhzta5y4JtEALM1pXm%2Fh%2Fr4q0%2BLNcs%2BlDDnhATVpIovzw1DVVIw%2Fc7bpAY6ngGKibZ2um0cWOvrzyqFhNvWdIlSyp0b5RUVcbrA97h80%2B1DrJMDFFy5tlwPGtQmslNAHYBYT82R%2Fq19BWH1tBOeVKA0DsYHDbjlFLShPKN%2BYkYebsM53iUdifpNToMKk7aJahQvtho9YCqOLqeJr9J4N%2BSdk4X1UVAKQKcMx1Pikoly2EKa5ilqvYsSNHrgieuc31MQnMuGdVD76sRbmQ%3D%3D&Expires=1688044012"}

I used min-split for quick verification. That was proved to be enough:

v1.0-trainval ./data/nuscenes
======
Loading NuScenes tables for version v1.0-trainval...
23 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
Done loading in 0.658 seconds.
======
Reverse indexing ...
Done reverse indexing in 0.1 seconds.
======
total scene num: 10
exist scene num: 10
train scene: 6, val scene: 4
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 404/404, 9.0 task/s, elapsed: 45s, ETA:     0s

[MillionaireChen]

https://github.com/li-xl/nuscenes-download.git

try this maybe