More robust shot data retrieval

[gtrevisan]

re-implement the good practice of closing the trees we opened after each shot.
if we catch an MDSplusERROR, it might mean that we encountered a severely-corrupted MDSplus tree.
in that case, the MDSplus connection itself might be compromised and would require a reconnection.
to mitigate this we purge any untrustworthy data related to our shot, but at least we will be able to retrieve healthy data for further shots processed by the same connection.

• catch any exception at setup
• reconnect if method fails
• catch any exception at cleanup
• close all trees at cleanup
• reconnect if cleanup failed
• better PID-specific logs for connections

tested with both a 1-proc and 2-proc job with a bad shot, followed by a good shot:

# C-MOD
poetry run disruption-py 1150528006 1150805012
# old -> [104 rows x 63 columns]
# new -> [187 rows x 63 columns]

# DIII-D
poetry run disruption-py 175316 161228
# old -> [165 rows x 59 columns]
# new -> [412 rows x 59 columns]

# EAST
poetry run disruption-py 72466 55555
# old ->  [92 rows x 79 columns]
# new -> [161 rows x 79 columns]

[yumouwei]

Here's a quick script I used to check if the bad shots affect the data fetched from the good shots. It passed for both CMOD & DIII-D. I'm having trouble testing on east EAST because VS Code release 1.99 stops supporting remote ssh into legacy linux distributions and I'm trying to sort that out.

Update: it works on EAST as well

from disruption_py.settings import RetrievalSettings
from disruption_py.workflow import get_shots_data
import random
from pandas.testing import assert_frame_equal

good_shots = [1150805012, 1150805015]
bad_shots = [1150528006, 1120831018, 1120914031, 1120918013]
all_shots = good_shots + bad_shots
random.shuffle(all_shots)

retrieval_settings = RetrievalSettings()

all_shot_data = get_shots_data(shotlist_setting = all_shots)
good_shot_data = get_shots_data(shotlist_setting = good_shots)

assert_frame_equal(
    good_shot_data[sorted(good_shot_data)].sort_values(by=['shot', 'time']).reset_index(drop=True), 
    all_shot_data[sorted(all_shot_data)].loc[all_shot_data['shot'].isin(good_shots)].sort_values(by=['shot', 'time']).reset_index(drop=True)
)