Rework Python callback functions.

[trivialfis]

This PR adds a new callback function interface for Python, also a new type of callback for checkpointing is added.

Please see documentation and demo for details. I added a demo for plotting animated evaluation scores, try it out. ;-)

Related #5612 I haven't made the parameter to dask. The PR is already huge as it is.
Related #3822 this is only for Python. Other language bindings are not supported.
Close #4955
Close #5495 now we can use early stopping on dask.

[hcho3]

@trivialfis Is this ready for review? Do you want me to review this?

[trivialfis]

@hcho3 I ran some of the tests locally. Feel free to review once the CI is green.

[hcho3]

CI might fail due to a Conda issue. Related: https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-6198/1/pipeline

[codecov-commenter]

Codecov Report

Merging #6199 into master will increase coverage by 0.82%.
The diff coverage is 95.95%.

@@            Coverage Diff             @@
##           master    #6199      +/-   ##
==========================================
+ Coverage   78.93%   79.75%   +0.82%     
==========================================
  Files          12       12              
  Lines        3104     3359     +255     
==========================================
+ Hits         2450     2679     +229     
- Misses        654      680      +26     

Impacted Files	Coverage Δ
python-package/xgboost/training.py	96.41% <94.59%> (+0.89%)	⬆️
python-package/xgboost/callback.py	88.39% <96.10%> (+0.39%)	⬆️
python-package/xgboost/dask.py	81.10% <100.00%> (+0.03%)	⬆️
python-package/xgboost/sklearn.py	91.68% <0.00%> (+0.24%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1013224...cc1ec18. Read the comment docs.

[trivialfis]

Note to myself:

Maybe add a hook for customizable allreduce.

[RAMitchell]

I like this reorganisation. Nice to see checkpointing as a callback, and very nice demo with the plot!

[JohnZed]

Looks good! Just added very trivial typo suggestion

[hcho3]

LGTM. This pull request is a great testament to the power of good abstractions in programming. Some minor comments follow:

[trivialfis]

@terrytangyuan @RAMitchell @JohnZed @hcho3 Thanks for the review!

[codecov-io]

Codecov Report

Merging #6199 into master will increase coverage by 1.31%.
The diff coverage is 95.98%.

@@            Coverage Diff             @@
##           master    #6199      +/-   ##
==========================================
+ Coverage   78.93%   80.24%   +1.31%     
==========================================
  Files          12       12              
  Lines        3104     3362     +258     
==========================================
+ Hits         2450     2698     +248     
- Misses        654      664      +10     

Impacted Files	Coverage Δ
python-package/xgboost/training.py	96.46% <94.80%> (+0.94%)	⬆️
python-package/xgboost/callback.py	92.61% <96.10%> (+4.61%)	⬆️
python-package/xgboost/dask.py	81.10% <100.00%> (+0.03%)	⬆️
python-package/xgboost/sklearn.py	91.68% <0.00%> (+0.24%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5b2435...8e45139. Read the comment docs.