Wait tokio tasks finish before CKB process exit

[eval-exec]

What problem does this PR solve?

Issue Number: close #3994

Problem Summary:

What is changed and how it works?

What's Changed:

Related changes

Implementation

• Add a std::threads JoinHandle register to collect all child threads' JoinHandle
• broadcast CKB's exit signal to child threads and tokio CancellationToken.
• wait all ckb threads finish before process exit.
• wait all ckb tokio spawned tasks finish before process exit.

Test

• Unit test for broadcast_exit_signals, wait_all_ckb_services_exit, register_thread, new_crossbeam_exit_rx and new_tokio_exit_rx
• Integration test: Bats clit test
  Ideas to implement integration test:
  1. start processes by ckb run
  2. wait a duration about 10 seconds, then kill ckb process.
  3. wait ckb process exit.
  4. assert there are log like all ckb threads have been stopped and ckb shutdown exist in the log file.
• Fix bat-cli timeout issue

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code ci-runs-only: [ quick_checks,linters ]

Side effects

• None

Release note

Title Only: Include only the PR title in the release note.

[chenyukang]

The name is only for printing messages?
so it's not an issue when multiple name are same.

[eval-exec]

Yes.

[chenyukang]

Should we add a new version for ckb-stop-handler?
I'm not sure whether any other projects are using it.

[eval-exec]

Yes, we should.

[chenyukang]

Did you verify the error log contains panic message?
https://stackoverflow.com/questions/69387239/what-is-in-err-when-i-try-to-join-a-panicked-thread

[eval-exec]

This WARN log didn't print thread's panic message, I should use:

    warn!("wait thread {}: ERROR: {:?}",name,e.downcast_ref::<&str>())

[eval-exec]

Update:
If the panic! use a formatter, e.downcast_ref::<&str>() will get None, so I have to use e.downcast_ref::<String>():

                let msg = e.downcast_ref::<str>().unwrap_or_else(|| {
                    e.downcast_ref::<String>()
                        .map(|s| s.as_str())
                        .unwrap_or("downcast_ref didn't get any error")
                });
                warn!("wait thread {}: ERROR: {:?}", name, msg)

[chenyukang]

can we add a test case, some sub-services terminated before receiving exit notification, and then the wait_all_ckb_services_exit also works for this scenario, maybe also double confirm the log messages.

[eval-exec]

Yes, we can.
I'd like to construct a bats cli test like this:

1. start a ckb process.
2. append some random bytes to ${DATA_DIR}/data/db/*.sst
3. wait the running ckb process' BlockDownload thread panic:

   ckb/sync/src/utils.rs

   Lines 171 to 173 in f6466c3

   	if error_kind == InternalErrorKind::DataCorrupted {
   	panic!("{}", error)
   	} else {

4. assert that wait_all_ckb_services_exit got the panic message.