ElasticSearch Availabilty Check and Mapping Self-Check

[bodo-hugo-barwich]

For the issue detected and documented at
MetaCPAN API - Indexing failed
there are several features needed in the MetaCPAN::Script::Runner scripts
Seen in this Test Report
Dependency on MetaCPAN API
GitHub Testing Workflow
the successful test depends on a development in MetaCPAN API.

This change implements several features:

• in MetaCPAN::Role::Script:
  • exit_code
  • cluster_info
  • indices_info
  • aliases_info
  • arg_await_timeout (triggered with --await)
  • await() Method
  • check_health() Method

The Developer Documentation is also a bit more extended:

$ perldoc lib/MetaCPAN/Role/Script.pm |more
NAME
    MetaCPAN::Role::Script - Base Role which is used by many command line
    applications

SYNOPSIS
    Roles which should be available to all modules.

OPTIONS
    This Role makes the command line application accept the following
    options

    Option "--await 15"
        This option will set the *ElasticSearch Availability Check Timeout*.
        After "await" seconds the Application will fail with an Exception
        and the Exit Code [112] ("112 - EHOSTDOWN - Host is down") will be
        returned

            bin/metacpan <script_name> --await 15

        See "Method "await()""

METHODS
    This Role provides the following methods

    "await()"
        This method uses the
        "Search::Elasticsearch::Client::2_0::Direct::ping()"
        <https://metacpan.org/pod/Search::Elasticsearch::Client::2_0::Direct
        #ping()> method to verify the service availabilty and wait for
        "arg_await_timeout" seconds. When the service does not become
        available within "arg_await_timeout" seconds it re-throws the
        Exception from the "Search::Elasticsearch::Client" and sets $! to
        112 . The "Search::Elasticsearch::Client" generates a
        "Search::Elasticsearch::Error::NoNodes" Exception. When the service
        is available it will populate the "cluster_info" "HASH" structure
        with the basic information about the cluster. See "Option "--await
        15"" See "Method "check_health()""

    "check_health( [ refresh ] )"
        This method uses the
        "Search::Elasticsearch::Client::2_0::Direct::cat()"
        <https://metacpan.org/pod/Search::Elasticsearch::Client::2_0::Direct
        #cat()> method to collect basic data about the cluster structure as
        the general information, the health state of the indices and the
        created aliases. This information is stored in "cluster_info",
        "indices_info" and "aliases_info" as "HASH" structures. If the
        parameter "refresh" is set to 1 the structures "indices_info" and
        "aliases_info" will always be updated. If the "cluster_info"
        structure is empty it calls first the "await()" method. If the
        service is unavailable the "await()" method will produce an
        exception and the structures will be empty The method returns 1 when
        the "cluster_info" is populated, none of the indices in
        "indices_info" has the Health State *red* and at least one alias is
        created in "aliases_info" otherwise the method returns 0

    "are_you_sure()"
        Requests the user to confirm the operation with "* YES *"

    "handle_error( error_message[, die_always ] )"
        Logs the string "error_message" with the log function as fatal
        error. If "exit_code" is not equel 0 sets its value in $! . If the
        option "--die_on_error" is enabled it throws an Exception with
        "error_message". If the parameter "die_always" is set it overrides
        the option "--die_on_error".

    "print_error( error_message )"
        Logs the string "error_message" with the log function and displays
        it in red. But it does not end the application.

• in MetaCPAN::Script::Mapping it implements:
  • arg_cluster_info (triggered with option --show_cluster_info)
  • verify_mapping() Method
    which depends on data from deploy_mapping() and compares it with information gathered in indices_info and aliases_info populated through the method check_health()
  • show_info()
    which prints the information gathered by check_health() as JSON on the command line.

$ perldoc lib/MetaCPAN/Script/Mapping.pm|more |grep -iE "(show|\-\-all)"
     # bin/metacpan mapping --show_cluster_info   # show basic info about the cluster, indices and aliases

$ perldoc src/metacpan-api/lib/MetaCPAN/Script/Mapping.pm |more
NAME
    MetaCPAN::Script::Mapping - Script to set the index and mapping the
    types

SYNOPSIS
     # bin/metacpan mapping --show_cluster_info   # show basic info about the cluster, indices and a
liases
     # bin/metacpan mapping --delete
     # bin/metacpan mapping --list_types
     # bin/metacpan mapping --delete_index xxx
     # bin/metacpan mapping --create_index xxx --reindex
     # bin/metacpan mapping --create_index xxx --reindex --patch_mapping '{"distribution":{"dynamic"
:"false","properties":{"name":{"index":"not_analyzed","ignore_above":2048,"type":"string"},"river":{
"properties":{"total":{"type":"integer"},"immediate":{"type":"integer"},"bucket":{"type":"integer"}}
,"dynamic":"true"},"bugs":{"properties":{"rt":{"dynamic":"true","properties":{"rejected":{"type":"in
teger"},"closed":{"type":"integer"},"open":{"type":"integer"},"active":{"type":"integer"},"patched":
{"type":"integer"},"source":{"type":"string","ignore_above":2048,"index":"not_analyzed"},"resolved":
{"type":"integer"},"stalled":{"type":"integer"},"new":{"type":"integer"}}},"github":{"dynamic":"true
","properties":{"active":{"type":"integer"},"open":{"type":"integer"},"closed":{"type":"integer"},"s
ource":{"type":"string","index":"not_analyzed","ignore_above":2048}}}},"dynamic":"true"}}}}'
     # bin/metacpan mapping --create_index xxx --patch_mapping '{...mapping...}' --skip_existing_map
ping
     # bin/metacpan mapping --update_index xxx --patch_mapping '{...mapping...}'
     # bin/metacpan mapping --copy_to_index xxx --copy_type release
     # bin/metacpan mapping --copy_to_index xxx --copy_type release --copy_query '{"range":{"date":{
"gte":"2016-01","lt":"2017-01"}}}'
     # bin/metacpan mapping --delete_from_type xxx   # empty the type

DESCRIPTION
    This is the index mapping handling script. Used rarely, but carries the
    most important task of setting the index and mapping the types.

OPTIONS
    This Script accepts the following options

    Option "--show_cluster_info"
        This option makes the Script show basic information about the
        *ElasticSearch* Cluster and its indices and aliases. This
        information has to be collected with the
        "MetaCPAN::Role::Script::check_health()" Method. On Script start-up
        it is empty.

            bin/metacpan mapping --show_cluster_info

        See "Method "MetaCPAN::Role::Script::check_health()""

    Option "--delete"
        This option makes the Script delete all indices configured in the
        project and re-create them emtpy. It verifies the index integrity of
        the indices and aliases calling the methods
        "MetaCPAN::Role::Script::check_health()" and "verify_mapping()". If
        the "verify_mapping()" Method fails it exits the Script with Exit
        Code 1 .

            bin/metacpan mapping --delete

        See "Method "deploy_mapping()""

        See "Method "verify_mapping()""

        See "Method "MetaCPAN::Role::Script::check_health()""

METHODS
    This Package provides the following methods

    "deploy_mapping()"
        Deletes and re-creates the indices and aliases defined in the
        Project. The user will be requested for manual confirmation on the
        command line before the elemination. The integrity of the indices
        and aliases will be checked with the "verify_mapping()" Method. On
        successful creation it returns 1 , otherwise it returns 0 .

        Returns: It returns 1 when the indices and aliases are created and
        verified as correct. Otherwise it returns 0 .

        Exceptions: It can throw exceptions when the connection to
        *ElasticSearch* fails or there is any issue in any *ElasticSearch*
        Request run by the Script.

        See "Option "--delete""

        See "Method "verify_mapping()""

        See "Method "MetaCPAN::Role::Script::check_health()""

    "verify_mapping( \%indices, \%aliases )"
        Checks the defined indices and aliases against the actually in the
        *ElasticSearch* Cluster existing indices and aliases which must have
        been requested with the "MetaCPAN::Role::Script::check_health()"
        Method.

        Parameters:

        "\%indices" - Reference to a hash that defines the indices required
        for the Project.

        "\%aliases" - Reference to a hash that defines the aliases required
        for the Project.

        Returns: It returns 1 when the indices and aliases are created and
        verified as correct. Otherwise it returns 0 .

        Exceptions: It can throw exceptions when the connection to
        *ElasticSearch* fails or there is any issue in any *ElasticSearch*
        Request run by the Script.

        See "Option "--delete""

        See "Method "verify_mapping()""

        See "Method "MetaCPAN::Role::Script::check_health()""

[oalders]

@bodo-hugo-barwich would you be able to rebase and squash this down to just the necessary commits?

[bodo-hugo-barwich]

I created a new issue to discuss the implementation of the functionality to fix a broken index at
Functionality to fix a Corrupted ElasticSearch Index (#1048)
and moved the logic to a new branch to exclude it from this submission.

[mickeyn]

hi @bodo-hugo-barwich we are waiting for your action on the review comments.
please let me know how you want to proceed so we can merge this.

[bodo-hugo-barwich]

@oalders , @mickeyn I also cleaned all unnecessary commits.
Please, let me know if I missed out any blocking issue.

[mickeyn]

@bodo-hugo-barwich please check the current review comments that are pending.

[mohawk2]

@mickeyn I'm looking at this thread and I only see "resolved" conversations - can you confirm those comments aren't "pending" because you haven't sent them?

[bodo-hugo-barwich]

I moved the ElasticSearch Queries to 2 different methods, _get_indices_info() and _get_aliases_info(), which represent their function a bit better and I also assigned the regular expressions to variables so that their usage is a bit better documented.

Please let me know if this is better so or anything else should be done still.

[mickeyn]

LGTM

[oalders]

The latest build is broken: https://app.circleci.com/pipelines/github/metacpan/metacpan-api/155/workflows/207d384b-844b-4e37-8da4-5b768480d256/jobs/164

Any chance it's related to this? (I haven't looked closely).

[bodo-hugo-barwich]

The message :

Dubious, test returned 254 (wstat 65024, 0xfe00)
Crash in t/00_setup.t
indicates an Exception in the test.

Actually it is reproducible with

$ docker-compose exec -T api_test prove -lr --jobs 2 t
# Tests were run but no plan was declared and done_testing() was not seen.
t/00_setup.t ............................... 
Dubious, test returned 254 (wstat 65024, 0xfe00)
All 2 subtests passed 

There is also a message that the test ended unexpectedly

done_testing() was not seen.

Just running the test t/00_setup.t shows:

$ docker-compose exec -T api_test prove -vl t/00_setup.t
2022/02/20 17:04:41 I mapping: Awaiting Elasticsearch ...
[2022/02/20 17:04:41] [catalyst] [INFO] Awaiting Elasticsearch ...
2022/02/20 17:04:41 I mapping: Awaiting 0 / 15 : ready
[2022/02/20 17:04:41] [catalyst] [INFO] Awaiting 0 / 15 : ready
2022/02/20 17:04:41 I mapping: Deleting index: contributor
[2022/02/20 17:04:41] [catalyst] [INFO] Deleting index: contributor
2022/02/20 17:04:41 I mapping: Creating index: contributor
[2022/02/20 17:04:41] [catalyst] [INFO] Creating index: contributor
2022/02/20 17:04:41 I mapping: Adding mapping: contributor/contributor
[2022/02/20 17:04:41] [catalyst] [INFO] Adding mapping: contributor/contributor
2022/02/20 17:04:41 I mapping: Deleting index: cover
[2022/02/20 17:04:41] [catalyst] [INFO] Deleting index: cover
2022/02/20 17:04:41 I mapping: Creating index: cover
[2022/02/20 17:04:41] [catalyst] [INFO] Creating index: cover
2022/02/20 17:04:41 I mapping: Adding mapping: cover/cover
[2022/02/20 17:04:41] [catalyst] [INFO] Adding mapping: cover/cover
2022/02/20 17:04:41 I mapping: Deleting index: cpan_v1_01
[2022/02/20 17:04:41] [catalyst] [INFO] Deleting index: cpan_v1_01
2022/02/20 17:04:41 I mapping: Creating index: cpan_v1_01
[2022/02/20 17:04:41] [catalyst] [INFO] Creating index: cpan_v1_01
2022/02/20 17:04:42 I mapping: Adding mapping: cpan_v1_01/author
[2022/02/20 17:04:42] [catalyst] [INFO] Adding mapping: cpan_v1_01/author
2022/02/20 17:04:43 I mapping: Adding mapping: cpan_v1_01/distribution
[2022/02/20 17:04:43] [catalyst] [INFO] Adding mapping: cpan_v1_01/distribution
2022/02/20 17:04:43 I mapping: Adding mapping: cpan_v1_01/favorite
[2022/02/20 17:04:43] [catalyst] [INFO] Adding mapping: cpan_v1_01/favorite
2022/02/20 17:04:43 I mapping: Adding mapping: cpan_v1_01/file
[2022/02/20 17:04:43] [catalyst] [INFO] Adding mapping: cpan_v1_01/file
2022/02/20 17:04:43 I mapping: Adding mapping: cpan_v1_01/mirror
[2022/02/20 17:04:43] [catalyst] [INFO] Adding mapping: cpan_v1_01/mirror
2022/02/20 17:04:44 I mapping: Adding mapping: cpan_v1_01/package
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: cpan_v1_01/package
2022/02/20 17:04:44 I mapping: Adding mapping: cpan_v1_01/permission
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: cpan_v1_01/permission
2022/02/20 17:04:44 I mapping: Adding mapping: cpan_v1_01/rating
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: cpan_v1_01/rating
2022/02/20 17:04:44 I mapping: Adding mapping: cpan_v1_01/release
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: cpan_v1_01/release
2022/02/20 17:04:44 I mapping: Deleting index: user
[2022/02/20 17:04:44] [catalyst] [INFO] Deleting index: user
2022/02/20 17:04:44 I mapping: Creating index: user
[2022/02/20 17:04:44] [catalyst] [INFO] Creating index: user
2022/02/20 17:04:44 I mapping: Adding mapping: user/account
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: user/account
2022/02/20 17:04:44 I mapping: Adding mapping: user/identity
[2022/02/20 17:04:44] [catalyst] [INFO] Adding mapping: user/identity
2022/02/20 17:04:45 I mapping: Adding mapping: user/session
[2022/02/20 17:04:45] [catalyst] [INFO] Adding mapping: user/session
2022/02/20 17:04:45 I mapping: Creating alias: 'cpan' -> 'cpan_v1_01'
[2022/02/20 17:04:45] [catalyst] [INFO] Creating alias: 'cpan' -> 'cpan_v1_01'
[Request] ** [http://elasticsearch_test:9200]-[400] [invalid_alias_name_exception] Invalid alias name [cpan], an index exists with the same name as the alias, with: {"index":"cpan_v1_01"}, called from sub MetaCPAN::Script::Mapping::deploy_mapping at /metacpan-api/lib/MetaCPAN/Script/Mapping.pm line 142. With vars: {'status_code' => 400,'body' => {'status' => 400,'error' => {'type' => 'invalid_alias_name_exception','root_cause' => [{'type' => 'invalid_alias_name_exception','reason' => 'Invalid alias name [cpan], an index exists with the same name as the alias','index' => 'cpan_v1_01'}],'reason' => 'Invalid alias name [cpan], an index exists with the same name as the alias','index' => 'cpan_v1_01'}},'request' => {'method' => 'PUT','path' => '/cpan_v1_01/_alias/cpan','serialize' => 'std','qs' => {},'ignore' => [],'body' => undef}}
 at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Util.pm line 71.
	Search::Elasticsearch::Util::throw("Request", "[http://elasticsearch_test:9200]-[400] [invalid_alias_name_ex"..., HASH(0x55c1782c9190)) called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Cxn.pm line 172
	Search::Elasticsearch::Role::Cxn::process_response(Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8), 400, "Bad Request", "{\"error\":{\"root_cause\":[{\"type\":\"invalid_alias_name_exception"..., "application/json") called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Cxn/HTTP.pm line 135
	Search::Elasticsearch::Role::Cxn::HTTP::__ANON__[/usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Cxn/HTTP.pm:136](CODE(0x55c1782c8bc0), Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8), 400, "Bad Request", "{\"error\":{\"root_cause\":[{\"type\":\"invalid_alias_name_exception"..., HASH(0x55c17f684f00)) called at (eval 2255)[/usr/local/lib/perl5/site_perl/5.30.1/Class/Method/Modifiers.pm:89] line 1
	Search::Elasticsearch::Cxn::HTTPTiny::__ANON__[(eval 2255)[/usr/local/lib/perl5/site_perl/5.30.1/Class/Method/Modifiers.pm:89]:1](Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8), 400, "Bad Request", "{\"error\":{\"root_cause\":[{\"type\":\"invalid_alias_name_exception"..., HASH(0x55c17f684f00)) called at (eval 2257)[/usr/local/lib/perl5/site_perl/5.30.1/Class/Method/Modifiers.pm:148] line 2
	Search::Elasticsearch::Cxn::HTTPTiny::process_response(Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8), 400, "Bad Request", "{\"error\":{\"root_cause\":[{\"type\":\"invalid_alias_name_exception"..., HASH(0x55c17f684f00)) called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Cxn/HTTPTiny.pm line 42
	Search::Elasticsearch::Cxn::HTTPTiny::perform_request(Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8)) called at (eval 2251)[/usr/local/lib/perl5/site_perl/5.30.1/Class/Method/Modifiers.pm:148] line 6
	Search::Elasticsearch::Cxn::HTTPTiny::perform_request(Search::Elasticsearch::Cxn::HTTPTiny=HASH(0x55c17f5db358), HASH(0x55c17f67b9f8)) called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Transport.pm line 29
	Search::Elasticsearch::Transport::try {...} () called at /usr/local/lib/perl5/site_perl/5.30.1/Try/Tiny.pm line 102
	eval {...} called at /usr/local/lib/perl5/site_perl/5.30.1/Try/Tiny.pm line 93
	Try::Tiny::try(CODE(0x55c17f66ad30), Try::Tiny::Catch=REF(0x55c17f67bff8)) called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Transport.pm line 41
	Search::Elasticsearch::Transport::perform_request(Search::Elasticsearch::Transport=HASH(0x55c17f18c308), HASH(0x55c17f685050)) called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Client.pm line 16
	Search::Elasticsearch::Role::Client::perform_request(Search::Elasticsearch::Client::2_0::Direct::Indices=HASH(0x55c17f66d030), HASH(0x55c17f680a10), "index", "cpan_v1_01", "name", "cpan") called at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Client/Direct.pm line 103
	Search::Elasticsearch::Role::Client::Direct::__ANON__[/usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Role/Client/Direct.pm:104](Search::Elasticsearch::Client::2_0::Direct::Indices=HASH(0x55c17f66d030), "index", "cpan_v1_01", "name", "cpan") called at /metacpan-api/lib/MetaCPAN/Script/Mapping.pm line 492
	MetaCPAN::Script::Mapping::deploy_mapping(MetaCPAN::Script::Mapping=HASH(0x55c17f62f110)) called at /metacpan-api/lib/MetaCPAN/Script/Mapping.pm line 142
	MetaCPAN::Script::Mapping::run(MetaCPAN::Script::Mapping=HASH(0x55c17f62f110)) called at /usr/local/lib/perl5/site_perl/5.30.1/x86_64-linux-gnu/Class/MOP/Method/Wrapped.pm line 44
	MetaCPAN::Script::Mapping::_wrapped_run(MetaCPAN::Script::Mapping=HASH(0x55c17f62f110)) called at /usr/local/lib/perl5/site_perl/5.30.1/x86_64-linux-gnu/Class/MOP/Method/Wrapped.pm line 95
	MetaCPAN::Script::Mapping::run(MetaCPAN::Script::Mapping=HASH(0x55c17f62f110)) called at t/lib/MetaCPAN/TestServer.pm line 168
	MetaCPAN::TestServer::put_mappings(MetaCPAN::TestServer=HASH(0x55c1703fe868)) called at t/lib/MetaCPAN/TestServer.pm line 65
	MetaCPAN::TestServer::setup(MetaCPAN::TestServer=HASH(0x55c1703fe868)) called at t/00_setup.t line 38
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 2.
t/00_setup.t .. 
ok 1 - /metacpan-api/var/t/tmp exists for testing
ok 2 - got ElasticSearch object
# {
#   'Elasticsearch info' => {
#     'cluster_name' => 'elasticsearch',
#     'cluster_uuid' => '7JWOywmMSrCv2Vw-2bzlrQ',
#     'name' => 'Doctor Minerva',
#     'tagline' => 'You Know, for Search',
#     'version' => {
#       'build_hash' => '5376dca9f70f3abef96a77f4bb22720ace8240fd',
#       'build_snapshot' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
#       'build_timestamp' => '2017-07-18T12:17:44Z',
#       'lucene_version' => '5.5.4',
#       'number' => '2.4.6'
#     }
#   }
# }
Dubious, test returned 255 (wstat 65280, 0xff00)
All 2 subtests passed 

Test Summary Report
-------------------
t/00_setup.t (Wstat: 65280 Tests: 2 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=2, 13 wallclock secs ( 0.04 usr  0.00 sys +  8.23 cusr  0.39 csys =  8.66 CPU)
Result: FAIL

Noticeable here is the ElasticSearch response with HTTP Error Code 400:

2022/02/20 17:04:45 I mapping: Creating alias: 'cpan' -> 'cpan_v1_01'
[2022/02/20 17:04:45] [catalyst] [INFO] Creating alias: 'cpan' -> 'cpan_v1_01'
[Request] ** [http://elasticsearch_test:9200]-[400] [invalid_alias_name_exception] Invalid alias name [cpan], an index exists with the same name as the alias, with: {"index":"cpan_v1_01"}, called from sub MetaCPAN::Script::Mapping::deploy_mapping at /metacpan-api/lib/MetaCPAN/Script/Mapping.pm line 142. With vars: {'status_code' => 400,'body' => {'status' => 400,'error' => {'type' => 'invalid_alias_name_exception','root_cause' => [{'type' => 'invalid_alias_name_exception','reason' => 'Invalid alias name [cpan], an index exists with the same name as the alias','index' => 'cpan_v1_01'}],'reason' => 'Invalid alias name [cpan], an index exists with the same name as the alias','index' => 'cpan_v1_01'}},'request' => {'method' => 'PUT','path' => '/cpan_v1_01/_alias/cpan','serialize' => 'std','qs' => {},'ignore' => [],'body' => undef}}
 at /usr/local/lib/perl5/site_perl/5.30.1/Search/Elasticsearch/Util.pm line 71.

The ElasticSearch error message:

'reason' => 'Invalid alias name [cpan], an index exists with the same name as the alias'
can be confirmed studying the created indices:

$ docker-compose exec api_test /bin/bash
root@405583882f62:/metacpan-api# curl -XGET 'elasticsearch_test:9200/_cat/indices?pretty=true'
yellow open cover       1 1  0 0    159b    159b 
yellow open cpan_v1_01  1 1  0 0    159b    159b 
yellow open contributor 1 1  0 0    159b    159b 
yellow open cpan        5 1 29 1 192.1kb 192.1kb 
yellow open user        1 1  0 0    159b    159b 
root@405583882f62:/metacpan-api#

which is actually the same incident as documented in the related issue #47
Corrupted ElasticSearch index

The part that is not quite clear yet is how the wrong cpan index could be created again even with a successful indexation before.

So the saves way to proceed is to revert this commit and study this test again.

[bodo-hugo-barwich]

I'm sorry.
I was not aware of those tests but now I found at:
t/lib/MetaCPAN/TestServer.pm

    ok( MetaCPAN::Script::Mapping->new_with_options( $self->_config )->run,
        'put mapping' );

That the MetaCPAN::Script::Runner::run() methods are expected to return a TRUE value on success.
So the exit() instruction in MetaCPAN::Script::Mapping::run() breaks this.
Which then causes the test to exit and not do the necessary index construction as it is needed for the following tests.
Which was documented in the test outputs as:

# Tests were run but no plan was declared and done_testing() was not seen.

which can be reproduced with:

$ docker-compose exec api_test /bin/bash
root@dc9d27d6af54:/metacpan-api# perl -Ilib t/00_setup.t
ok 1 - /metacpan-api/var/t/tmp exists for testing
ok 2 - got ElasticSearch object
# {
#   'Elasticsearch info' => {
#     'cluster_name' => 'elasticsearch',
#     'cluster_uuid' => '7JWOywmMSrCv2Vw-2bzlrQ',
#     'name' => 'Kilmer',
#     'tagline' => 'You Know, for Search',
#     'version' => {
#       'build_hash' => '5376dca9f70f3abef96a77f4bb22720ace8240fd',
#       'build_snapshot' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
#       'build_timestamp' => '2017-07-18T12:17:44Z',
#       'lucene_version' => '5.5.4',
#       'number' => '2.4.6'
#     }
#   }
# }
*** Warning ***: this will delete EVERYTHING and re-create the (empty) indexes
Are you sure you want to do this (type "YES" to confirm) ? YES
alright then...
# Tests were run but no plan was declared and done_testing() was not seen.
root@dc9d27d6af54:/metacpan-api# exit
exit

So the MetaCPAN::Script::Mapping::run() method needs to be changed to finish with

    return ( $self->exit_code == 0 );

which then will make the tests succeed:

$ docker-compose exec api_test /bin/bash
root@51cdf071bd43:/metacpan-api# perl -Ilib t/00_setup.t
ok 1 - /metacpan-api/var/t/tmp exists for testing
ok 2 - got ElasticSearch object
# {
#   'Elasticsearch info' => {
#     'cluster_name' => 'elasticsearch',
#     'cluster_uuid' => '7JWOywmMSrCv2Vw-2bzlrQ',
#     'name' => 'Hybrid',
#     'tagline' => 'You Know, for Search',
#     'version' => {
#       'build_hash' => '5376dca9f70f3abef96a77f4bb22720ace8240fd',
#       'build_snapshot' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
#       'build_timestamp' => '2017-07-18T12:17:44Z',
#       'lucene_version' => '5.5.4',
#       'number' => '2.4.6'
#     }
#   }
# }
*** Warning ***: this will delete EVERYTHING and re-create the (empty) indexes
Are you sure you want to do this (type "YES" to confirm) ? YES
alright then...
ok 3 - put mapping
ok 4 - make fake cpan
ok 5 - index permissions
ok 6 - index packages
ok 7 - index releases
ok 8 - latest
ok 9 - first
ok 10 - index authors
ok 11 - prepare user
ok 12 - add pause identity
ok 13 - put user
ok 14 - put bot user
ok 15 - index cpantesters
ok 16 - index mirrors
ok 17 - index favorite
ok 18 - index cover
ok 19 - tickets
1..19

Unfortunately this will not correctly propagate the application "exit code" to implement a fail fast logic and prevent index corruption as explained in the issue.

[bodo-hugo-barwich]

I will refactor this and provide a better solution with it's corresponding tests.

[bodo-hugo-barwich]

According to the master branch revision 50332b1 as of 2021-10-27 the ElasticSearch index status after the succeeded tests is expected to be:

$ docker-compose exec api_test /bin/bash
root@bdca2655a1d9:/metacpan-api# curl -XGET 'elasticsearch_test:9200/_cat/indices?pretty=true'
yellow open cover       1 1    2   0  6.2kb  6.2kb 
yellow open cpan_v1_01  1 1 2301 274  2.4mb  2.4mb 
yellow open contributor 1 1   10   0  9.7kb  9.7kb 
yellow open user        1 1    7   0 24.7kb 24.7kb 
root@bdca2655a1d9:/metacpan-api# curl -XGET 'elasticsearch_test:9200/_cat/aliases?pretty=true'
cpan cpan_v1_01 - - - 

[bodo-hugo-barwich]

The refactoring at:
Capture Exit Code in MetaCPAN::Script::Runner
will resolve the tests and extend them over the new functionality.