(chore): updating and cleaning parametric.md

docs/scenarios/parametric.md

@@ -68,15 +68,10 @@ The following dependencies are required to run the tests locally:
 - Docker
 - Python 3.12
 
-then, run the following command, which will create a Python virtual environment and install the Python dependencies from the root directory:
-
-```sh
-./build.sh -i runner
-```
-
-
 ### Running the tests
 
+Build will happen at the beginning of the run statements.
+
 Run all the tests for a particular tracer library:
 
 ```sh
@@ -97,9 +92,36 @@ TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -k test_metrics_
 
 Tests can be aborted using CTRL-C but note that containers maybe still be running and will have to be shut down.
 
-#### Go
+### Using Pytest
+
+The tests are executed using pytest. Below are some common command-line options you can use to control and customize your test runs.
+- `-k EXPRESSION`: Run tests that match the given expression (substring or pattern). Useful for running specific tests or groups of tests.
+
+```sh
+TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -k test_metrics_msgpack_serialization_TS001
+```
+
+- `-v`: Increase verbosity. Shows each test name and its result (pass/fail) as they are run.
+
+```sh
+TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -v
+```
+
+- `-vv`: Even more verbose output. Provides detailed information including setup and teardown for each test.
+
+```sh
+TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -vv -k test_metrics_
+```
+
+- `-s`: Disable output capture. Allows you to see print statements and logs directly in the console.
+
+```sh
+TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -s
+```
+
+### Running the tests for a custom tracer
 
-For running the Go tests, see the README in apps/golang.
+#### Go
 
 To test unmerged PRs locally, run the following in the utils/build/docker/golang/parametric directory:
 
@@ -110,14 +132,14 @@ go mod tidy
 
 #### dotnet
 
-Add a file datadog-dotnet-apm-<VERSION>.tar.gz in binaries/. <VERSION> must be a valid version number.
+- Add a file `datadog-dotnet-apm-<VERSION>.tar.gz` in `binaries/`. `<VERSION>` must be a valid version number.
+  - One way to get that file is from an Azure pipeline (either a recent one from master if the changes you want to test were merged recently, or the one from your PR if it's open)
 
 #### Java
 
-##### Run Parametric tests with a custom Java Tracer version
+Follow these steps to run Parametric tests with a custom Java Tracer version:
 
-1. Clone the repo and checkout to the branch you'd like to test
-Clone the repo:
+1. Clone the repo and checkout to the branch you'd like to test:
 ```bash
 git clone git@github.com:DataDog/dd-trace-java.git
 cd dd-trace-java
@@ -168,18 +190,16 @@ From the repo root folder:
 
 #### Python
 
-To run the Python tests "locally" push your code to a branch and then specify ``PYTHON_DDTRACE_PACKAGE``.
-
-
-```sh
-TEST_LIBRARY=python PYTHON_DDTRACE_PACKAGE=git+https://github.com/Datadog/dd-trace-py@2.x ./run.sh PARAMETRIC [-k ...]
+To run the Python tests against a custom tracer:
+```bash
+echo “ddtrace @ git+https://github.com/DataDog/dd-trace-py.git@<name-of-your-branch>” > binaries/python-load-from-pip
 ```
 
 #### NodeJS
 
 There is three ways for running the NodeJS tests with a custom tracer:
 1. Create a file `nodejs-load-from-npm` in `binaries/`, the content will be installed by `npm install`. Content example:
-    * `DataDog/dd-trace-js#master`
+    - `DataDog/dd-trace-js#master`
 2. Clone the dd-trace-js repo inside `binaries`
 3. Create a file `nodejs-load-from-local` in `binaries/`, this will disable installing with `npm install dd-trace` and
    will instead get the content of the file, and use it as a location of the `dd-trace-js` repo and then mount it as a
@@ -202,41 +222,22 @@ There is two ways for running the C++ library tests with a custom tracer:
     * `https://github.com/DataDog/dd-trace-cpp@<COMMIT HASH>`
 2. Clone the dd-trace-cpp repo inside `binaries`
 
-The parametric shared tests can be run against the C++ library,
-[dd-trace-cpp][1], this way:
-```console
-$ TEST_LIBRARY=cpp ./run.sh PARAMETRIC
-```
-
-Use the `-k` command line argument, which is forwarded to [pytest][2], to
-specify a substring within a particular test file, class, or method. Then only
-matching tests will run, e.g.
-```console
-$ TEST_LIBRARY=cpp ./run.sh PARAMETRIC -k test_headers
-```
-
-It's convenient to have a pretty printer for the tests' XML output. I use
-[xunit-viewer][3].
-```console
-$ npm install junit-viewer -g
-```
-
-My development iterations then involve running the following at the top of the
-repository:
-```console
-$ TEST_LIBRARY=cpp ./run.sh PARAMETRIC -k test_headers; xunit-viewer -r logs_parametric/reportJunit.xml
+#### When you are done testing against a custom tracer:
+```bash
+rm -rf binaries/python-load-from-pip
 ```
 
-This will create a file `index.html` at the top of the repository, which I then
-inspect with a web browser.
-
-The C++ build can be made to point to a different GitHub branch by modifying the
-`FetchContent_Declare` command's `GIT_TAG` argument in [CMakeLists.txt][4].
+### Understanding the test outcomes
+Please refer to this chart:
 
-In order to coerce Docker to rebuild the C++ gRPC server image, one of the build
-inputs must change, and so whenever I push changes to the target branch, I also
-modify a scratch comment in `CMakeLists.txt` to trigger a rebuild on the next
-test run.
+| Declaration            | Test is executed  | Test actual outcome | System test output  | Comment
+| -                      | -                 | -                   | -                   | -
+| \<no_declaration>      | Yes               | ✅ Pass             | 🟢 Success          | All good :sunglasses:
+| Missing feature or bug | Yes               | ❌ Fail             | 🟢 Success          | Expected failure
+| Missing feature or bug | Yes               | ✅ Pass             | 🟠 Success          | XPASS: The feature has been implemented, bug has been fixed -> easy win
+| Flaky                  | No                | N.A.                | N.A.                | A flaky test doesn't provide any usefull information, and thus, is not executed.
+| Irrelevant             | No                | N.A.                | N.A                 | There is no purpose of running such a test
+| \<no_declaration>      | Yes               | ❌ Fail             | 🔴 Fail             | Only use case where system test fails : the test should have been ok, and is not
 
 ### Debugging
 
@@ -246,6 +247,7 @@ These can be used to debug the test case.
 The output also contains the commands used to build and run the containers which can be run manually to debug the issue
 further.
 
+The logs are contained in this folder:  `./logs_parametric`
 
 ## Troubleshooting
 
@@ -254,6 +256,14 @@ further.
 - Exiting the tests abruptly maybe leave some docker containers running. Use `docker ps` to find and `docker kill` any
   containers that may still be running.
 
+### Tests failing locally but not in CI
+
+A cause for this can be that the Docker image containing the APM library is cached locally with an older version of the
+library. Deleting the image will force a rebuild which will resolve the issue.
+
+```sh
+docker image rm <library>-test-library
+```
 
 ### Port conflict on 50052
 
@@ -278,16 +288,15 @@ are being produced then likely build kit has to be disabled.
 
 To do that open the Docker UI > Docker Engine. Change `buildkit: true` to `buildkit: false` and restart Docker.
 
-
-### Tests failing locally but not in CI
-
-A cause for this can be that the Docker image containing the APM library is cached locally with an older version of the
-library. Deleting the image will force a rebuild which will resolve the issue.
+### Docker Cleanup
+If you encounter an excessive number of errors during your workflow, one potential solution is to perform a cleanup of Docker resources. This can help resolve issues related to corrupted containers, dangling images, or unused volumes that might be causing conflicts.
 
 ```sh
-docker image rm <library>-test-library
+docker system prune
 ```
 
+**⚠️ Warning:**
+Executing `docker system prune` will remove all stopped containers, unused networks, dangling images, and build caches. This action is **irreversible** and may result in the loss of important data. Ensure that you **do not** need any of these resources before proceeding.
 
 ## Developing the tests
 
@@ -296,7 +305,26 @@ docker image rm <library>-test-library
 The Python implementation of the interface `app/python`, when run, provides a specification of the API when run.
 See the steps below in the HTTP section to run the Python server and view the specification.
 
-## Updating protos
+### Shared Interface
+
+#### HTTP
+
+We have transitioned to using an HTTP interface, replacing the legacy GRPC interface. To view the HTTP interface, follow these steps:
+
+1. ```
+./utils/scripts/parametric/run_reference_http.sh
+```
+
+2. Navigate to http://localhost:8000/docs in your web browser to access the documentation.
+
+3. You can download the OpenAPI schema from http://localhost:8000/openapi.json. This schema can be imported into tools like [Postman](https://learning.postman.com/docs/integrations/available-integrations/working-with-openAPI/) or other API clients to facilitate development and testing.
+
+#### Legacy GRPC
+**Important:** The legacy GRPC interface will be **deprecated** and no longer in use. All client interactions and tests will be migrated to use the HTTP interface.
+
+Previously, we used a shared GRPC interface to enable shared testing across different clients. Each client would implement the GRPC interface server, allowing shared tests to be run against the client libraries. The GRPC service definition included methods like StartSpan, FinishSpan, SpanSetMeta, and others, which facilitated span and trace operations.
+
+#### Updating protos for GRPC (will be deprecated)
 
 In order to update the `parametric/protos`, these steps must be followed.
 
@@ -327,51 +355,14 @@ cd utils/parametric
 
 Then you should have updated proto files. This script will generate weird files, you can ignore/delete these.
 
-## Implementation
-
-### Shared Interface
-
-#### HTTP
-
-An HTTP interface can be used instead of the GRPC. To view the interface run
-
-```
-./utils/scripts/parametric/run_reference_http.sh
-```
-
-and navigate to http://localhost:8000/docs. The OpenAPI schema can be downloaded at
-http://localhost:8000/openapi.json. The schema can be imported
-into [Postman](https://learning.postman.com/docs/integrations/available-integrations/working-with-openAPI/) or
-other tooling to assist in development.
-
-
-#### Legacy GRPC
-
-In order to achieve shared tests, we introduce a shared GRPC interface to the clients. Thus, each client need only implement the GRPC interface server and then these shared tests can be run against the library. The GRPC interface implements common APIs across the clients which provide the building blocks for test cases.
-
-```proto
-service APMClient {
-  rpc StartSpan(StartSpanArgs) returns (StartSpanReturn) {}
-  rpc FinishSpan(FinishSpanArgs) returns (FinishSpanReturn) {}
-  rpc SpanSetMeta(SpanSetMetaArgs) returns (SpanSetMetaReturn) {}
-  rpc SpanSetMetric(SpanSetMetricArgs) returns (SpanSetMetricReturn) {}
-  rpc SpanSetError(SpanSetErrorArgs) returns (SpanSetErrorReturn) {}
-  rpc InjectHeaders(InjectHeadersArgs) returns (InjectHeadersReturn) {}
-  rpc FlushSpans(FlushSpansArgs) returns (FlushSpansReturn) {}
-  rpc FlushTraceStats(FlushTraceStatsArgs) returns (FlushTraceStatsReturn) {}
-  rpc StopTracer(StopTracerArgs) returns (StopTracerReturn) {}
-}
-```
-
-
 ### Architecture
+Below is an overview of how the  testing architecture is structured:
+- Shared Tests in Python: We write shared test cases using Python's pytest framework. These tests are designed to be generic and interact with clients through the HTTP interface.
+- HTTP Servers in Docker: For each language client, we build and run an HTTP server within a Docker container. These servers expose the required endpoints defined in the OpenAPI schema and handle the client-specific logic.
+- [Test Agent](https://github.com/DataDog/dd-apm-test-agent/) in Docker: We start a test agent in a separate Docker container. This agent collects data (such as spans and traces) submitted by the HTTP servers. It serves as a centralized point for aggregating and accessing test data.
+- Test Execution: The Python test cases use an HTTP client to communicate with the servers. The servers generate data based on the interactions, which is then sent to the test agent. The tests can query the test agent to retrieve  data and perform assertions to verify correct behavior.
 
-- Shared tests are written in Python (pytest).
-- GRPC/HTTP servers for each language are built and run in docker containers.
-- [test agent](https://github.com/DataDog/dd-apm-test-agent/) is started in a container to collect the data from the GRPC servers.
-
-Test cases are written in Python and target the shared GRPC interface. The tests use a GRPC client to query the servers and the servers generate the data which is submitted to the test agent. Test cases can then query the data from the test agent to perform assertions.
-
+This architecture allows us to ensure that all clients conform to the same interface and behavior, making it easier to maintain consistency across different languages and implementations.
 
 <img width="869" alt="image" src="https://user-images.githubusercontent.com/6321485/182887064-e241d65c-5e29-451b-a8a8-e8d18328c083.png">