How to Publish std::stringstream & receive it on other side.

[piyushmz]

How do I publish a std::stringstream? following is my implementation. Please also give an idea how subscriber will receive it.

#include <sstream>
#include<string>
#include "iceoryx_dust/posix_wrapper/signal_watcher.hpp"
#include "iceoryx_posh/popo/publisher.hpp"
#include "iceoryx_posh/runtime/posh_runtime.hpp"

constexpr char APP_NAME[] = "iox-cpp-publisher-vector";

int main()
{
    // initialize runtime
    iox::runtime::PoshRuntime::initRuntime(APP_NAME);

    // initialize publisher
    //! [create publisher]
    iox::popo::Publisher<std::stringstream> publisher({"Radar", "FrontRight", "VectorData"});
    //! [create publisher]

    while (!iox::posix::hasTerminationRequested())
    {
        std::string s = "This is hello world and I want to send it to the publish this";
        auto tR = publisher.loan();
        std::stringstream stm;
        
        if (!tR.has_error())
        {
            auto& sample = tR.value();
            // sample->stm;
            sample->stm(s);
            sample.publish();
            std::cout << "This is ok\n";
        }
        else
            std::cout << "not OK\n";

        std::this_thread::sleep_for(std::chrono::seconds(1));
    }

    return (EXIT_SUCCESS);
}

[elBoberido]

@piyushmz a std::stringstream cannot be transferred via shared memory since it uses the heap. What you probably can do is to use our iox::cxx::string with a fixed capacity. You should be able to get a char* from the data of the string string and use the iox::cxx::string like iox::cxx::string<100> str{iox::cxx::TruncateToCapacity, stm.str().c_str()};. You just need to use a capacity which suits your needs.

[mossmaurice]

@piyushmz Besides the heap restriction you can find few more in this article. If you want to learn more about iceoryx's shared memory mechanism, refer to this article.

[piyushmz]

@elBoberido @mossmaurice I've the following structs defined. I've tried a lot of things but I'm not able to publish & receive this. If you have any better option for structuring this thing please do suggest. I want to publish Lanes and then pick these values by subscriber.
I've tried protobuf but I'm not able to convert this struct in proto file.
I've also tried vector of vector and that is working fine in iceoryx. So just need some help how this could be done or restructured so that I can easily send the data.

struct lane_coordinate
{
    float32_t x;
    float32_t y;
};

struct lane
{
    iox::cxx::vector<lane_coordinate, 100> l;
    uint32_t ptsPL;
};

struct lanePerFrame
{
    iox::cxx::vector<lane,100> lanes;
};

struct Lanes
{
    iox::cxx::vector<int,100> camID;
    iox::cxx::vector<lanePerFrame,100> lanepf;
};

Thanks.

[elfenpiff]

@piyushmz Your data structure looks actually perfect and this should be transmittable via iceoryx.

But when I take a look at the data construct it seems huge: sizeof(Lanes) == 8161216. I would be you got an error like:

022-11-18 12:59:51.392 [Fatal]: The following mempools are available:  MemPool [ ChunkSize = 168, ChunkPayloadSize = 128, ChunkCount = 10000 ]  MemPool [ ChunkSize = 1064, ChunkPayloadSize = 1024, ChunkCount = 5000 ]  MemPool [ ChunkSize = 16424, ChunkPayloadSize = 16384, ChunkCount = 1000 ]  MemPool [ ChunkSize = 131112, ChunkPayloadSize = 131072, ChunkCount = 200 ]  MemPool [ ChunkSize = 524328, ChunkPayloadSize = 524288, ChunkCount = 50 ]  MemPool [ ChunkSize = 1048616, ChunkPayloadSize = 1048576, ChunkCount = 30 ]  MemPool [ ChunkSize = 4194344, ChunkPayloadSize = 4194304, ChunkCount = 10 ]Could not find a fitting mempool for a chunk of size 8161256
2022-11-18 12:59:51.112 [Warn ]: ICEORYX error! MEPOO__MEMPOOL_GETCHUNK_CHUNK_IS_TOO_LARGE
iox-cpp-publisher-helloworld: /home/elchris/Development/iceoryx/iceoryx_hoofs/source/error_handling/error_handler.cpp:43: static void iox::ErrorHandler::reactOnErrorLevel(const iox::ErrorLevel, const char *): Assertion `false' failed.

You have here multiple options:

1. Configure a custom mem pool so so that we have memory chunks available which can hold this struct, see: https://github.com/eclipse-iceoryx/iceoryx/blob/master/doc/website/advanced/configuration-guide.md#configuring-mempools-for-roudi
2. Reduce the size of some members. Here it is important to know that the iceoryx vector guarantees you a capacity of 100 - a hard requirement for safety critical contexts - but with the downside that it always has the size as if it were full. When you take a look at the members you have:

• lane_coordinate == 8 byte
• lane = 100 * lane_coordinate + 4 = 800 byte
• lanePerFrame = 100 * 800 ~ 80 kb
• Lanes = 100 * 4 + 80 kb * 100 ~ 8 mb

3. Serialize the data but with this the advantage of zero copy is gone. For the serialization you require a copy and for deserialization you require a copy. It is likely that you will be still faster than transmitting the data via unix domain sockets but serialization will cost you some CPU time.
4. Restructure the data (best approach but this can be a complex task): In the end you make it more memory efficient, here a quick pseudo code example

struct lane_coordinate
{
    float32_t x;
    float32_t y;
};

struct lane
{
    iox::cxx::vector<lane_coordinate, 100> l;
    uint32_t ptsPL;
};

struct lanePerFrame
{
    // 1000 = maximum amount of lanes which will be tracked over frames
    iox::cxx::vector<lane,1000> lanes;
    uint32_t frameId;
};

struct Lanes
{
    iox::cxx::vector<int,100> camID;
    lanePerFrame lanepf;
};

Instead of always assuming the worst case that you have 100 lanes always to track in every frame you could assume that you over time have never to track more than 1000 lanes. Than you add a frameId to the lanePerFrame and increase here the number but in Lanes you only have now a single struct. When you take a look again at the data size you see:

• lane_coordinate == 8 byte
• lane = 100 * lane_coordinate + 4 = 800 byte
• lanePerFrame = 1000 * 800 ~ 800 kb
• Lanes = 100 * 4 + 800 kb ~ 800 kb

With this approach you reduced the data massively and should be able to send it via iceoryx without adjusting the mempool. If you can apply this approach also to other properties you may reduce the data even further.

[piyushmz]

@elfenpiff I was getting the same error but I thought it was my mistake so got a bit frustrated then switched to ecal and implemented with protobuf it worked(This also took a lot of my time). Now I'll try this and check it in my code. Now I've two good options over here for IPC.
Thank you so much. You were of great help. Thanks