How to properly configure queueCapacity and historyCapacity?

[shuhaowu]

My understanding of the queueCapacity and historyCapacity is relatively rudimentary. Based on the docs, it seems like queueCapacity is the capacity of the queue at the subscriber side, where as historyCapacity seems like a buffer on the publisher side, in case some subscribers are arriving late. So far so good.

Let's start simple with an application with a single subscriber and a single publisher with the following characteristics:

1. The publisher is publishing data at 1000 Hz. The publisher thread is an RT thread and shouldn't block.
2. The subscriber wakes up every 1 second to read the data and logs it. Assume the data read and log takes <10ms.
3. We don't care if the subscriber is late joining, as the system creates the subscriber before the publisher and thus this shouldn't happen.
4. The subscriber should get every data emitted by the publisher.
5. Memory usage is unimportant.

Under these assumptions, I would choose the following capacities:

• queueCapacity = 2000: This value needs to be a minimum of 1000. Since we never want to lose data, I'm multiplying by 2 to be safe (memory usage is unimportant).
• historyCapacity = 0: We don't need this feature.
• historyRequest = 0: We don't need this feature.

First of all, is this thinking correct? Am I misinterpreting these options somehow? Second, when I set this, I get a message saying that:

Requested queue capacity 2000 exceeds the maximum possible one for this subscriber, limiting from 2000 to 256

It seems like this is governed by IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY. Why is this constant set to 256? What's the drawbacks of increasing these? I see the various Iceoryx options are documented here, but the descriptions are pretty basic and it doesn't discuss the implications of changing variables like IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY and IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY, which seems important from a correctness and memory perspective.

[elfenpiff]

@shuhaowu

seems like this is governed by IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY

You are correct.

Why is this constant set to 256?

We had to start somewhere and 256 seems to be a number which fits 99% of all use cases. Congratulations you are now in the 1% use case 😄

the implications of changing variables like IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY

This defines the capacity of the internal fixed size queues which are storing the samples. You can increase those numbers to 2000 but this would massively affect the size of a subscriber port stored inside the management segment in roudi. If memory is not important to you , you can go ahead and adjust it but be aware that the management segment /dev/shm/iceoryx_mgmt can become several gigabytes in size - but I never measured it.

the implications of changing variables like IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY

This and the previous constant also affects the internal sample tracking. When an application crashes roudi has to cleanup the samples they reserved and therefore it has to track which application owns what samples. This also increases the size of the iceoryx management segment.

[shuhaowu]

Thanks for the detailed answers. One additional question: how much memory are we talking about here? Is it based on the size of the struct that I'm passing around? Let's say if I have

struct A {
  int64_t a;
  int64_t b;
}

If I reserve 2000 elements, won't this only be 2000 * (8 + 8) = 32000 bytes = 31 KiB? Is there a significant amount of overhead? How can I calculate how much RAM is being used?

[elfenpiff]

@shuhaowu It is independent of the data, it stores pointers to the underlying shared memory chunks. So to get the memory size required you have to follow this rough calculations:

1. SIZE_OF_ALL_SUBSCRIBERS = IOX_MAX_SUBSCRIBERS * (IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY * sizeof(void*) + sizeof(SubscriberPortData))
2. SIZE_OF_ALL_PUBLISHERS = IOX_MAX_PUBLISHERS * ( IOX_MAX_SUBSCRIBERS + sizeof(PublisherPortData))
3. ROUDI_MGMT = IOX_MAX_PUBLISHERS * IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY * sizeof(void) + IOX_MAX_SUBSCRIBERS * IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY * sizeof(void*)

The overall memory size should be then something around: SIZE_OF_ALL_SUBSCRIBERS + SIZE_OF_ALL_PUBLISHERS + ROUDI_MGMT + SOME_OTHER_MGMT_DATA

Here the results when I compile iceoryx with the default config

$ ls -alh /dev/shm
-rw-rw----+  1 me users 143M Dez  6 11:58 users // <- here resides the payload
-rw-rw----   1 me users  64M Dez  6 11:58 iceoryx_mgmt // <- here are the port residing
# mem usage iox-roudi: 256M (acquired with htop)

And here the results when I use IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY=2000 and IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY=2000

$ ls -alh /dev/shm
-rw-rw----+  1 me users 143M Dez  6 12:00 users
-rw-rw----   1 me users 151M Dez  6 12:00 iceoryx_mgmt
# mem usage iox-roudi: 342M (acquired with htop)

[shuhaowu]

Thanks for the detailed answer. I'm still trying to grasp how Iceoryx works internally, especially around how it actually passes data around, especially since you mention the memory usage is independent of the size of the data I'm passing around.

I've found some documentations that I should read once I have a bit more time: chunk_header.md, and lockfree_queue.md. Are there any other docs I should read? Thanks again.

[elfenpiff]

@shuhaowu maybe the relocatable_pointer.md could be also interesting for you. Different shared memory segments have different memory offsets and a pointer from an application A may points to the wrong location in application B when the memory offsets differ - the relocatable pointer solves this issue.