qa: barbican: restrict python packages with upper-constraints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

… ctor

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Fixes: https://tracker.ceph.com/issues/67604
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

(recover/backfill)_delete should remove oids in the snap mapper together
with the objects

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Fixes: https://tracker.ceph.com/issues/67564
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

…-ool-writes

Revert "crimson/os/seastore: wait ool writes in DeviceSubmission phase"

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Fixes https://tracker.ceph.com/issues/67223

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

rbd: fix CLI output of `rbd group snap info` command when a group snapshot with no member images

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Sunil Angadi <Sunil.Angadi@ibm.com>

Presently, the requests array in the REST module has the potential to grow
indefinitely, leading to excessive memory consumption, particularly when
dealing with lengthy and intricate request results.

To address this issue, a limit will be imposed on the requests array within
the REST module.
This limitation will be governed by the `mgr/restful/x/max_requests` configuration
parameter specific to the REST module.
when submit_request called we will check request array if exceed max_request option
if it does we will check if the future trimmed request finished and log error
message in case we are trimming un-finished requests.

Fixes: https://tracker.ceph.com/issues/59580
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

…max_request option

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

…ols-newline-trim

test: test_rados_tools compare output without trimming newline

mgr/cephadm: require "group" parameter in nvmeof specs

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

adding support for SSO based on auth2-proxy

Reviewed-by: Adam King <adking@redhat.com>

Explain how to deal with "unfound objects" when restarting OSDs in a
cache-tiered environment.

Fixes: https://tracker.ceph.com/issues/44286

Signed-off-by: Zac Dover <zac.dover@proton.me>

…guration

Fixes https://tracker.ceph.com/issues/67629

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Fixes https://tracker.ceph.com/issues/67629

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Add a new context getter function to fetch a daemon's rank and rank
generation value.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Allow subcomponent names up to 32 chars long.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Add parameters related to ctdb clustering to the smb service
deployment spec.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

The clustered mode of a logical smb cluster needs certain additional
capabilities in the rados pool. Improve, reorganize the key
configuration functions, and add the new caps.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Add support for using rados object locks to the rados store classes.
Callers directly using the rados store outside the store interface will
be able to make use of locking.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Add a new module clustermeta that implements a JSON based interface
compatible with sambacc. This module will be called directly by cephadm
as it places the daemons on the cluster nodes.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Add support to the smb service module so that cephadm will provide
information about the layout of the smb daemons to the clustermeta
module that, in turn, will provide the information sambacc needs to
configure ctdb.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Add a new `clustering` field to the smb cluster resource. This field can
be used to select either automatic clustering with ctdb, or disable it,
or require it. The default is automatic and is based on the count value
in the placement spec. A count of 1 disables clustering and any other
value it is enabled.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Encapsulate the logic determining the name of a stray service into a
method reducing the length and levels of indent in the stray checker
function.

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Extend the system's current stray service detection with a new method on
the service classes so that new classes can hook into the stray services
in the case that ceph services and cephadm services have differing names
or use subsystems that call into ceph with different names (my use
case).

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Tell cephadm that any `ctdb` services are "owned" by the smb service
and should be ignored as not a stray.
Ideally, we do this on a per service basis but the info that the ctdb
lock helper provides to its registration function is pretty generic.
Future versions of samba may improve upon this.

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

…lustering

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

ceph/ceph@2946b19
incorrectly removed this line and since then these certs are
not being properly mounted into the container. This commit
adds the line back

Signed-off-by: Adam King <adking@redhat.com>

…undary-keep-requests

mgr/rest: Trim  requests array and limit size

test/rgw/notifications: don't check for full queue if topics expired

Reviewed-By: Casey Bodley <cbodley@ibm.com>

mgr/cephadm: change SPDK RPC fields in nvmeof configuration

Fixes: https://tracker.ceph.com/issues/67549

Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

…cache-tiering

doc/rados: document unfound object cache-tiering scenario

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

The journal reset effectively cleared the cache so the rank may not have the
dirfrag in memory when we verify alternate name recovery.

Fixes: https://tracker.ceph.com/issues/67511
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

SchedEntry holds the scheduling details for scrubbing a specific PG at
a specific scrub level. Namely - it identifies the [pg,level]
combination, the 'urgency' attribute of the scheduled scrub
(which determines most of its behavior and scheduling decisions)
and the actual time attributes for scheduling (target,
deadline, not_before).

Added a table detailing, for each type of scrub, what limitations apply
to it, and what restrictions are waived.

The following commits will reshape the ScrubJob objects to hold
two instances of SchedTarget-s - two wrappers around SchedEntry-s,
one for the next shallow scrub and one for the next deep scrub.

Sched-entries (wrapped in sched-targets) have a defined order:

For ready-to-scrub entries (those that have an n.b. in the past),
the order is first by urgency, then by target time (and then by
level - deep before shallow - and then by the n.b. itself).

'Future' entries are ordered by n.b., then urgency,
target time, and level.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

…sserts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

This API provides the mechanism to update the existing client
ids for a particular OIDC Provider identified by its arn.

fixes: https://tracker.ceph.com/issues/63213

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

This API adds a mechanism to update the thumbprint list
of an oidc provider identified by its arn. The existing
thumbprint list is completely erased and then the new
list is added.

fixes: https://tracker.ceph.com/issues/63214

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

user to run oidc provider iam tests in teuthology.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

validation of a web token signature.

A boolean flag is incorrectly set to true outside the 'if'
statement, which causes an incorrect cert to be picked.

fixes: https://tracker.ceph.com/issues/54562

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

…clusters in Manage-clusters page

Fixes: https://tracker.ceph.com/issues/67192

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

* refs/pull/59166/head:
	mon/thrasher: set stopping

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

* refs/pull/56816/head:
	doc: mention the peer status failed when snapshot created on the remote filesystem.
	qa: add test_cephfs_mirror_remote_snap_corrupt_fails_synced_snapshot
	cephfs_mirror: update peer status for invalid metadata in remote snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

* refs/pull/55144/head:
	client: fix file cache cap leak which can stall async read call
	test/client: test contiguous read for a non-contiguous write

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

This commit extends NativeFormat (aka migration where the migration
source is an RBD image) to support external Ceph clusters, limited to
import-only mode.

Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

"optional if image in trash" can be easily interpreted as "required if
image not in trash".

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

…ve_format()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

fmt: bump up version + related changes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

ceph-volume: add "packaging" to install_requires

mgr/cephadm: add "original_weight" parameter to OSD class

Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw: load copy source bucket attrs in putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

To work around a potential linking issue between
nvme-cli ad libnvme that prevents nvme-cli from
correctly generating a hostnqn, causing

nvme_fabrics: found same hostid edb4e426-766f-44c6-b127-da2a5b7446ef but different hostnqn hostnqn

messages in dmesg and the inability to setup nvme
loop devices

Fixes: https://tracker.ceph.com/issues/67684

Signed-off-by: Adam King <adking@redhat.com>

doc/releases: Correct mimic.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

laddr_t will no longer be a plain uint64_t but a struct with a
customized formatter in the future, so these hexadecimal log outputs
should be removed at first.

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

librbd/migration: add external clusters support

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

…ts_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Dirty extent version is supposed to be non-zero.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

* refs/pull/56602/head:
	mds: always make getattr wait for xlock to be released by the previous client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

During teuthology tests, the tearing down of the cluster between two
tests causes the config to be reset and a config_notify generated. This
leads to a race to create a new mount using the old fscid. But by the
time the mount is attempted the new fs gets created with a new fscid.
This situation leads to the client mount waiting for a connection
completion notification from the mds for 5 minutes (default timeout)
and eventually giving up.
However, the default teuthology command timeout is 2 minutes. So,
teuthology fails the command and declares the job as failed way before
the mount can timeout.

The resolution to this case is to lower the client mount timeout to 30
seconds so that the config_notify fails fast paving the way for
successive commands to get executed with the new fs.

An unhandled cluster warning about an unresponsive client also gets
emitted later during qa job termination which leads to teuthology
declaring the job as failed. As of now this warning seems harmless since
it is emitted during cluster cleanup phase.
So, this warning is added to the log-ignorelist section in the
snap-schedule YAML.

Fixes: https://tracker.ceph.com/issues/66009
Signed-off-by: Milind Changire <mchangir@redhat.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

laddr_t is no longer POD type after changing it to struct, use
laddr_le_t instead

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Use laddr_offset_t as parameter for ValueDeltaRecorder, since the
payload offset may not be block aligned.

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Add a (very basic) definition of object storage.

Signed-off-by: Zac Dover <zac.dover@proton.me>

…ng_for_ceph_orch_daemon_add_osd

mgr/orchestrator: fix encrypted flag handling in orch daemon add osd

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

smb: ctdb clustering

Reviewed-by: Adam King <adking@redhat.com>

…bject-storage

doc/glossary: add "object storage"

Compressor: add UADK support

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

* refs/pull/58487/head:
	qa/suites/fs/workload: drop mgrmodules stanza
	qa/tasks/ceph: fix "ceph mgr module enable" command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

The cephadm_from_container allows one to do a single container build
and then point teuthology at that image as the "single source of truth".
I find this extremely convenient when running teuthology locally and
I keep carrying this patch around - I figure having it upstream will
simplify my workflow. Maybe someday it'll benefit others too.

To use it I set up a yaml overrides file with the following content:
```yaml
overrides:
  cephadm:
    image: "quay.io/phlogistonjohn/ceph:dev"
    cephadm_from_container: true
  verify_ceph_hash: false
verify_ceph_hash: false
```

This let's me test my custom builds fairly easily!

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

It's expected that the buffer is filled with the same byte, but the
byte should differ from run to run:

    memset(bp.c_str(), rand() & 0xff, io_size);

This was broken in commit c7f71d1 ("rbd: migrated existing command
logic to new namespaces") which inadvertently moved the call to srand(),
leaving rand() unseeded for the above memset().

Fixes: https://tracker.ceph.com/issues/67698
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

The variable was added in commit 9b6b7c3 ("Handle
differently-named xmlstarlet binary for *suse") but this
compatibility business is long outdated:

  Mon Oct 13 08:52:37 UTC 2014 - toms@opensuse.org

  - SPEC file changes
    - Added link from /usr/bin/xml to /usr/bin/xmlstarlet as other
      distributions do the same
    - Did the same for the manpage

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Add beacon_lock to mitigate potential beacon delays caused by slow message
handling, particularly in handle_nvmeof_gw_map.

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

ScrubJob will now hold two SchedTarget-s - two sets of scheduling
information (times, levels, etc.) for the next shallow and deep scrubs.

This is in preparation for the upcoming changes to the scheduling queue.
The change cannot stand on its own, as the partial implementation
creates some inconsistencies in the scheduling logic.

Specifically, here is what changes here, and how it differs from the
desired implementation:
- The OSD still maintains a queue of scrub jobs - one object only per
  PG.
  But now - each queue element holds two SchedTarget-s.
- When a scrub is initiated, the Scrubber is handed a ScrubJob object.
  Only in the next commit will it also receive the ID of the selected
  level. That causes some issues when re-determining the level of the
  initiated scrub. A failure to match the queue "intent" results in
  failures.
- the 'planned scrub' flags are still here, instead of directly
  encoding the characteristics of the next scrub in the relevant
  sched-entry.
- the 'urgency' levels do not cover the full required range of
  behaviors and priorities.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Signed-off-by: Samuel Just <sjust@redhat.com>

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

for readability

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

The OSD's scrub queue now holds SchedEntry-s, instead of ScrubJob-s.
The queue itself is implemented using the 'not_before_queue_t' class.

Note: this is not a stable state of the scrubber code. In the next
commits:
- modifying the way sched targets are modified and updated, to match the
  new queue implementation.
- removing the 'planned scrub' flags.

Important note: the interaction of initiate_scrub() and pop_ready_pg()
is not changed by this commit. Namely:

Currently - pop..() loops over all eligible jobs, until it finds one
that matches the environment restrictions (which most of the time, as the
concurrency limit is usually reached, would be 'high-priority-only').

The other option is to maintain Sam's 'not_before_q' clean interface: we
always pop the top, and if that top fails the preconds tests - we delay and
re-push. This has the following troubling implications:

- it would take a long time to find a viable scrub job, if the problem
  is related to, for example, 'no scrub'.
- local resources failure (inc_scrubs() failure) must be handles
  separately, as we do not want to reshuffle the queue for this
  very very common case.
- but the real problem: unneeded shuffling of the queue, even as the
  problem is not with the scrub job itself, but with the environment
  (esp. no-scrub etc.).
  This is a common case, and it would be wrong to reshuffle the queue
  for that.
- and - remember that any change to a sched-entry must be done under PG
  lock.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>