Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modules/zstd: Rework ZSTD Decoder #1654

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft

Conversation

lpawelcz
Copy link
Contributor

@lpawelcz lpawelcz commented Oct 9, 2024

This PR reworks the existing ZSTD Decoder to memory-based approach instead of relying on stream-based architecture.

The PR relies on peripherals (MemReader and MemWriter) for accessing external memory through AXI interface that are introduced in: #1613. It is also based on #1616.

The Decoder is able to decompress ZSTD frames that consist of RAW and RLE blocks. This is tested in a verilog simulation with a cocotb testbench.

The changes in this PR include:

  • Rewriting ZstdDecoder proc to work with memory-based approach
  • Adding verilog-level tests of the ZstdDecoder
    • Introducing python library for generating ZSTD frames with decodecorpus
    • Adding python bindings for ZSTD reference library (zstandard) used for crosschecking decoding results in the simulation
    • Adding cocotb testbench that generates multiple ZSTD frames with RAW and RLE blocks and decodes those with simulated ZstdDecoder
  • Reworking FrameHeaderDecoder, BlockHeaderDecoder, Raw- and RleBlockDecoders to use memory-based approach instead of streaming input data from the input of the decoder
  • Adding CsrConfig and AxiCsrAccessor procs that implement the internal registers of the ZstdDecoder and provide external (AXI) and internal (native) facing interfaces for accessing those
  • Simplifying:
    • RleBlockDecoder proc
    • BlockHeader library
  • Removing components not used in memory-based approach:
    • magic number functions
    • BlockDecoder proc
    • DecDemux proc
  • Added common_codegen_args list to the BUILD file

This PR is currently a WIP. It is opened to showcase the work done and to create a space for discussion on certain topics. The following list enumerates task that are currently being addressed and which are required for this PR to be completed before it is marked as ready for review.

Next steps:

  • Update documentation - describe the new architecture of the decoder and how to interact with the IP from the software (CSRs)
  • Add DSLX tests for the ZstdDecoder proc - create facilities for preparing test data and write DSLX tests
  • Improve error handling - write error codes to the Status CSR
  • Improve verilog tests - reorganize testbench to handle decoding of multiple ZSTD frames in one test case
  • Change the output format from stream-based to memory-based (write decoded frame to memory)
  • Organize structs and new type aliases (cleanup)
  • Remove a workaround (85f1f80) for type check failure during IR generation for the ZstdDecoder proc (TypeCheck failure for a parameterized proc #1655)

CC @proppy

lpawelcz and others added 29 commits October 11, 2024 10:09
bazel run executes only the first target from the list of targets acquired
from the output of bazel query. In order to properly call all targets
it is required to loop through the targets and run one at a time

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Required to pass place_and_route

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Michal Czyz <mczyz@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
This comit adds implementation of AxiReader proc that can be used to
to issue AXI read requests as an AXI Manager device.

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
This commits adds AxiStreamRemoveEmpty proc, that can be used
to remove bytes marked as containing no data in the Axi Stream

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
This commit adds AxiStreamDownscaler that can be used to convert AxiStream
transactions from a wider bus, to multiple transactions on more narrow bus

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
This commit adds MemReader and MemReaderAdv procs for handling
read transactions on the AXI bus.

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Internal-tag: [#62924]

Co-authred-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authred-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Michal Czyz <mczyz@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Internal-tag: [#64376]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Internal-tag: [#65205]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
- XLSStruct for easier handling and serializing/deserializing XLS structs
- XLSChannel that serves as a dummy receiving channel
- XLSMonitor that monitors transactions on an XLS channel
- XLSDriver that can send data on an XLS channel
- LatencyScoreboard that can measure latency between corresponding transactions on input and output buses
- File-backed AXI memory python model

Internal-tag: [#64075]
Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authred-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authred-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Michal Czyz <mczyz@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
Remove references to buffer structs as those are not used anywhere

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Krzysztof Oblonczek <koblonczek@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
m-torhan and others added 6 commits October 11, 2024 10:12
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Torhan <mtorhan@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants