Feature request: add FlareSolverr solver to bypass CloudFlare protection

[soredake]

Ther are a lot of sites that use cloudflare as protection, checking sites that use cloudflare can result in 403, would be nice to have support for FlareSolverr to fix this situation.

Related:
#1157
jobobby04/TachiyomiSY#1124
Suwayomi/Suwayomi-VaadinUI@712ca2a

[mre]

First time I hear of FlareSolverr, but it looks nice. We wanted to add proxy support anyway at some point. There's a proxy config file that browsers like Firefox use. Not sure if it could also be used to call FlareSolverr.

[mre]

The proxy support request I was referring to: #869
Would it "just work" if we added support for that?

[mre]

Looks like FlareSolverr requires data to be sent as a JSON POST request? So the proxy approach wouldn't work.
There's a prototype for WebAssembly support, which could be used to modify the request. @thomas-zahner, this should work, right?

[thomas-zahner]

Yes this seems like a perfect use case of the recently implemented RequestChain. The WebAssembly part is indeed only a prototype and not merged yet. However, the request chain itself is already fully functional and available in lychee_lib through the ClientBuilder.

An example on how to make use of the RequestChain can be found here: https://github.com/lycheeverse/lychee/blob/master/examples/chain/chain.rs

[soredake]

Another example Suwayomi/Suwayomi-VaadinUI@712ca2a

[thomas-zahner]

@soredake @mre So I've created an example on how to use the RequestChain to call the FlareSolverr proxy instead of the actual website and then use the status code returned inside FlareSolverr's response body. Here you can find the code: 141b537...thomas-zahner:lychee:flaresolverr

Note that I didn't test it on a protected website and there is no error handling (but many unwraps). But as long as the status code returned by FlareSolverr is accurate it should work as expected. I felt that FlareSolverr's documentation is not very good as I didn't see the "official" JSON response schema, so I've used their provided "example response" which hopefully should represent the consistent JSON response schema.

Maybe it would make sense at some point to create a new repository in the lychee project which contains a list of response chain handlers that people might find helpful.

[thomas-zahner]

Noticed a bug in my implementation, currently Status:Ok is always returned. Fix would be a little hacky (copy paste) so I've opened up an issue