Update JSON::safeEncode to sanitize, rather than attempting convert/repair

[anthonyryan1]

Right now, we've got a lot of code attempting to correct every malformed UTF8 string into something usable. This pull request is more of an alternate approach where we don't try and coax every string into something valid, but instead replace invalid sequences with a broken character encoding character.

Also ensures that on JSON encode failures we don't fail silently and return a HTTP 200.

One of two possible fixes for #2977

The other approcah is collecting every broken UTF8 name and adding unit tests for the UTF8 repair class. I have saved some strings that I've observed causing these problems in the wild, and can share them with someone motivated to write tests and fix the UTF8 fixer instead.

[anthonyryan1]

Just to clarify, this problem affects every endpoint that uses safeEncode, not just the list in this example.

[ranirahn]

If all calls to safeEncode are affected why not fix the main function in
Rutorrent/php/utility/json.php ?

[anthonyryan1]

That is the long term plan, yes. This isn't ready to merge, it's a discussion starting point for how we want to try solving these UTF8 normalization failures.

[ranirahn]

What do you think if the json class is like this?

class JSON
{
    public static function safeEncode($value)
    {
        if (defined('JSON_THROW_ON_ERROR')) {
           try {
               return json_encode($value, JSON_THROW_ON_ERROR | JSON_INVALID_UTF8_SUBSTITUTE);
           } catch (JsonException $e) {
               throw $e;
           }
        }
        $encoded = json_encode($value);
        if ($encoded === false) {
            require_once('utf.php');
            return json_encode(UTF::utf8ize($value));
        }
        return $encoded;
    }
}

I would leave the old version just incase someone have PHP 5 still in use and then these substitude and stuff dont work. So far dont see anything break with this.

[anthonyryan1]

There hasn't been much discussion on the trade-offs between attempting to repair vs sanitizing the corrupt strings.

So I've just made a call, and amended this PR to my personal preference for how to handle this issue.

---

UTF::utf8ize previously attempted to "repair" as many mangled strings
as possible, but I've observed a number of examples over the years where
it choked and broke rutorrent.

I would argue that taking arbitrary strings of bytes, and attempting
to convert them with perfect accuracy into valid UTF8 (when those strings
come to us mangled, corrupt and always with unknown encodings) isn't possible.

Here's a handful of examples of of byte sequences that choke the UTF8 repair code:

require('php/utility/json.php');
var_dump(JSON::safeEncode("\xC0\x80"));
var_dump(JSON::safeEncode("\xED\xA0\x80"));
var_dump(JSON::safeEncode("\xED\xBF\xBF"));
var_dump(JSON::safeEncode("\xF5\x80\x80\x80"));
var_dump(JSON::safeEncode("\xF7\xBF\xBF\xBF"));

The problem is that I think this list is nowhere near comprehensive, and
even if we fix all of these, there will be many more corrupt sequences in
the wild we can't predict.

Instead of trying to repair all this corrupt data, let's just use the xFFFD
replacement character. Users who add files with corrupt strings will see the
replacement character (adding some awareness), but that becomes a content
problem and not a "ruTorrent is broken" problem.

[ranirahn]

I think it breaks everyones rutorrent that still runs on PHP 5. Also why delete code in utf8 file if its not in use anymore? What I propose was that it try-s without utf8 file and if keys exist that means its newer PHP and uses that and if not then it has fall back on old system. Just to cut it off seems little too big change. Cant predict how many people coming then that rutorrent dont work at all. Maybe someone with more knowledge can tell what will happen with all the people who still use PHP5 and older. I have no idea, but i am sure its not nothing.

[anthonyryan1]

You are correct that this won't work with PHP 5, it'll only work with PHP 7.2 or higher (when JSON_INVALID_UTF8_SUBSTITUTE was added).

PHP 7.2 was released in Nov 2017. The last version of PHP that wouldn't work with this change is 7.1 which hit it's security end of life on Dec 2019.

I think it's probably safe to say that anyone who hasn't installed updates in ~9 years and is running very unsafe PHP versions, probably also isn't updating ruTorrent. But if @xirvik's thinks we should preserve PHP 5 compatibility, I can rework this into feature detection.

[xirvik]

I think it breaks everyones rutorrent that still runs on PHP 5. Also why delete code in utf8 file if its not in use

People running PHP 5 need to update. There's no way we're going to keep supporting a PHP version that the PHP maintainers themselves declared end-of-life many years ago.

If you really really can't update PHP, then you have to stay on whatever version of ruTorrent still happens to work, which will also support an old version of rtorrent, etc.

[ranirahn]

Probably just need to add a warning somewhere that if you run PHP7.1 or older rutorrent does not run. Same with if its older rtorrent. Then i think its fine to clean this up. Then rutorrent tells you what is wrong and how to fix it. Right now its just not work and there is no info about it on rutorrent itself. Need to come and find that on internet. in content.js there is check for rtorrent version. If there would be also PHP version check then that would work. Example this

if(theWebUI.systemInfo.rTorrent.iVersion < 0x907) {
		const title = theUILang.requiresAtLeastRtorrent.replace('{version}', 'v0.9.7');
		$($$('webui.show_open_status')).attr({ disabled: '', title });
		$($$('lbl_webui.show_open_status')).attr({ title }).addClass('disabled');
	}

rutorrent should be also disabled if it does not have new enough PHP. Not sure how to do that. Same problem was with one plugin. File was removed but plugin still required it. This time its not plugin, but dont know where this file was required more. May not be only place where this utf8 was required.