How to make readonly structs XML serializable?

I have an immutable struct with only one field:

struct MyStruct
{
    private readonly double number;

    public MyStruct(double number)
        => this.number = number;
}

And I want this to be able to get serialized/deserialized by:

• Data contract serializer
• Binary formatter
• XML serializer (edit: forgotten in the original question)
• Json.NET (without adding Json.NET as a dependency)

So the struct becomes this:

[Serializable]
struct MyStruct : ISerializable, IXmlSerializable
{
    private readonly double number;

    public MyStruct(double number)
        => this.number = number;

    private MyStruct(SerializationInfo info, StreamingContext context)
        => this.number = info.GetDouble(nameof(this.number));

    void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
        => info.AddValue(nameof(this.number), this.number);

    XmlSchema IXmlSerializable.GetSchema() => null;

    void IXmlSerializable.ReadXml(XmlReader reader)
    {
        // Necessary evil
        reader.Read();
        this = new MyStruct(double.Parse(reader.Value, CultureInfo.InvariantCulture));
    }

    void IXmlSerializable.WriteXml(XmlWriter writer)
        => writer.WriteString(this.number.ToString(CultureInfo.InvariantCulture));
}

Because:

• [Serializable] is required by the binary formatter.
• Json.NET honors both [DataContract] and ISerializable.
• [DataContract] and ISerializable can't be used together.
• Luckily, IXmlSerializer is supported by the data contract serializer.

C# 7.2 introduces the readonly modifier for structs and MyStruct, being an immutable struct seems like an ideal candidate for this.

The problem is that IXmlSerializable interface requires the ability to mutate MyStruct. That's what we did above, assigning to this in IXmlSerializable.ReadXml implementation.

readonly struct MyStruct : IXmlSerializable
{
    // ...
    void IXmlSerializable.ReadXml(XmlReader reader)
    {
        // No longer works since "this" is now readonly.
        reader.Read();
        this = new MyStruct(double.Parse(reader.Value, CultureInfo.InvariantCulture));
    }
    // ...
}

I tried cheating via reflection but FieldInfo.SetValue boxes the value, and FieldInfo.SetValueDirect requires a TypedReference, which I can't obtain since __makeref is also forbidden when this is read-only.

So what are ways that would allow MyStruct to get serialized by the XML serializer?

I should also mention that I do not care what the output XML looks like, I don't really need the fine grained control provided by the IXmlSerializable interface. I only need to make MyClass consistently serializable using the serializers I listed.