XML and Dash and SCTE-214

[futzu]

SCTE-214
Adrian of Doom and @davemevans and @wabiloo are currently implementing Dash and XML support to threefive, this is where we discuss it. Feel free to join us.

Adrian

[futzu]

@davemevans
I cleaned up Upids a little, upids now have a var called upid_value, which is the Upid.
I did the xml() method for the Upid class. Move your MPU and MID xml() stuff over to upid.py
and do separate pull request for just that.
then we'll figure out how we want to implement the from_xml() stuff.

• this is the current output from the Upid class. I added name, for clarity, we can add any attributes we want, and that one does make sense to use.
• I stuck with your hexbinary idea
• base16 is an integer and unsigned byte

    <SegmentationUpid name="AiringID" segmentationUpidType="0x8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8"/>

[futzu]

@davemevans I know exactly what you need to do, make a __ threefive.xsd__ , that handles the ##any properly and get rid of the component and other old stuff and we can post on one of my servers and include in the releases. I'll try to push it on SCTE and everyone else. That's would be __HUGE. We need a threefive schema, that can be validated against.
Make sure your name is in it as the author too. Maybe give us a namespace or something. Whatever you think best.
Hell Yes. Can you do that?

[futzu]

Even if we only use it to validate threefive output, that would save us so much time. Just like you said , people are going to want to validate, let's give them a way to do it.

[futzu]

@davemevans
On the MID upid thing, we could call all the UPIDs in xml with the segmentation descriptor node as an arg and attach child nodes directly to it. All the upiids have to attach to it anyway.

  sd=Node('SegmentationDescriptor',attrs=sd_attrs)
        the_upid = self.mk_the_upid()
        upid_node = the_upid.xml()

that's how I do it right now, but we could do it like

  sd=Node('SegmentationDescriptor',attrs=sd_attrs)
        the_upid = self.mk_the_upid()
        sd = the_upid.xml(sd) <---- just pass the sd Node in and attach child nodes directly to it and return  the node.

[davemevans]

Cool cool - will have some faith, let you work, and come back in a bit when things are more stable.

My use case is to take an arbitrary binary cue, convert it to XML representation, then run an arbitrary xpath evaluation against that eg /SpliceInfoSection/SegmentationDescriptor[@segmentationTypeId=48], //SegmentationUpid[@segmentationUpidType=8 and .='0000000012345678'] and such.

[davemevans]

@davemevans On the MID upid thing, we could call all the UPIDs in xml with the segmentation descriptor node as an arg and attach child nodes directly to it. All the upiids have to attach to it anyway.

  sd=Node('SegmentationDescriptor',attrs=sd_attrs)
        the_upid = self.mk_the_upid()
        upid_node = the_upid.xml()

that's how I do it right now, but we could do it like

  sd=Node('SegmentationDescriptor',attrs=sd_attrs)
        the_upid = self.mk_the_upid()
        sd = the_upid.xml(sd) <---- just pass the sd Node in and attach child nodes directly to it and return  the node.

Yes, that approach makes sense - tell the child its parent and have it attach itself.

[futzu]

class Upid:
    """
    Upid base class handles URI UPIDS
    """

    def __init__(self, bitbin=None, upid_type=0, upid_length=0):
        self.bitbin = bitbin
        self.upid_type = upid_type
        self.upid_name = upid_map[upid_type][0]
        self.upid_length = upid_length
        self.bit_length = upid_length << 3
        self.upid_value = None

    def decode(self):
        """
        decode Upid
        """
        self.upid_value = self.bitbin.as_charset(self.bit_length, charset)
        return self.upid_name, self.upid_value

    def encode(self, nbin ):
        """
        encode Upid
        """
        if self.upid.value:
            self.upid.value = self.upid.value.encode("utf8")
            nbin.add_bites(self.upid.value)

    def xml(self):
        """
        xml return a upid xml node
        """
        ud_attrs = { 'name': self.upid_name, # this is for clarity
                    'segmentation_upid_type': hex(self.upid_type),
                    'segmentation_upid_format':'hexbinary',
                    'segmentation_upid_length':self.upid_length,}
        return  Node('SegmentationUpid',attrs= ud_attrs, value=self.upid_value)

[davemevans]

What I didn't realise at the time I said that is that these things are constructed every time they are needed rather than hanging around. So it probably didn't actually really matter.

This could help but as it is now, it is possibly worse - it's not really clear to me how this can work when it is called in SegmentationDescriptor.xml() since there is no bitbin from which to decode any value. So you have to explicitly set the_upid.upid_value before calling the_upid.xml() which feels a bit janky.

Also, self.upid.value appears to be a typo.

For MPU support, there will need to be a way to pass in at least one more attribute (format_identifier) which I don't see how to do now, but perhaps I miss something. Would also be nice to be able to set the upid format to text for some types.

[futzu]

You're over thinking it .

• It's constructed every time by design, I didn't want to worry about changed values.
• we don't pass in format_identifier, it already in the upid after decoding
• the is the Upid instance, and a Upid xml Node. All the data is in the instance, it is copied to thee xml node.
  You don't need bitbin, it's used by the instance, not the xml node.
  this is just for the xml() method call, when you writing xml, the cue has already been decoded.
  Here is an MPU upid decoded, format_identifier is already there.

  "segmentation_upid": {
                "format_identifier": "RTLN",
                "private_data": "0x3148010000000031333736393230323534343935483100"
            },

Here's a mid decoded

  "segmentation_upid_type_name": "MID",
            "segmentation_upid_length": 17,
            "segmentation_upid": [
                {
                    "upid_type": 14,
                    "upid_type_name": "ADS Info",
                    "upid_length": 5,
                    "segmentation_upid": "LA309"
                },
                {
                    "upid_type": 8,
                    "upid_type_name": "AiringID",
                    "upid_length": 8,
                    "segmentation_upid": "0x2e538481"
                }
            ],
            "segmentation_type_id": 52,

[davemevans]

In the segmentationDescriptor.xml example above, the value isn't in the Upid instance because it isn't set when you mk_the_upid, and no decode is called (as you say it's already decoded). In this case, self.upid_value in Upid instance will be None unless it's set explicitly.

[futzu]

Scratch that last part. It's better to have one special instance that to change them all.
Have the MID return a list of Nodes, one for each UPID,
This is a better way

# Segmentation Descriptor xml has this in it 

    def xml(self):
        """
        Create a Node describing a SegmentationDescriptor
        """
        .
        .
        .
        
        sd=Node('SegmentationDescriptor',attrs=sd_attrs)
        the_upid = self.mk_the_upid()
        upid_node = the_upid.xml()
        if isinstance(upid_node,list): #<-- check if it's a list 
            _=(sd.add_child(node) for node in upid_node)      #<----- add each mid upid as a child of sd 


This is the MID class xml()

 def xml(self):
        """
        xml return a upid xml node

        """
        mid_nodes =[]
        for u in self.upid_value:
            u_attrs = ["upid_type": u['upid_type'],
                       "name":  u['upid_type_name'],}
            value= u["segmentation_upid"]
            node= Node('SegmentationUpid', attrs=u_attrs, value=value)
            mid_nodes.append(node)
        return mid_nodes

[futzu]

Sorry, I'm doing three things at once.

don't call mk_the_upid,, you're right. I'll sort this out, but I gotta go for a little bit,
Make us a schema and I'll sort this out tonight after work

[futzu]

This is easy, they all have the same vars

• in segmentationDescriptor.xml()

    def xml(self):
        """
        Create a Node describing a SegmentationDescriptor
        """    

          ....
        
        self.segmentation_upid_type = bitbin.as_int(8)
        self.segmentation_upid_length = bitbin.as_int(8)
        the_upid = self.mk_the_upid(bitbin=None)
        # populate the upid's values
        the_upid.upid_name= self.segmentation_upid_type_name   # mk_the_upid passes segmentation_upid_type  and length in  already
        the_upid.upid_value= self.segmentation_upid
        # call the_upid.xml()
        upid_node = the_upid.xml()
        if isinstance(upid_node,list):
            _=(sd.add_child(node) for node in upid_node)
       else:
             sd.add_child(upid_node)
         ......

• This is the MID class xml()

 def xml(self):
        """
        xml return a upid xml node

        """
        mid_nodes =[]
        for u in self.upid_value:
            u_attrs = ["upid_type": u['upid_type'],
                       "name":  u['upid_type_name'],}
            value= u["segmentation_upid"]
            node= Node('SegmentationUpid', attrs=u_attrs, value=value)
            mid_nodea.append(node)
        return mid_nodes

[futzu]

Does that make sense?

[futzu]

Dave,
does this look right?

<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="8" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="3522714355"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationEventId="1207959743" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0" segmentationDuration="16317027">
      <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
      <SegmentationUpid upidType="14" name="ADS Info">LA309</SegmentationUpid>
      <SegmentationUpid upidType="8" name="AiringID">0x2e538481</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

[futzu]

There is super easy way SCTE could fix this, is to make SegmentationDescriptor.segmentation_upid, segmentation_upids, a list.

[futzu]

Dave Evans needs validation

[futzu]

okay. We'll change it back

[futzu]

done.

[futzu]

@davemevans
We need the segmentation message on the descriptor. People need to be able to visually inspect a manifest, to know when the break starts.I know this from HLS. Without the segmentation, you need to memorize like thirty different codes.

We could just make comment nodes, couldn't we?
For the segmentation descriptor segmentation message and upid name ?

Also , you need to tell me what you're trying to accomplish so I can keep it mind.

[futzu]

I can do it like this

<!-- Created with Unified Streaming Platform  (version=1.14.2-30503) -->

I'll just put the stuff that isn't in the xsd in comment nodes. Easy fix.

, that works just fine.

Any problem with comment nodes?

[davemevans]

Yeah, this is a great shout 👌

[futzu]

Most of the UPiDa are now working.

Upid Xml output

0x3 : ABCD0123456H
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
      <SegmentationUpid name="AdID" segmentationUpidType="3" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">ABCD0123456H</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0x6 : 0x3a8d000000000000
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="6" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
      <SegmentationUpid name="ISAN" segmentationUpidType="6" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">0x3a8d000000000000</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0x7 : MV0004146400
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
      <SegmentationUpid name="TID" segmentationUpidType="7" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">MV0004146400</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0xe : ADS-UPID:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="11" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
      <SegmentationUpid name="ADS Info" segmentationUpidType="14" segmentationUpidFormat="hexbinary" segmentationUpidLength="45">ADS-UPID:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0x4 : 060a2b34.01010105.01010d20.13000000.d2c9036c.8f195343.ab7014d2
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="None" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="215" segmentNum="None" segmentsExpected="None">
      <SegmentationUpid name="UMID" segmentationUpidType="4" segmentationUpidFormat="hexbinary" segmentationUpidLength="32">060a2b34.01010105.01010d20.13000000.d2c9036c.8f195343.ab7014d2</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0xd : [{'upid_type': 14, 'upid_type_name': 'ADS Info', 'upid_length': 5, 'segmentation_upid': 'LA309'}, {'upid_type': 8, 'upid_type_name': 'AiringID', 'upid_length': 8, 'segmentation_upid': '0x2e538481'}]
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="8" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="3522714355"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity Start" segmentationEventId="1207959743" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0" segmentationDuration="16317027">
      <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
      <SegmentationUpid upidType="14" upidTypeName="ADS Info">LA309</SegmentationUpid>
      <SegmentationUpid upidType="8" upidTypeName="AiringID">0x2e538481</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0x8 : 0x072d08c7
0xc : {'format_identifier': 'RTLN', 'private_data': '0x3148010000000031333736393230323534343935483100'}
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="5469869394"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Provider Advertisement End" segmentationEventId="1859042901" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="49" segmentNum="11" segmentsExpected="25">
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x072d08c7</SegmentationUpid>
   </SegmentationDescriptor>
   <SegmentationDescriptor segmentationMessage="Distributor Placement Opportunity End" segmentationEventId="1858902152" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="55" segmentNum="1" segmentsExpected="1">
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x011836b21611</SegmentationUpid>
   </SegmentationDescriptor>
   <SegmentationDescriptor segmentationMessage="Provider Advertisement Start" segmentationEventId="1859063048" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="48" segmentNum="12" segmentsExpected="25" subSegmentNum="2" subSegmentsExpected="42" segmentationDuration="1821600">
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x072d5612</SegmentationUpid>
   </SegmentationDescriptor>
   <SegmentationDescriptor segmentationMessage="Content Identification" segmentationEventId="1859063094" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="1" segmentNum="0" segmentsExpected="0">
      <SegmentationUpid name="MPU" segmentationUpidType="12" segmentationUpidFormat="hexbinary" segmentationUpidLength="27">{'format_identifier': 'RTLN', 'private_data': '0x3148010000000031333736393230323534343935483100'}</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0xb : {'TSID': 241, 'reserved': 3, 'end_of_day': 23, 'unique_for': 511, 'content_id': 'human012'}
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
      <SegmentationUpid name="ATSC" segmentationUpidType="11" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">{'TSID': 241, 'reserved': 3, 'end_of_day': 23, 'unique_for': 511, 'content_id': 'human012'}</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="2832024813"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity End" segmentationEventId="1207959725" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="53" segmentNum="2" segmentsExpected="0">
      <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d79d</SegmentationUpid>
   </SegmentationDescriptor>
   <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="1207959590" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
      <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d79d</SegmentationUpid>
   </SegmentationDescriptor>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="1207959591" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0">
      <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
      <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d7b3</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0x9 : SIGNAL:3-sQ4NgFT0OjPsG4WqUQow
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="5400000"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity End" segmentationEventId="1644168586" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="53" segmentNum="0" segmentsExpected="0">
      <SegmentationUpid name="ADI" segmentationUpidType="9" segmentationUpidFormat="hexbinary" segmentationUpidLength="29">SIGNAL:3-sQ4NgFT0OjPsG4WqUQow</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0xa : 0x1478f85ae100b0685b8fb1c8
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
      <SegmentationUpid name="EIDR" segmentationUpidType="10" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">0x1478f85ae100b0685b8fb1c8</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

0xf : urn:uuid:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
   <TimeSignal>
      <SpliceTime ptsTime="0"/>
   </TimeSignal>
   <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="10" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2700000">
      <SegmentationUpid name="URI" segmentationUpidType="15" segmentationUpidFormat="hexbinary" segmentationUpidLength="45">urn:uuid:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

[futzu]

@wabiloo A lot of the UPIDS are working, upids that are dictionaries don't work yet when creating xml,
everything should be working.

[futzu]

@davemevans we still need to do the other Splice Commands, Splice Null, Private, and BandwidthReservatiuon.
You want to do the xml parts?

[davemevans]

Yeah, happy to take a look at this over the next few days.

[davemevans]

Looking at PrivateCommand, decode doesn't currently read the private_bytes so I'm working on that, but there's nothing within the command itself to say how many private_bytes there are.

I note that the other command calculate their own length and set the splice_command_length based on what they read, rather than pass in the signalled splice_command_length. I assume this is to deal with the legacy length of 0xFFF.

I'm a bit stuck working out what to do here as I need a length to know when to stop consuming private_bytes, but clearly can't always trust the splice_command_length. Got any ideas?

[futzu]

@davemevans First, lets talk comment nodes.

I tried doing Comment as a subclass last too, my problem with it is the init,
if we subclass then a comment node can have a value , children, an attributes,
and that 's a problem for me .

However, when I saw how you did it in Node.mk() and checking the instance, that works,
so I don't think it should be a subclass.

you have mk() already, you need an init , repr, and set depth.

    def __init__(self,comment):
        self.comment = comment
        self.depth= None

    def __repr__(self):
        return self.mk()

    def set_depth(self):
        """
        set_depth is used to format
        tabs in output
        """
        if not self.depth:
            self.depth = 0

I would MUCH rather have it as a separate class, I absolutely agree with you there,
make it it's own class,

[futzu]

@davemevans the Command length is calculated and then set in the Info Section because a lot o people put bad values in the info section, and this way if something changes, when I re-encode, values are verified.

The reason it's not done in private, probably because nobody ever uses it.

the Private Command doesn't use bitn, it's just raw bytes,
Since it calls super().init it has the command_length var.
You can add it to deccode, just not encode.

    def decode(self):  
        """
        PrivateCommand.decode method
        """
        self.identifier = int.from_bytes(
            self.bites[0:3], byteorder="big"
        )  # 3 bytes = 24 bits
        self.command_length= len(self.bites)  # For Command length it goes here  
        self.bites = self.bites[3:]   

       

[futzu]

Bacl in the Node.mk()
take out the else, you don't need it.

   if isinstance(obj, Comment):
            return obj.mk(obj)
    new_attrs = mk_xml_attrs(obj.attrs)
    rendrd = f"{ndent}<{obj.name}{new_attrs}>"
    if obj.value:
        return f"{rendrd}{obj.value}</{obj.name}>\n"
    rendrd = f"{rendrd}\n"
    for child in obj.children:
        rendrd += obj.mk(child)
    if obj.children:
        return f"{rendrd}{ndent}</{obj.name}>\n"
    return rendrd.replace(">", "/>")

[futzu]

@davemevans

Here's an updated private command class

  class PrivateCommand(SpliceCommand):
    """
    Table 13 - private_command
    """

    def __init__(self, bites=None):
        super().__init__(bites)
        self.command_type = 255
        self.name = "Private Command"
        self.identifier = None
        self.private_bytes = None

    def decode(self):
        """
        PrivateCommand.decode method
        """
        self.identifier = int.from_bytes(
            self.bites[0:3], byteorder="big"
        )  # 3 bytes = 24 bits
        self.command_length = len(self.bites)
        self.private_bytes = self.bites[3:]
        
  def encode(self, nbin=None):
        """
        encode private command
        """
        nbin = self._chk_nbin(nbin)
        self._chk_var(int, nbin.add_int, "identifier", 24)  # 3 bytes = 24 bits
        nbin.add_bites(self.private_bytes)
        return nbin.bites

[futzu]

anything threefive specific uses 'bites' with an i , early on I had some name collisions .

[davemevans]

Unfortunately this doesn't work because when decode is called self.bites contains the remainder of the cue unparsed up til that point ie it has the descriptor loop length, descriptors and crc.

[futzu]

I need cue.py, where are you at on the comment class?

[futzu]

I'm just going to accept the pull request and update he comment class.

[futzu]

@davemevans , going forward, if you see a problem bring it up and let's decide the plan before you do a pull request.

[futzu]

@davemevans
Hold up on any changes, I got to figure something out

    def xml(self, binary=False):
        """
        xml returns a threefive.Node instance
        which can be edited as needed or printed.
        """
        if binary:
            sig_attrs = {"xmlns": "https://scte.org/schemas/35"}
            sig_node = Node("Signal", attrs=sig_attrs)
            bin_node = Node("Binary", value=self.encode())
            sig_node.add_child(bin_node)
            return sig_node
        sis = self.info_section.xml()
        self.decode()                          <---  Having this here breaks  loading from json and converting to xml 

without that, the comments don't work.

[futzu]

I have no idea why.

[futzu]

I also added xml to the threefive cli tool.

threefive xml  '/DBMAAAAAsrbAP/wBQb+zTXUKAA2AhlDVUVJABWWDH+DCAgAAAAALfOq1zQAAAAAAhlDVUVJAAA3Qn+DCAgAAAAALfOq1zQAAAAAAhmCQA=='

it also works on streams

threefive xml video.ts

[futzu]

and json to xml

threefive '/DBMAAAAAsrbAP/wBQb+zTXUKAA2AhlDVUVJABWWDH+DCAgAAAAALfOq1zQAAAAAAhlDVUVJAAA3Qn+DCAgAAAAALfOq1zQAAAAAAhmCQA==' 2>&1 | ./threefive  encode xml
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="183003" protocolVersion="0" sapType="3" tier="4095">
   <TimeSignal>
      <SpliceTime ptsTime="3442857000"/>
   </TimeSignal>
   <!-- Provider Placement Opportunity Start -->
   <SegmentationDescriptor segmentationEventId="1414668" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0">
      <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="false" archiveAllowedFlag="false" deviceRestrictions="3"/>
      <!-- UPID: AiringID -->
      <SegmentationUpid segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">2df3aad7</SegmentationUpid>
   </SegmentationDescriptor>
   <!-- Provider Placement Opportunity Start -->
   <SegmentationDescriptor segmentationEventId="14146" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0">
      <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="false" archiveAllowedFlag="false" deviceRestrictions="3"/>
      <!-- UPID: AiringID -->
      <SegmentationUpid segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">2df3aad7</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

[futzu]

@wabiloo command line xml to Cue

xemel.txt

a@fu:~/build/SCTE35_threefive$ cat xemel.txt | threefive encode
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 76,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 2.033367,
        "cw_index": "0x0",
        "tier": "0xfff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 54,
        "crc": "0x2198240"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 38253.966667
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 25,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x15960c",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": false,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "No Restrictions",
            "segmentation_upid_type": 8,
            "segmentation_upid_length": 8,
            "segmentation_upid": "2df3aad7",
            "segmentation_type_id": 52,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        },
        {
            "tag": 2,
            "descriptor_length": 25,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x3742",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": false,
            "web_delivery_allowed_flag": false,
            "no_regional_blackout_flag": false,
            "archive_allowed_flag": false,
            "device_restrictions": "No Restrictions",
            "segmentation_upid_type": 8,
            "segmentation_upid_length": 8,
            "segmentation_upid": "2df3aad7",
            "segmentation_type_id": 52,
            "segment_num": 0,
            "segments_expected": 0,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        }
    ]
}

[futzu]

@davemevans We've added a little over 400 lines of code, xml.py is almost 300 of them. I it's a new ability, it should require code.
We haven't had to make many changes to the rest of threefive, it's mostly new code. That's a strong indicator that we're doing it well.

[futzu]

If it's a method is only called by it's own instance, prefix it with an underscore and that way it doesn't show up in the help.

class XmlParser(builtins.object)
 |  Methods defined here:
 |  
 |  __init__(self)
 |  
 |  chk_node_list(self, node)
 |      chk_node_list is used to track open xml nodes
 |  
 |  mk_active(self, node)
 |      mk_active sets self.active to the current node name.
 |  
 |  mk_attrs(self, node)
 |      mk_attrs parses the current node for attributes
 |      and stores them in self.stuff[self.active]
 |  
 |  mk_value(self, value, stuff)
 |      mk_value, if the xml node has a value, write it to self.stuff
 |      
 |      <name>value</name>
 |  
 |  parse(self, exemel, descriptor_parse=False)
 |      parse parses an xml string for a SCTE-35 Cue.
 |  
 |  ----------------------------------------------------------------------

The only that I should have there is parse, the rest I need to prefix with an understorre.
This is what I try to target with my comments.
I always have python shell open and I use help extensively,.

[futzu]

Current Cyclomatic Complexity Score

a@fu:~/build/SCTE35_threefive$ radon cc -sa threefive
threefive/packetdata.py
    C 8:0 PacketData - A (3)
    M 21:4 PacketData.mk_pcr - A (2)
    M 31:4 PacketData.mk_pts - A (2)
    M 15:4 PacketData.__init__ - A (1)
threefive/upids.py
    C 151:0 Eidr - A (3)
    C 201:0 Mid - A (3)
    M 242:4 Mid.xml - A (3)
    C 303:0 Umid - A (3)
    C 23:0 Upid - A (2)
    M 43:4 Upid.encode - A (2)
    C 63:0 NoUpid - A (2)
    C 91:0 AirId - A (2)
    C 121:0 Atsc - A (2)
    M 156:4 Eidr.decode - A (2)
    M 170:4 Eidr.encode - A (2)
    C 182:0 Isan - A (2)
    M 206:4 Mid.decode - A (2)
    M 230:4 Mid.encode - A (2)
    C 262:0 Mpu - A (2)
    M 280:4 Mpu.decode - A (2)
    M 308:4 Umid.decode - A (2)
    M 320:4 Umid.encode - A (2)
    M 28:4 Upid.__init__ - A (1)
    M 36:4 Upid.decode - A (1)
    M 51:4 Upid.xml - A (1)
    M 68:4 NoUpid.decode - A (1)
    M 74:4 NoUpid.encode - A (1)
    M 80:4 NoUpid.xml - A (1)
    M 96:4 AirId.decode - A (1)
    M 103:4 AirId.encode - A (1)
    M 109:4 AirId.xml - A (1)
    M 126:4 Atsc.decode - A (1)
    M 140:4 Atsc.encode - A (1)
    M 187:4 Isan.decode - A (1)
    M 194:4 Isan.encode - A (1)
    M 267:4 Mpu._decode_adfr - A (1)
    M 292:4 Mpu.encode - A (1)
threefive/decode.py
    F 27:0 _read_stuff - A (3)
    F 43:0 decode - A (3)
threefive/encode.py
    F 49:0 mk_splice_insert - A (3)
    F 24:0 mk_time_signal - A (2)
    F 12:0 mk_splice_null - A (1)
threefive/smoketest.py
    F 33:0 smoke - A (4)
    F 24:0 _decode_test - A (2)
threefive/stream.py
    M 544:4 Stream._parse_scte35 - B (8)
    M 568:4 Stream._parse_sdt - B (7)
    M 332:4 Stream.show - B (6)
    M 470:4 Stream._parse_tables - B (6)
    M 625:4 Stream._parse_pmt - B (6)
    M 220:4 Stream._find_start - A (5)
    M 265:4 Stream.decode_fu - A (5)
    M 278:4 Stream.decode_next - A (5)
    M 352:4 Stream.show_pts - A (5)
    M 497:4 Stream._parse - A (5)
    M 250:4 Stream.decode - A (4)
    M 317:4 Stream.proxy - A (4)
    M 431:4 Stream._parse_pts - A (4)
    M 604:4 Stream._parse_pat - A (4)
    C 39:0 ProgramInfo - A (3)
    M 53:4 ProgramInfo.show - A (3)
    C 116:0 Stream - A (3)
    M 405:4 Stream._unpad - A (3)
    M 421:4 Stream._parse_cc - A (3)
    M 514:4 Stream._same_as_last - A (3)
    M 677:4 Stream._set_scte35_pids - A (3)
    C 78:0 Pids - A (2)
    C 96:0 Maps - A (2)
    M 136:4 Stream.__init__ - A (2)
    M 214:4 Stream._split_by_idx - A (2)
    M 244:4 Stream._mk_pkts - A (2)
    M 309:4 Stream.decode_start_time - A (2)
    M 370:4 Stream.pid2prgm - A (2)
    M 380:4 Stream.pid2pts - A (2)
    M 390:4 Stream.pid2pcr - A (2)
    M 400:4 Stream._unpad_afc - A (2)
    M 449:4 Stream._parse_pcr - A (2)
    M 459:4 Stream._parse_payload - A (2)
    M 487:4 Stream._parse_info - A (2)
    M 509:4 Stream._chk_partial - A (2)
    M 521:4 Stream._section_incomplete - A (2)
    M 528:4 Stream._parse_cue - A (2)
    M 536:4 Stream._strip_scte35_pes - A (2)
    M 652:4 Stream._parse_program_streams - A (2)
    F 15:0 no_op - A (1)
    F 23:0 show_cue - A (1)
    F 31:0 show_cue_stderr - A (1)
    M 46:4 ProgramInfo.__init__ - A (1)
    M 86:4 Pids.__init__ - A (1)
    M 106:4 Maps.__init__ - A (1)
    M 160:4 Stream.__repr__ - A (1)
    M 164:4 Stream.as_90k - A (1)
    M 171:4 Stream._pusi_flag - A (1)
    M 175:4 Stream._afc_flag - A (1)
    M 179:4 Stream._pcr_flag - A (1)
    M 183:4 Stream._spi_flag - A (1)
    M 187:4 Stream._pts_flag - A (1)
    M 192:4 Stream._parse_length - A (1)
    M 199:4 Stream._parse_pid - A (1)
    M 207:4 Stream._parse_program - A (1)
    M 238:4 Stream.iter_pkts - A (1)
    M 292:4 Stream.decode_program - A (1)
    M 300:4 Stream.decode_pids - A (1)
    M 364:4 Stream.pts - A (1)
    M 414:4 Stream._mk_packet_data - A (1)
    M 667:4 Stream._parse_stream_type - A (1)
threefive/commands.py
    M 203:4 SpliceInsert.decode - A (5)
    M 229:4 SpliceInsert.encode - A (5)
    M 263:4 SpliceInsert.xml - A (5)
    C 172:0 SpliceInsert - A (4)
    M 143:4 TimeSignal._encode_splice_time - A (3)
    M 295:4 SpliceInsert.from_xml - A (3)
    C 10:0 SpliceCommand - A (2)
    C 41:0 BandwidthReservation - A (2)
    C 57:0 PrivateCommand - A (2)
    C 90:0 SpliceNull - A (2)
    C 101:0 TimeSignal - A (2)
    M 131:4 TimeSignal._splice_time - A (2)
    M 157:4 TimeSignal.xml - A (2)
    M 167:4 TimeSignal.from_xml - A (2)
    M 15:4 SpliceCommand.__init__ - A (1)
    M 21:4 SpliceCommand.decode - A (1)
    M 26:4 SpliceCommand._set_len - A (1)
    M 33:4 SpliceCommand.encode - A (1)
    M 46:4 BandwidthReservation.__init__ - A (1)
    M 51:4 BandwidthReservation.decode - A (1)
    M 62:4 PrivateCommand.__init__ - A (1)
    M 69:4 PrivateCommand.decode - A (1)
    M 80:4 PrivateCommand.encode - A (1)
    M 95:4 SpliceNull.__init__ - A (1)
    M 106:4 TimeSignal.__init__ - A (1)
    M 113:4 TimeSignal.decode - A (1)
    M 122:4 TimeSignal.encode - A (1)
    M 177:4 SpliceInsert.__init__ - A (1)
    M 194:4 SpliceInsert.decode_break_duration - A (1)
    M 254:4 SpliceInsert.encode_break_duration - A (1)
threefive/xml.py
    M 130:4 Node.mk - B (6)
    F 26:0 un_xml - A (5)
    M 217:4 XmlParser._mk_attrs - A (5)
    F 42:0 iter_attrs - A (4)
    F 17:0 un_camel - A (3)
    F 53:0 val2xml - A (3)
    C 82:0 Node - A (3)
    M 116:4 Node.set_depth - A (3)
    C 166:0 Comment - A (3)
    C 174:0 XmlParser - A (3)
    M 190:4 XmlParser._chk_node_list - A (3)
    M 239:4 XmlParser._parse_nodes - A (3)
    M 246:4 XmlParser._parse_most - A (3)
    F 64:0 key2xml - A (2)
    F 74:0 mk_xml_attrs - A (2)
    M 104:4 Node.__init__ - A (2)
    M 167:4 Comment.mk - A (2)
    M 199:4 XmlParser._mk_value - A (2)
    M 228:4 XmlParser.parse - A (2)
    M 264:4 XmlParser._parse_descriptor - A (2)
    F 8:0 t2s - A (1)
    M 113:4 Node.__repr__ - A (1)
    M 126:4 Node._get_indent - A (1)
    M 153:4 Node.add_child - A (1)
    M 159:4 Node.add_comment - A (1)
    M 186:4 XmlParser.__init__ - A (1)
    M 210:4 XmlParser._mk_active - A (1)
threefive/stuff.py
    F 8:0 print2 - A (1)
threefive/cue.py
    M 144:4 Cue._mk_bits - B (7)
    M 103:4 Cue.get - A (5)
    M 412:4 Cue.xml - A (5)
    M 319:4 Cue.load_descriptors - A (4)
    M 334:4 Cue.load - A (4)
    M 366:4 Cue._xml_splice_command - A (4)
    C 25:0 Cue - A (3)
    M 89:4 Cue._descriptor_loop - A (3)
    M 280:4 Cue._unloop_descriptors - A (3)
    M 306:4 Cue.load_command - A (3)
    M 374:4 Cue._xml_splice_descriptor - A (3)
    M 387:4 Cue._xml_event_signal - A (3)
    M 393:4 Cue.from_xml - A (3)
    M 55:4 Cue.__init__ - A (2)
    M 74:4 Cue.decode - A (2)
    M 121:4 Cue.get_descriptors - A (2)
    M 136:4 Cue.fix_bad_b64 - A (2)
    M 173:4 Cue._mk_descriptors - A (2)
    M 198:4 Cue._set_splice_command - A (2)
    M 296:4 Cue.load_info_section - A (2)
    M 361:4 Cue._xml_splice_info_section - A (2)
    M 69:4 Cue.__repr__ - A (1)
    M 128:4 Cue.get_json - A (1)
    M 187:4 Cue.mk_info_section - A (1)
    M 213:4 Cue.show - A (1)
    M 221:4 Cue.encode - A (1)
    M 245:4 Cue.encode_as_int - A (1)
    M 252:4 Cue.encode2int - A (1)
    M 258:4 Cue.encode_as_hex - A (1)
    M 265:4 Cue.encode2hex - A (1)
    M 272:4 Cue._encode_crc - A (1)
threefive/base.py
    M 139:4 SCTE35Base.load - A (5)
    M 26:4 SCTE35Base._chk_var - A (3)
    M 87:4 SCTE35Base.kv_clean - A (3)
    C 9:0 SCTE35Base - A (2)
    M 21:4 SCTE35Base._chk_nbin - A (2)
    M 54:4 SCTE35Base.as_hms - A (2)
    M 105:4 SCTE35Base.has - A (2)
    M 126:4 SCTE35Base.idxsplit - A (2)
    M 17:4 SCTE35Base.__repr__ - A (1)
    M 40:4 SCTE35Base.as_90k - A (1)
    M 47:4 SCTE35Base.as_ticks - A (1)
    M 68:4 SCTE35Base.fix_hex - A (1)
    M 74:4 SCTE35Base.get - A (1)
    M 80:4 SCTE35Base.get_json - A (1)
    M 113:4 SCTE35Base.xml - A (1)
threefive/segment.py
    M 119:4 Segment.decode - B (6)
    M 57:4 Segment.__init__ - A (4)
    C 16:0 Segment - A (3)
    M 109:4 Segment.show_cue - A (2)
    M 77:4 Segment.__repr__ - A (1)
    M 80:4 Segment._mk_tmp - A (1)
    M 84:4 Segment._aes_get_key - A (1)
    M 88:4 Segment._aes_decrypt - A (1)
    M 95:4 Segment._add_cue - A (1)
    M 102:4 Segment.shushed - A (1)
threefive/bitn.py
    C 8:0 BitBin - A (2)
    M 29:4 BitBin.as_int - A (2)
    M 47:4 BitBin.as_charset - A (2)
    C 93:0 NBin - A (2)
    M 127:4 NBin.add_int - A (2)
    M 145:4 NBin.add_hex - A (2)
    M 168:4 NBin.reserve - A (2)
    M 185:4 NBin.zeroed - A (2)
    M 16:4 BitBin.__init__ - A (1)
    M 21:4 BitBin.as_90k - A (1)
    M 39:4 BitBin.as_hex - A (1)
    M 62:4 BitBin.as_bytes - A (1)
    M 71:4 BitBin.as_flag - A (1)
    M 77:4 BitBin.forward - A (1)
    M 84:4 BitBin.negative_shift - A (1)
    M 101:4 NBin.__init__ - A (1)
    M 106:4 NBin.nbits2bites - A (1)
    M 117:4 NBin.add_bites - A (1)
    M 136:4 NBin.add_90k - A (1)
    M 159:4 NBin.add_flag - A (1)
    M 179:4 NBin.forward - A (1)
threefive/descriptors.py
    M 469:4 SegmentationDescriptor.xml - B (6)
    M 516:4 SegmentationDescriptor.from_xml - A (5)
    M 69:4 SpliceDescriptor.encode - A (4)
    F 12:0 k_by_v - A (3)
    C 22:0 SpliceDescriptor - A (3)
    C 299:0 SegmentationDescriptor - A (3)
    M 367:4 SegmentationDescriptor._decode_segmentation - A (3)
    M 380:4 SegmentationDescriptor._chk_sub_segments - A (3)
    M 458:4 SegmentationDescriptor._encode_segments - A (3)
    F 551:0 splice_descriptor - A (2)
    M 39:4 SpliceDescriptor.parse_tag_and_len - A (2)
    M 50:4 SpliceDescriptor.parse_id - A (2)
    M 61:4 SpliceDescriptor.decode - A (2)
    C 92:0 DVBDASDescriptor - A (2)
    M 111:4 DVBDASDescriptor.decode - A (2)
    C 142:0 AvailDescriptor - A (2)
    M 176:4 AvailDescriptor.from_xml - A (2)
    C 184:0 DtmfDescriptor - A (2)
    M 207:4 DtmfDescriptor.encode - A (2)
    M 234:4 DtmfDescriptor.from_xml - A (2)
    C 244:0 TimeDescriptor - A (2)
    M 291:4 TimeDescriptor.from_xml - A (2)
    M 342:4 SegmentationDescriptor.decode - A (2)
    M 355:4 SegmentationDescriptor._decode_flags - A (2)
    M 389:4 SegmentationDescriptor._decode_segments - A (2)
    M 401:4 SegmentationDescriptor.encode - A (2)
    M 419:4 SegmentationDescriptor._encode_flags - A (2)
    M 432:4 SegmentationDescriptor.mk_the_upid - A (2)
    M 446:4 SegmentationDescriptor._encode_segmentation - A (2)
    M 29:4 SpliceDescriptor.__init__ - A (1)
    M 83:4 SpliceDescriptor._encode_id - A (1)
    M 97:4 DVBDASDescriptor.__init__ - A (1)
    M 128:4 DVBDASDescriptor.encode - A (1)
    M 147:4 AvailDescriptor.__init__ - A (1)
    M 154:4 AvailDescriptor.decode - A (1)
    M 161:4 AvailDescriptor.encode - A (1)
    M 169:4 AvailDescriptor.xml - A (1)
    M 189:4 DtmfDescriptor.__init__ - A (1)
    M 198:4 DtmfDescriptor.decode - A (1)
    M 221:4 DtmfDescriptor.xml - A (1)
    M 249:4 TimeDescriptor.__init__ - A (1)
    M 258:4 TimeDescriptor.decode - A (1)
    M 267:4 TimeDescriptor.encode - A (1)
    M 277:4 TimeDescriptor.xml - A (1)
    M 315:4 SegmentationDescriptor.__init__ - A (1)
threefive/streamtypes.py
    F 105:0 add_stream_types - A (2)
    F 116:0 mk_streamtype_map - A (2)
    F 98:0 hex_literal - A (1)
threefive/crc.py
    F 16:0 _bytecrc - A (2)
    F 25:0 _mk_table - A (2)
    F 31:0 crc32 - A (2)
    F 44:0 crc32hex - A (1)
threefive/section.py
    M 116:4 SpliceInfoSection._encode_encrypted - A (3)
    M 150:4 SpliceInfoSection._encode_splice_command - A (3)
    M 196:4 SpliceInfoSection.from_xml - A (3)
    C 20:0 SpliceInfoSection - A (2)
    M 45:4 SpliceInfoSection.decode - A (2)
    M 91:4 SpliceInfoSection._encode_sap - A (2)
    M 100:4 SpliceInfoSection._encode_section_length - A (2)
    M 108:4 SpliceInfoSection._encode_protocol_version - A (2)
    M 134:4 SpliceInfoSection._encode_cw_index - A (2)
    M 142:4 SpliceInfoSection._encode_tier - A (2)
    M 26:4 SpliceInfoSection.__init__ - A (1)
    M 70:4 SpliceInfoSection._encode_table_id - A (1)
    M 77:4 SpliceInfoSection._encode_section_syntax_indicator - A (1)
    M 84:4 SpliceInfoSection._encode_private_flag - A (1)
    M 128:4 SpliceInfoSection._encode_pts_adjustment - A (1)
    M 162:4 SpliceInfoSection.encode - A (1)
    M 182:4 SpliceInfoSection.xml - A (1)

309 blocks (classes, functions, methods) analyzed.

Average complexity: A (2.087378640776699)                                   <------Boom goes the dynamite.,

a@fu:~/build/SCTE35_threefive$ 

That's a sweet score

python 3.11 scored : A (3.076061694485527) . We beat python.

[futzu]

[futzu]

On Hex: You're right about the attributes , but wrong about the values. we can use hex in the xml value, like for UPIDs.

xsd: Schema says unsignedByte in an attribute cannot be hex or octal

[davemevans]

Correct.

For the UPID value, what's valid will depend on the format specified. Currently this is hexbinary, so my understanding is that it cannot have 0x (https://www.w3.org/TR/xmlschema11-2/#hexBinary), so text would be needed which is what I think Table 22 examples imply they are encoded as.

There are a few examples in other specs (SCTE224, ESAM), open source projects (scte35-go) and various services outputting/ingesting this stuff (eg AWS Media Services), but you are right that there are no canonical examples in any spec, other than Table 22.

[futzu]

@davemevans I don't think that format going to used much , but let's do text for safety.
Any upids , like ATSC, that have Dict for the values, if we use NBin we can pack them together , just like the encoding and get bytes back and well return those as text hex, sound good?

[futzu]

@davemevans , the only upids without a text representation in table 22 are ATSC, and type 1.
type 1 is deprecated but I still see them a lot , they are usually ASCII we can use that as is,
and I'l repack ATSC.

[futzu]

We need a hard target for the next release. I think Dash is not clear, the info is spread out every, and I'm tired of guessing. What we can do , fairly easily, is meet the SCTE-35 XML spec, so let's finish that up, cool? It's on the way to DASH anyway. If there is something you want to work on, just call it. If you got a question, just ask it. Everybody Cue, InfoSection, all the SpliceCommands, all the Descriptors, and all the UPIDS need a xml() method that returns a Node Instance and a from_xml(stuff) that loads from the dict stuff.
I'm going to be in Cue for a minute setting up _xml_splice_command

[futzu]

@davemevans they mention timestamps in ID3tags for TimeDescriptors, I've just about finished that.

[futzu]

I'm just about done with encryption too.

[futzu]

@davemevans where is that dot notation for dictionaries function? I wanted to check that out.

[davemevans]

https://github.com/futzu/SCTE35_threefive/pull/104/files#diff-a3eb7a9382a17babb6cdb0d66b2d3979c529dfe8fd801b27ce17f57e4a807ef4

[futzu]

new features @davemevans @wabiloo
from xml to json , base64, hex, or bytes

a@fu:~/build/SCTE35_threefive$ cat xml.xml | ./threefive-exp encode
/DBRAAAAAyiYAP/wFAUAAAABf+//hqqjQv4ApMbEmZkBAQAsAhlDVUVJAAAAAH+AAQgxMDEwMDAwMDQAAAAAAg9DVUVJAAAAAH+AAQA1AAB2ET7C


a@fu:~/build/SCTE35_threefive$ cat xml.xml | ./threefive-exp encode hex
0xfc305100000003289800fff01405000000017fefff86aaa342fe00a4c6c499990101002c021943554549000000007f80010831303130303030303400000000020f43554549000000007f80010035000076113ec2



a@fu:~/build/SCTE35_threefive$ cat xml.xml | ./threefive-exp encode int
19303799839681125151192699317799016597380785501437947955479178660148601939352676218289447838560748251043382782718707769451403069255977371387116302731347908410111893562262597904427004019893411478196272834




a@fu:~/build/SCTE35_threefive$ cat xml.xml | ./threefive-exp encode bytes
b'\xfc0Q\x00\x00\x00\x03(\x98\x00\xff\xf0\x14\x05\x00\x00\x00\x01\x7f\xef\xff\x86\xaa\xa3B\xfe\x00\xa4\xc6\xc4\x99\x99\x01\x01\x00,\x02\x19CUEI\x00\x00\x00\x00\x7f\x80\x01\x08101000004\x00\x00\x00\x00\x02\x0fCUEI\x00\x00\x00\x00\x7f\x80\x01\x005\x00\x00v\x11>\xc2'

a@fu:~/build/SCTE35_threefive$ cat xml.xml | ./threefive-exp encode json

[futzu]

upids with verified working xml()

• 0x0 : No Upid
• 0x1 : Deprecated
• 0x3 : AdID
• 0x4 : UMID
• 0x6 : ISAN
• 0x7 : TID
• 0x8 : AiringID
• 0x9 : ADI
• 0xa : EIDR
• 0xb : ATSC
• 0xc : MPU
• 0xd : MID
• 0xe : ADS Info
• 0xf : URI

[davemevans]

This is looking awesome - almost there!

One thing I noticed is that MID didn't quite work properly since it dumps the contained upids directly with incorrect attrs, whereas it should call the underlying upid.xml(). That's an easy enough fix - can PR that later.

When calling Cue.from_xml, I found that when you have a MID in XML and want to from_xml it, only one of them ends up in the JSON and some values are overwritten. This will need special handling in the XML parser to ensure a list of segmentation upids attached to the descriptor. I'm wasn't really sure on the way forward here as I'm not that experienced with XMLParser and the codes looks pretty gnarly 😄. Here's an example XML to demonstrate - see that the ADI is passed to the EIDR encoder:

<SpliceInfoSection xmlns="http://www.scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <TimeSignal>
      <SpliceTime ptsTime="4294967296"/>
   </TimeSignal>
   <!-- Distributor Placement Opportunity End -->
   <SegmentationDescriptor segmentationEventId="2" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="55" segmentNum="1" segmentsExpected="3">
      <!-- UPID: MID -->
      <SegmentationUpid upidType="10" name="EIDR">0x14778be5e3f6000000000000</SegmentationUpid>
      <SegmentationUpid upidType="10" name="EIDR">0x14778be5e3f6000000000000</SegmentationUpid>
      <SegmentationUpid upidType="9" name="ADI">SIGNAL:Ly9EMGxKR0hFZUtpMHdCUVZnRUFnZz1</SegmentationUpid>
   </SegmentationDescriptor>
</SpliceInfoSection>

[futzu]

@davemevans , you got to shorten these names up , unescape_special_chars, you sound like a damn dot Net programmer :)

unescape and escape or something is fine by me , I mean most of my vars are named "stuff" or "data",

[futzu]

After we do this release, I want to fold XmlParser into Node.
XmlParser parses xml and makes a dictionary,
but it should be making Node instances

Look at these methods, it all lines up

    def mk_active(self, node):            #Node.name

    def mk_value(self, value, stuff):      # Node.value

    def mk_attrs(self, node):                 # Node.attrs

instead passing the stuff dict to SpliceInfoSection, we should pass a Node instance.

SpliceInfoSections currently loads stuff['SpliceInfoSection"],
it just as easily load Node.attrs from a Node instance, it's the same thing.

Does that make sense?

[davemevans]

Yeah, makes total sense - love it.

[futzu]

Have you ever look at cuei, my other SCTE-35 parser in Go?
It's done very differently from threefive. I was paid to do it but that wasn't the issue, it was strong typing in Go, it broke my dot notation. Since a Command can be several different types,, as well as Upids and Descriptors. I made a struct for a SpliceCommand, type that had all the variables for all the SpliceCommands. To return a Splice Command that was a SpliceInsert, I marshaled the data into a SpliceInsert struct.

https://github.com/futzu/cuei/blob/0a62872963adccb57004e29d06718dd13c0b4fa4/commands.go

I feel like was the best way to do it.
The company that paid me to do cuei, didn't get the marshal thing so I put it back

I feel like that's the approach we should be using.
I think we have xml working but it's not right.

However this OO, we need more classes.

Every xml Node that we use , should have a class.

XmlSpliceInsert
XmlTimeSignal
etc...

This not just displaying the data in a different format, this is different data.
When We see a "<SpliceInsert" in xml we should load the chunk from there to " "
into a XmlSplixceInsert Node

I think XmlSpliceInsert and SpliceInsert should marshal into each other .
The logic is with the object.

[futzu]

The Xml Types in the xsd should be classes in threefive.

[futzu]

This is why we aren't using etree.

[futzu]

@davemevans I'm going to do release on Wednesday I think.as long as what is working is stable, even if some of the UPIDs are not complete.
There is a lot o new code, and only about half is xml related. I want to get it to the people so we can get the bugs out.
Wednesday is not hard date, we'll see how it goes.

[futzu]

Did you see fixsix, it works pretty well, but the code is kind of clunky.
I tried to compress the cli help down a little , here's what it looks like now....

[futzu]

@davemevans

Upids: with a working from_xml() and xml()

    0x0 : No Upid
    0x1 : Deprecated
    0x2: Deprecated
    0x3 : AdID
    0x4 : UMID
    0x6 : ISAN
    0x7 : TID
    0x8 : AiringID
    0x9 : ADI
    0xa : EIDR
    0xe : ADS Info
    0xf : URI

UPIDS with just a working xml()

    0xb : ATSC
    0xc : MPU
    0xd : MID

I am okay with that for this release.

[futzu]

I have 45 GB worth of mpegts I use for testing. They are from ABC, Disney, CBS, CNN, MsNBC, NBC, NewMax and a few more sources.

They contain: 665 upids:

• 8 No Upid ,
• 7 AiringID,
• 640 type 1 (Deprecated)

[davemevans]

Worth noting many of the from_xml only appear to work because the XML output from threefive has the non-standard segmentationUpidLength attribute included.

I would agree that this is ok for this release, but in the long term this would need addressing. Maybe worth opening a tracking issue.

[futzu]

0,3,4,7,9,10,14,15 all work without length.
1 and 8 don't though, and they are the most used.

[futzu]

Now, 0,1,2,3,4,5,6,7,8,9,10,14,15 all work without length

[futzu]

"Worth noting many of the from_xml only appear to work because the XML output from threefive has the non-standard segmentationUpidLength attribute included." said Dave.

You got in my head with that one, I added length to the upid_map for all the upids with fixed sizes, and for the ones that are variable length, I am grabbing the length after we load the xml and call encode. Now, we don't need length in xml, and I think I took them all out.

[futzu]

@davemevans

Howw about you tune xml.py ?

cyclomatic complexity needs to be closer to 2 , do what you can, don't let it make you crazy.

pylint score needs to be over 9.75

python3 -mpip install radon


a@fu:~/build/SCTE35_threefive$ radon cc -sa threefive/xml.py 

threefive/xml.py
    M 160:4 Node.mk - B (6)
    F 26:0 un_xml - A (5)
    M 248:4 XmlParser.mk_attrs - A (5)
    F 42:0 iter_attrs - A (4)
    M 271:4 XmlParser._parse_nodes - A (4)
    F 17:0 un_camel - A (3)
    F 53:0 val2xml - A (3)
    F 90:0 escape - A (3)
    F 101:0 unescape - A (3)
    C 112:0 Node - A (3)
    M 146:4 Node.set_depth - A (3)
    C 195:0 Comment - A (3)
    C 204:0 XmlParser - A (3)
    M 220:4 XmlParser.chk_node_list - A (3)
    M 280:4 XmlParser._parse_most - A (3)
    F 64:0 key2xml - A (2)
    F 74:0 mk_xml_attrs - A (2)
    M 134:4 Node.__init__ - A (2)
    M 196:4 Comment.mk - A (2)
    M 229:4 XmlParser.mk_value - A (2)
    M 260:4 XmlParser.parse - A (2)
    M 299:4 XmlParser._parse_descriptor - A (2)
    F 8:0 t2s - A (1)
    M 143:4 Node.__repr__ - A (1)
    M 156:4 Node.get_indent - A (1)
    M 183:4 Node.add_child - A (1)
    M 189:4 Node.add_comment - A (1)
    M 216:4 XmlParser.__init__ - A (1)
    M 241:4 XmlParser.mk_active - A (1)
    M 315:4 XmlParser._skip_comment - A (1)

30 blocks (classes, functions, methods) analyzed.
Average complexity: A (2.533333333333333)

a@fu:~/build/SCTE35_threefive$ pylint threefive/xml.py

************* Module threefive.xml
threefive/xml.py:134:4: W0102: Dangerous default value {} as argument (dangerous-default-value)    # <---- I don't care about these

threefive/xml.py:156:4: C0116: Missing function or method docstring (missing-function-docstring)
threefive/xml.py:195:0: C0115: Missing class docstring (missing-class-docstring)
threefive/xml.py:204:0: C0115: Missing class docstring (missing-class-docstring)
threefive/xml.py:248:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)


threefive/xml.py:308:8: W0702: No exception type(s) specified (bare-except)        # <---- I don't care about these

------------------------------------------------------------------
Your code has been rated at 9.61/10 (previous run: 9.42/10, +0.19)

[davemevans]

Yes, happy to take a crack at this. But it won't be til tomorrow morning now.

[futzu]

no rush.

[futzu]

@davemevans in XmlParser.mkattrs

Just call unescape on the whole node,

    def mk_attrs(self, node):
        """
        mk_attrs parses the current node for attributes
        and stores them in self.stuff[self.active]
        """
        if "<!--" not in unescape(node):  #    <---- Call unescape   on on the whole node
            attrs = [x for x in node.split(" ") if "=" in x]
            parsed = {
                x.split('="')[0]: x.split('="')[1].split('"')[0] #  <----- You're call it here and  in mk_value
                for x in attrs
            }
            it = iter_attrs(parsed)
            return it

It's way more efficient to call it on the whole thing.

[davemevans]

It would be more efficient, but parsing would then fail for attributes that contain " because eg """ would be unescaped to """ which would not be split correctly. Probably not an issue for schema-compliant attributes, but anything in ##any could contain this construct.

[futzu]

i got a hex fix for section.py that cleans up so other stuff I did earlier too. in xml.py val2xml will handle hex to int conversion,

[futzu]

@davemevans when you have a minute, Read the README and and see if the xml stuff makes sense, please.

[davemevans]

Yeah, that's annoying. The spec is clear that you derive the length after converting the XML representation to bitstream format, so the measuring should be done after encoding back to the native format.

This is fine for now, but probably a more complete solution for parsing from XML is needed for the future. Worth opening an issue to cover this and come to it later. I have a list of other things that need addressing down the line - where would be the best place to log these - GH issues? This discussion? New discussion?

[futzu]

I got all the UPID stuff sorted, everything is working.
I don't see me doing further xml development, I asked everyone I've worked for in the last couple of years and nobody cares about DASH, and I don't see a need to invest anymore time in it. threefive is SCTE-35 specific, it's not a DASH parser.


The way to use threefive with DASH:

1. parse the mpd
2. when you get to an event node, pass it to threefive for parsing.

   <EventStream
      schemeIdUri="urn:scte:scte35:2014:xml+bin"
      timescale="1000">
      <!-- 2024-09-10T05:33:47.520000Z - 2024-09-10T05:34:25.920000Z -->
      <Event
        presentationTime="1725946427520"
        duration="38400"
        id="14268737">
        <Signal
          xmlns="http://www.scte.org/schemas/35/2016">
          <Binary>/DAgAAAAAAAAAP/wDwUA2blBf//+ADS8AMAAAAAAAORhJCQ=</Binary>
        </Signal>
      </Event>
      <!-- 2024-09-10T05:39:50.400000Z - 2024-09-10T05:40:28.800000Z -->
    </EventStream>

from threefive import Cue

cue = Cue()
cue.load(data)

• That's all xml parsing threefive needs to do , you have the Cue instance cue, and all the SCTE-35 decoded.

cue.show()

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0x961b236d"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14268737,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": [],
    "dash_data": {
        "EventStream": {
            "scheme_id_uri": "urn:scte:scte35:2014:xml+bin",
            "timescale": 1000
        },
        "Event": {
            "presentation_time": 1725946427520,
            "duration": 0.426667,
            "id": 14268737
        },
        "Signal": {
            "xmlns": "http://www.scte.org/schemas/35/2016"
        },
        "Binary": {
            "binary": "/DAgAAAAAAAAAP/wDwUA2blBf//+ADS8AMAAAAAAAORhJCQ="
        }
    }
}

[davemevans]

XML representation of cues is part of SCTE35 so I think it is reasonable for parsing from and serialising to a SpliceInfoSection to be in scope for a SCTE35-specific parser.

Regardless, thanks for your efforts on this up to this point - it's genuinely much appreciated.

[futzu]

@davemevans I'm not following you man, how is it not doing that?

>>>> from threefive import Cue
>>>> more_xml = """<Period start="PT444334H55M0.010S" id="break-1">
....     <EventStream timescale="90000" schemeIdUri="urn:scte:scte35:2013:xml">
....           <Event duration="540">
....                   <scte35:SpliceInfoSection protocolVersion="0" ptsAdjustment="140429" tier="4095">
....                             <scte35:SpliceInsert spliceEventId="11" spliceEventCancelIndicator="false" outOfNetworkIndicator="true"                                                                           spliceImmediateFlag="true" uniqueProgramId="1"availNum="1" availsExpected="1">
....                     <scte35:Program><scte35:SpliceTime ptsTime="5272624400"/></scte35:Program>
....                       <scte35:BreakDuration autoReturn="true" duration="9450000"/>
....                       </scte35:SpliceInsert>
....                         </scte35:SpliceInfoSection>
....                  </Event>
....                </EventStream>
....                 </Period>
....                """
>>>> cue=Cue()
>>>> cue.load(more_xml)
True
>>>> print(cue.xml())
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="140429" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="11" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="1" availsExpected="true" outOfNetworkIndicator="true" uniqueProgramId="1">
      <Program>
         <SpliceTime ptsTime="5272624400"/>
      </Program>
      <BreakDuration autoReturn="true" duration="9450000"/>
   </SpliceInsert>
</SpliceInfoSection>

[futzu]

I can even modify the cue

>>>> ad=AvailDescriptor()
>>>> ad.provider_avail_id = 0x1234567
>>>> cue.descriptors.append(ad)
>>>> cue.encode()
'/DBHAAGRZOeYAP/wBQb+jzHOMAAxAiVDVUVJAAAAAX+/ARRtc25iY19FUDA0MzExMjIxMDU1NzYBAQAAAAhDVUVJASNFZ/rzHLQ='
>>>> print(cue.xml())
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="6734276504" protocolVersion="0" sapType="3" tier="4095">
   <TimeSignal>
      <SpliceTime ptsTime="2402405936"/>
   </TimeSignal>
   <!-- Distributor Placement Opportunity Start -->
   <SegmentationDescriptor segmentationEventId="1" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="54" segmentNum="1" segmentsExpected="1" subSegmentNum="0" subSegmentsExpected="0">
      <!-- UPID: Deprecated -->
      <SegmentationUpid segmentationUpidType="1" segmentationUpidFormat="hexbinary">msnbc_EP043112210557</SegmentationUpid>
   </SegmentationDescriptor>
   <AvailDescriptor providerAvailId="19088743"/>
</SpliceInfoSection>

>>>> 

print it as json

>>>> cue.show()
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 71,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 74825.294489,
        "cw_index": "0x0",
        "tier": 4095,
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 49,
        "crc": "0xfaf31cb4"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 26693.399289
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 37,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "segmentation_event_id": "0x1",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": true,
            "segmentation_duration": 60.033333,
            "segmentation_message": "Distributor Placement Opportunity Start",
            "segmentation_upid_type": 1,
            "segmentation_upid_type_name": "Deprecated",
            "segmentation_upid_length": 20,
            "segmentation_upid": "msnbc_EP043112210557",
            "segmentation_type_id": 54,
            "segment_num": 1,
            "segments_expected": 1,
            "sub_segment_num": 0,
            "sub_segments_expected": 0
        },
        {
            "tag": 0,
            "descriptor_length": 8,
            "name": "Avail Descriptor",
            "identifier": "CUEI",
            "provider_avail_id": 19088743
        }
    ]
}
>>>> 

the cli tool can convert from xml to base64,bytes, int, hex, json and vice versa. It will output xml when you parse mpegts.

What does it not do ?

[futzu]

@davemevans @wabiloo I can parse mpd filles over the network and convert the data to cues, no problem.

a@fu:~/build/SCTE35_threefive$ pypy3 mpdp.py https://demo.unified-streaming.com/k8s/live/stable/scte35-no-splicing.isml/.mpd
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294827,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294828,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294829,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294830,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294831,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []

[futzu]

You can convert them to cues, and the convert the cues back to xml

}
a@fu:~/build/SCTE35_threefive$ pypy3 mpdp.py https://demo.unified-streaming.com/k8s/live/stable/scte35-no-splicing.isml/.mpd
{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294830,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="14294830" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
      <BreakDuration autoReturn="true" duration="3456000"/>
   </SpliceInsert>
</SpliceInfoSection>

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294831,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="14294831" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
      <BreakDuration autoReturn="true" duration="3456000"/>
   </SpliceInsert>
</SpliceInfoSection>

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294832,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="14294832" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
      <BreakDuration autoReturn="true" duration="3456000"/>
   </SpliceInsert>
</SpliceInfoSection>

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294833,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="14294833" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
      <BreakDuration autoReturn="true" duration="3456000"/>
   </SpliceInsert>
</SpliceInfoSection>

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x03",
        "sap_details": "No Sap Type",
        "section_length": 32,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment": 0.0,
        "cw_index": "0x00",
        "tier": "0x0fff",
        "splice_command_length": 15,
        "splice_command_type": 5,
        "descriptor_loop_length": 0,
        "crc": "0xe4612424"
    },
    "command": {
        "command_length": 15,
        "command_type": 5,
        "name": "Splice Insert",
        "break_auto_return": true,
        "break_duration": 38.4,
        "splice_event_id": 14294834,
        "splice_event_cancel_indicator": false,
        "out_of_network_indicator": true,
        "program_splice_flag": true,
        "duration_flag": true,
        "splice_immediate_flag": true,
        "event_id_compliance_flag": true,
        "unique_program_id": 49152,
        "avail_num": 0,
        "avails_expected": 0
    },
    "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
   <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
      <BreakDuration autoReturn="true" duration="3456000"/>
   </SpliceInsert>
</SpliceInfoSection>

a@fu:~/build/SCTE35_threefive$