Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aTalk Jingle implementation for review #520

Open
wants to merge 42 commits into
base: 4.4
Choose a base branch
from

Conversation

cmeng-git
Copy link
Contributor

Thank you for your contribution!

Before creating a Pull Request, please make sure to

aTalk jingle implementation

jingle classes implementatation in aTalk are sub-classed/based on two defined based classes.
a. AbstractXmlElement
b. DefaultXmlElementProvide

With the above two based classes, all jingle classes implementations are simpler and clearer.
The pull requested classes for review are based on the above implementation topologies;
these classe provide the support of the aTalk media call (video/audio) setup as shown in the logcat below.

Perhaps the smack team would review the aTalk jingle implementation, possibly consider for integration
with the samck future releases.

You may refer to aTalk github (v2.9.0) for full implementation and use the jingle classes.
https://github.com/cmeng-git/atalk-android

============== aTalk logcat for media call setup (caller) with encryption support ================
2022-03-02 08:17:32.535 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <message xml:lang='us' to='peacock@atalk.sytes.net' id='jm-propose-mfifd5cm6vh9' type='chat'>
      <propose xmlns='urn:xmpp:jingle-message:0' id='mfifd5cm6vh9'>
        <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='audio'/>
        <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='video'/>
      </propose>
    </message>

2022-03-02 08:17:33.811 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <message xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='chat'>
      <proceed xmlns='urn:xmpp:jingle-message:0' id='mfifd5cm6vh9'/>
    </message>

2022-03-02 08:17:37.187 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='7FUY4-17' type='set'>
      <jingle xmlns='urn:xmpp:jingle:1' initiator='swordfish@atalk.sytes.net/atalk' action='session-initiate' sid='mfifd5cm6vh9'>
        <content creator='initiator' name='audio'>
          <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='audio'>
            <payload-type id='96' name='opus' channels='2' clockrate='48000'>
              <parameter name='useinbandfec' value='1'/>
              <parameter name='ptime' value='20'/>
            </payload-type>
            <payload-type id='97' name='SILK' channels='1' clockrate='24000'/>
            <payload-type id='98' name='SILK' channels='1' clockrate='16000'/>
            <payload-type id='100' name='speex' channels='1' clockrate='32000'/>
            <payload-type id='102' name='speex' channels='1' clockrate='16000'/>
            <payload-type id='0' name='PCMU' channels='1' clockrate='8000'/>
            <payload-type id='8' name='PCMA' channels='1' clockrate='8000'/>
            <payload-type id='103' name='iLBC' channels='1' clockrate='8000'/>
            <payload-type id='3' name='GSM' channels='1' clockrate='8000'/>
            <payload-type id='104' name='speex' channels='1' clockrate='8000'/>
            <payload-type id='101' name='telephone-event' channels='1' clockrate='8000'/>
            <rtp-hdrext xmlns='urn:xmpp:jingle:apps:rtp:rtp-hdrext:0' uri='urn:ietf:params:rtp-hdrext:ssrc-audio-level' senders='both' id='1'>
              <parameter name='attributes'/>
            </rtp-hdrext>
            <rtp-hdrext xmlns='urn:xmpp:jingle:apps:rtp:rtp-hdrext:0' uri='urn:ietf:params:rtp-hdrext:csrc-audio-level' senders='initiator' id='2'>
              <parameter name='attributes'/>
            </rtp-hdrext>
            <rtcp-mux/>
            <encryption>
              <crypto tag='1' crypto-suite='AES_256_CM_HMAC_SHA1_80' key-params='inline:Xh9N87qTsildR2scJBrB92Sga3SvmVuWv+PbkrzJqY7/FOlbIw5j1KeOCayIcw=='/>
              <crypto tag='2' crypto-suite='AES_256_CM_HMAC_SHA1_32' key-params='inline:gQraPzdhXW9Gm08gEECuyuV6JOVIitYk5tfFOV8Qn9NJLVh6kAqHX2zkKXruMQ=='/>
              <crypto tag='3' crypto-suite='AES_192_CM_HMAC_SHA1_80' key-params='inline:TPB6qgU/ltJ9yE6ASNoTFdcOhHEC8SILDpmEsR53rhX1ESoq1pE='/>
              <crypto tag='4' crypto-suite='AES_192_CM_HMAC_SHA1_32' key-params='inline:DWGell8boXShsX0vfoHtF5+sgLAfq/i4KhBpezAp0M7FIvWAPTM='/>
              <zrtp-hash xmlns='urn:xmpp:jingle:apps:rtp:zrtp:1' version='1.10'>
                be20efff52dcfb1a8701a71e81eabc8683740a01b5d4996496f6d6a3da6e986e
              </zrtp-hash>
            </encryption>
          </description>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='35pt1ft3vqgfb' pwd='3pjs4fpsq663fijlkgd7ppthfd'>
            <rtcp-mux/>
            <candidate foundation='1' component='1' protocol='udp' priority='2130706431' generation='0' id='25' ip='fe80::1004:3eff:fe07:ba63' port='5006' type='host' network='0'/>
            <candidate foundation='2' component='1' protocol='udp' priority='2130706431' generation='0' id='26' ip='192.168.1.167' port='5006' type='host' network='0'/>
            <candidate foundation='3' component='1' protocol='udp' priority='2113932031' generation='0' id='27' ip='10.168.71.219' port='5006' type='host' network='0'/>
            <candidate foundation='4' component='1' protocol='udp' priority='1677724415' generation='0' id='28' ip='42.60.7.13' port='5006' type='srflx' rel-addr='192.168.1.167' rel-port='5006' network='0'/>
            <fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0' hash='sha-256' setup='actpass'>
              2D:2B:8E:3E:CD:E4:8B:D9:9F:D4:18:3C:11:F8:72:F5:62:92:C0:A8:1B:06:55:36:54:A7:59:D2:22:33:9A:C3
            </fingerprint>
          </transport>
        </content>
        <content creator='initiator' name='video'>
          <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='video'>
            <payload-type id='105' name='h264' clockrate='90000'>
              <parameter name='profile-level-id' value='42E01f'/>
              <parameter name='packetization-mode' value='1'/>
            </payload-type>
            <payload-type id='99' name='h264' clockrate='90000'>
            <parameter name='profile-level-id' value='42E01f'/>
            </payload-type>
            <payload-type id='106' name='VP8' clockrate='90000'/>
            <payload-type id='107' name='VP9' clockrate='90000'/>
            <rtcp-mux/>
            <encryption>
              <crypto tag='1' crypto-suite='AES_256_CM_HMAC_SHA1_80' key-params='inline:QgnRERIHA+vNb5YNKxw+6NKroV4DWPb2LbTZXn6CKppseGSXnL+FbcgcunVuiQ=='/>
              <crypto tag='2' crypto-suite='AES_256_CM_HMAC_SHA1_32' key-params='inline:Xkutf5bMSXngXsqtG2o80lCtYjtlKoQciBuNtIIgDOFKK6imBObfs0Th9xdA1A=='/>
              <crypto tag='3' crypto-suite='AES_192_CM_HMAC_SHA1_80' key-params='inline:Vm4ag0qn/3DGwawtXS9q20GfPz68HioZyu/wgQ0fdlEvFTLjFNc=
              <crypto tag='4' crypto-suite='AES_192_CM_HMAC_SHA1_32' key-params='inline:8Zvyt9/Aw8XZxNEHlt9VORH/IzJQiYHR4ExQankLbLcJwu0z9w8='/>
              <zrtp-hash xmlns='urn:xmpp:jingle:apps:rtp:zrtp:1' version='1.10'>
                f41a577b69081a4f1f76acbcbc5fd79632c36dee69cde297dd4e867e15826f58
              </zrtp-hash>
            </encryption>
          </description>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='35pt1ft3vqgfb' pwd='3pjs4fpsq663fijlkgd7ppthfd'>
            <rtcp-mux/>
            <candidate foundation='1' component='1' protocol='udp' priority='2130706431' generation='0' id='29' ip='fe80::1004:3eff:fe07:ba63' port='5007' type='host' network='0'/>
            <candidate foundation='2' component='1' protocol='udp' priority='2130706431' generation='0' id='30' ip='192.168.1.167' port='5007' type='host' network='0'/>
            <candidate foundation='3' component='1' protocol='udp' priority='2113932031' generation='0' id='31' ip='10.168.71.219' port='5007' type='host' network='0'/>
            <candidate foundation='4' component='1' protocol='udp' priority='1677724415' generation='0' id='32' ip='42.60.7.13' port='5007' type='srflx' rel-addr='192.168.1.167' rel-port='5007' network='0'/>
            <fingerprint xmlns='urn:xmpp:jingle:apps:dtls:0' hash='sha-256' setup='actpass'>
              2D:2B:8E:3E:CD:E4:8B:D9:9F:D4:18:3C:11:F8:72:F5:62:92:C0:A8:1B:06:55:36:54:A7:59:D2:22:33:9A:C3
            </fingerprint>
          </transport>
        </content>
      </jingle>
    </iq>

2022-03-02 08:17:37.344 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='result' id='7FUY4-17'/>


2022-03-02 08:17:37.734 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='set' id='D4YPW-70'>
      <jingle xmlns='urn:xmpp:jingle:1' initiator='peacock@atalk.sytes.net/atalk' action='transport-info' sid='mfifd5cm6vh9'>
        <content creator='initiator' name='audio'>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='6pov41ft3vqml4' pwd='4j8lih1qf01o74ar6ou0a7cdq4'>
            <rtcp-mux/>
            <candidate foundation='1' component='1' protocol='udp' priority='2130706431' generation='0' id='97' ip='fe80::362d:dff:fe00:ae96' port='5032' type='host' network='0'/>
            <candidate foundation='2' component='1' protocol='udp' priority='2130706431' generation='0' id='98' ip='192.168.1.37' port='5032' type='host' network='0'/>
            <candidate foundation='3' component='1' protocol='udp' priority='1677724415' generation='0' id='99' ip='42.60.7.13' port='5032' type='srflx' rel-addr='192.168.1.37' rel-port='5032' network='0'/>
          </transport>
        </content>
      </jingle>
    </iq>

2022-03-02 08:17:37.758 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='D4YPW-70' type='result'>
    </iq>

2022-03-02 08:17:38.491 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='set' id='D4YPW-71'>
      <jingle xmlns='urn:xmpp:jingle:1' initiator='peacock@atalk.sytes.net/atalk' action='transport-info' sid='mfifd5cm6vh9'>
        <content creator='initiator' name='video' senders='initiator'>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='6pov41ft3vqml4' pwd='4j8lih1qf01o74ar6ou0a7cdq4'>
            <rtcp-mux/>
            <candidate foundation='1' component='1' protocol='udp' priority='2130706431' generation='0' id='100' ip='fe80::362d:dff:fe00:ae96' port='5033' type='host' network='0'/>
            <candidate foundation='2' component='1' protocol='udp' priority='2130706431' generation='0' id='101' ip='192.168.1.37' port='5033' type='host' network='0'/>
            <candidate foundation='3' component='1' protocol='udp' priority='1677724415' generation='0' id='102' ip='42.60.7.13' port='5033' type='srflx' rel-addr='192.168.1.37' rel-port='5033' network='0'/>
          </transport>
        </content>
      </jingle>
    </iq>

2022-03-02 08:17:38.513 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='D4YPW-71' type='result'>
    </iq>

2022-03-02 08:17:38.587 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='set' id='D4YPW-72'>
      <jingle xmlns='urn:xmpp:jingle:1' initiator='peacock@atalk.sytes.net/atalk' action='session-info' sid='mfifd5cm6vh9'>
        <ringing xmlns='urn:xmpp:jingle:apps:rtp:info:1'/>
      </jingle>
    </iq>

2022-03-02 08:17:38.604 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='D4YPW-72' type='result'>
    </iq>

2022-03-02 08:17:38.916 5702-6136/org.atalk.android D/SMACK: RECV (0): 
    <iq xml:lang='en-GB' to='swordfish@atalk.sytes.net/atalk' from='peacock@atalk.sytes.net/atalk' type='set' id='D4YPW-73'>
      <jingle xmlns='urn:xmpp:jingle:1' responder='peacock@atalk.sytes.net/atalk' action='session-accept' sid='mfifd5cm6vh9'>
        <content creator='initiator' name='audio'>
          <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='audio' ssrc='1928004519'>
            <payload-type id='96' name='opus' channels='2' clockrate='48000'>
              <parameter name='useinbandfec' value='1'/>
              <parameter name='usedtx' value='1'/>
              <parameter name='ptime' value='20'/>
            </payload-type>
            <payload-type id='97' name='SILK' channels='1' clockrate='24000'/>
            <payload-type id='98' name='SILK' channels='1' clockrate='16000'/>
            <payload-type id='100' name='speex' channels='1' clockrate='32000'/>
            <payload-type id='102' name='speex' channels='1' clockrate='16000'/>
            <payload-type id='0' name='PCMU' channels='1' clockrate='8000'/>
            <payload-type id='8' name='PCMA' channels='1' clockrate='8000'/>
            <payload-type id='103' name='iLBC' channels='1' clockrate='8000'/>
            <payload-type id='3' name='GSM' channels='1' clockrate='8000'/>
            <payload-type id='104' name='speex' channels='1' clockrate='8000'/>
            <payload-type id='101' name='telephone-event' channels='1' clockrate='8000'/>
            <rtp-hdrext xmlns='urn:xmpp:jingle:apps:rtp:rtp-hdrext:0' uri='urn:ietf:params:rtp-hdrext:ssrc-audio-level' senders='both' id='1'>
              <parameter name='attributes'/>
            </rtp-hdrext>
            <rtp-hdrext xmlns='urn:xmpp:jingle:apps:rtp:rtp-hdrext:0' uri='urn:ietf:params:rtp-hdrext:csrc-audio-level' senders='none' id='2'>
              <parameter name='attributes'/>
            </rtp-hdrext>
            <rtcp-mux/>
            <encryption>
              <zrtp-hash xmlns='urn:xmpp:jingle:apps:rtp:zrtp:1' version='1.10'>
                015f636c40d9cbb431be09612ec309203776d0d43115ab175c948697532bb357
              </zrtp-hash>
            </encryption>
            <source xmlns='urn:xmpp:jingle:apps:rtp:ssma:0' ssrc='1928004519'>
              <parameter name='cname' value='05e900d8-46ee-4d1f-8b6f-ec21b6dc90e0'/>
              <parameter name='msid' value='082cdfe1-1fb7-409e-864d-740349055a81 audio108491334'/>
              <parameter name='mslabel' value='082cdfe1-1fb7-409e-864d-740349055a81'/>
              <parameter name='label' value='audio108491334'/>
            </source>
          </description>
          <inputevt xmlns='http://jitsi.org/protocol/inputevt'/>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='6pov41ft3vqml4' pwd='4j8lih1qf01o74ar6ou0a7cdq4'/>
        </content>
        <content creator='initiator' name='video' senders='both'>
          <description xmlns='urn:xmpp:jingle:apps:rtp:1' media='video' ssrc='1974578385'>
            <payload-type id='106' name='VP8' clockrate='90000'/>
            <payload-type id='107' name='VP9' clockrate='90000'/>
            <rtcp-mux/>
            <encryption>
              <zrtp-hash xmlns='urn:xmpp:jingle:apps:rtp:zrtp:1' version='1.10'>
                daf5a5f92b9713ab023d2e847874a6209ccf8152532df9a1915a1f3d3512154d
              </zrtp-hash>
            </encryption>
            <source xmlns='urn:xmpp:jingle:apps:rtp:ssma:0' ssrc='1974578385'>
              <parameter name='cname' value='05e900d8-46ee-4d1f-8b6f-ec21b6dc90e0'/>
              <parameter name='msid' value='082cdfe1-1fb7-409e-864d-740349055a81 video108491334'/>
              <parameter name='mslabel' value='082cdfe1-1fb7-409e-864d-740349055a81'/>
              <parameter name='label' value='video108491334'/>
            </source>
          </description>
          <inputevt xmlns='http://jitsi.org/protocol/inputevt'/>
          <transport xmlns='urn:xmpp:jingle:transports:ice-udp:1' ufrag='6pov41ft3vqml4' pwd='4j8lih1qf01o74ar6ou0a7cdq4'/>
        </content>
      </jingle>
    </iq>

2022-03-02 08:17:38.946 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='D4YPW-73' type='result'>
    </iq>


2022-03-02 08:17:39.239 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() Building flow graph for: null
2022-03-02 08:17:39.242 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() Building Track: 0
2022-03-02 08:17:39.249 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() Input: opus/rtp, 48000.0 Hz, Stereo
2022-03-02 08:17:39.252 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() Custom options specified.
2022-03-02 08:17:39.255 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() A custom codec is specified: org.atalk.impl.neomedia.audiolevel.AudioLevelEffect@b9299c6
2022-03-02 08:17:39.259 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() A custom renderer is specified: org.atalk.impl.neomedia.jmfext.media.renderer.audio.AudioTrackRenderer@d69fd87
2022-03-02 08:17:39.410 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()     
2022-03-02 08:17:39.420 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.comment() Here's the completed flow graph:
2022-03-02 08:17:39.425 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()       net.sf.fmj.media.parser.RawPushBufferParser@eca44b4
2022-03-02 08:17:39.428 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.codec.audio.opus.JNIDecoder@44648dd
2022-03-02 08:17:39.432 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          format: opus/rtp, 48000.0 Hz, Stereo
2022-03-02 08:17:39.437 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.codec.audio.opus.JNIDecoder@44648dd
2022-03-02 08:17:39.441 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.audiolevel.AudioLevelEffect@b9299c6
2022-03-02 08:17:39.445 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          format: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed, FrameSize=16 bits
2022-03-02 08:17:39.449 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.audiolevel.AudioLevelEffect@b9299c6
2022-03-02 08:17:39.453 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.jmfext.media.renderer.audio.AudioTrackRenderer@d69fd87
2022-03-02 08:17:39.458 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()          format: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed, FrameSize=16 bits
2022-03-02 08:17:39.463 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.write()     
2022-03-02 08:17:39.466 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.profile() Profile: graph building: 224 ms
2022-03-02 08:17:39.470 5702-8507/org.atalk.android D/aTalk: [746] net.sf.fmj.media.Log.profile() Profile: realize, post graph building: 1 ms

2022-03-02 08:17:39.567 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() Building flow graph for: audiorecord:
2022-03-02 08:17:39.573 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() Building Track: 0
2022-03-02 08:17:39.579 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() Input: LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed
2022-03-02 08:17:39.582 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() Custom options specified.
2022-03-02 08:17:39.584 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() A custom codec is specified: org.atalk.impl.neomedia.audiolevel.AudioLevelEffect2@25eeb23
2022-03-02 08:17:39.588 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() An output format is specified: opus/rtp, 48000.0 Hz, Stereo
2022-03-02 08:17:39.591 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() An output content type is specified: raw.rtp
2022-03-02 08:17:39.644 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()     
2022-03-02 08:17:39.648 5702-8528/org.atalk.android D/AudioTrack: setVolume(1.000000, 1.000000) pid : 5702
2022-03-02 08:17:39.662 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.comment() Here's the completed flow graph:
2022-03-02 08:17:39.668 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()       com.sun.media.parser.RawPullBufferParser@e20f820
2022-03-02 08:17:39.670 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.audiolevel.AudioLevelEffect2@25eeb23
2022-03-02 08:17:39.673 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          format: LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed
2022-03-02 08:17:39.675 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.audiolevel.AudioLevelEffect2@25eeb23
2022-03-02 08:17:39.678 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.codec.audio.speex.SpeexResampler@86395d9
2022-03-02 08:17:39.680 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          format: LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed, FrameSize=16 bits
2022-03-02 08:17:39.683 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.codec.audio.speex.SpeexResampler@86395d9
2022-03-02 08:17:39.687 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.codec.audio.opus.JNIEncoder@8e51b9e
2022-03-02 08:17:39.690 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          format: LINEAR, 48000.0 Hz, 16-bit, Mono, LittleEndian, Signed
2022-03-02 08:17:39.692 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.codec.audio.opus.JNIEncoder@8e51b9e
2022-03-02 08:17:39.694 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          connects to: com.sun.media.multiplexer.RTPSyncBufferMux@22b167f
2022-03-02 08:17:39.697 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()          format: opus/rtp, 48000.0 Hz, Stereo
2022-03-02 08:17:39.699 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.write()     
2022-03-02 08:17:39.701 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.profile() Profile: graph building: 131 ms
2022-03-02 08:17:39.704 5702-8536/org.atalk.android D/aTalk: [754] net.sf.fmj.media.Log.profile() Profile: realize, post graph building: 1 ms


2022-03-02 08:17:39.923 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.comment() Building flow graph for: null
2022-03-02 08:17:39.932 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.comment() Building Track: 0
2022-03-02 08:17:39.941 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.comment() Input: VP8/RTP, fmtps={}
2022-03-02 08:17:39.974 5702-8577/org.atalk.android D/vpx_jni: VPX decode using: WebM Project VP8 Decoder v1.10.0
2022-03-02 08:17:39.974 5702-8577/org.atalk.android D/(VP8Decoder.java:124)#doOpen: VP8 decoder opened successfully
2022-03-02 08:17:39.991 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write() 
2022-03-02 08:17:40.000 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.comment() Here's the completed flow graph:
2022-03-02 08:17:40.009 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()   net.sf.fmj.media.parser.RawPushBufferParser@dfa18b
2022-03-02 08:17:40.016 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      connects to: org.atalk.impl.neomedia.codec.video.vp8.DePacketizer@ef25268
2022-03-02 08:17:40.024 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      format: VP8/RTP, fmtps={}
2022-03-02 08:17:40.031 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()   org.atalk.impl.neomedia.codec.video.vp8.DePacketizer@ef25268
2022-03-02 08:17:40.038 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      connects to: org.atalk.impl.neomedia.codec.video.vp8.VP8Decoder@6b8f681
2022-03-02 08:17:40.045 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      format: VP8
2022-03-02 08:17:40.050 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()   org.atalk.impl.neomedia.codec.video.vp8.VP8Decoder@6b8f681
2022-03-02 08:17:40.055 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      connects to: org.atalk.impl.neomedia.codec.video.SwScale@d6b6f05
2022-03-02 08:17:40.063 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      format: AVFRAME, class org.atalk.impl.neomedia.codec.video.AVFrame, pixFmt 0
2022-03-02 08:17:40.071 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()   org.atalk.impl.neomedia.codec.video.SwScale@d6b6f05
2022-03-02 08:17:40.082 5702-8560/org.atalk.android D/aTalk: [769] net.sf.fmj.media.Log.comment() RTP video buffer size: 36 pkts, 15934 bytes.
2022-03-02 08:17:40.088 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      connects to: org.atalk.impl.neomedia.jmfext.media.renderer.video.JAWTRenderer@56ffb5a
2022-03-02 08:17:40.094 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write()      format: RGB, 32-bit, Masks=255:65280:16711680, LineStride=-1, class [I
2022-03-02 08:17:40.100 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.write() 
2022-03-02 08:17:40.106 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.profile() Profile: graph building: 177 ms
2022-03-02 08:17:40.114 5702-8577/org.atalk.android D/aTalk: [783] net.sf.fmj.media.Log.profile() Profile: realize, post graph building: 0 ms

2022-03-02 08:17:40.329 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.comment() Building flow graph for: null
2022-03-02 08:17:40.333 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.comment() Building Track: 0
2022-03-02 08:17:40.362 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.comment() Input: YUV Video Format: Size = [width=1280, height=720] MaxDataLength = -1 DataType = class [B yuvType = -1 StrideY = -1 StrideUV = -1 OffsetY = -1 OffsetU = -1 OffsetV = -1
2022-03-02 08:17:40.368 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.comment() Custom options specified.
2022-03-02 08:17:40.372 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.comment() An output format is specified: VP8/RTP
2022-02-21 11:24:18.357 1402-2510/org.atalk.android D/aTalk: [404] net.sf.fmj.media.Log.comment() An output content type is specified: raw.rtp
2022-03-02 08:17:40.441 5702-8607/org.atalk.android D/(VP8Encoder.java:158)#doOpen: VP8 encode video size: [width=1280, height=720]
2022-03-02 08:17:40.442 5702-8607/org.atalk.android D/vpx_jni: VPX encode using: WebM Project VP8 Encoder v1.10.0
2022-03-02 08:17:40.455 5702-8607/org.atalk.android D/(VP8Encoder.java:200)#doOpen: VP8 encoder opened successfully
2022-03-02 08:17:40.480 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()       net.sf.fmj.media.parser.RawPushBufferParser@70d0fb0
2022-03-02 08:17:40.485 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.codec.video.vp8.VP8Encoder@d777e29
2022-03-02 08:17:40.492 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          format: YUV Video Format: Size = [width=1280, height=720] MaxDataLength = -1 DataType = class [B yuvType = -1 StrideY = -1 StrideUV = -1 OffsetY = -1 OffsetU = -1 OffsetV = -1
2022-03-02 08:17:40.497 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.codec.video.vp8.VP8Encoder@d777e29
2022-03-02 08:17:40.503 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          connects to: org.atalk.impl.neomedia.codec.video.vp8.Packetizer@10765ae
2022-03-02 08:17:40.511 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          format: VP8, 1280x720, FrameRate=2.0
2022-03-02 08:17:40.518 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()       org.atalk.impl.neomedia.codec.video.vp8.Packetizer@10765ae
2022-03-02 08:17:40.526 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          connects to: com.sun.media.multiplexer.RTPSyncBufferMux@3b57f4f
2022-03-02 08:17:40.530 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()          format: VP8/RTP
2022-03-02 08:17:40.534 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.write()     
2022-03-02 08:17:40.540 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.profile() Profile: graph building: 206 ms
2022-03-02 08:17:40.545 5702-8607/org.atalk.android D/aTalk: [796] net.sf.fmj.media.Log.profile() Profile: realize, post graph building: 0 ms

2022-03-02 08:17:47.475 5702-6135/org.atalk.android D/SMACK: SENT (0): 
    <iq to='peacock@atalk.sytes.net/atalk' id='7FUY4-18' type='set'>
      <jingle xmlns='urn:xmpp:jingle:1' action='session-terminate' sid='mfifd5cm6vh9'>
        <reason>
          <success/>
          <text>
            Nice talking to you!
          </text>
        </reason>
      </jingle>
    </iq>

@cmeng-git cmeng-git changed the base branch from master to 4.4 March 2, 2022 01:56
@ibauersachs
Copy link
Contributor

The licensing header on these files is not correct, there's a LOT of copy/paste from Jitsi. Apache-2.0 should be mostly fine, but (c) is 8x8.

Ping @damencho

@damencho
Copy link
Contributor

damencho commented Apr 2, 2022

Thanks for pinging me. I think that should keep the licenses from the project those are copied from ...
I need to ask legal department for advice how to proceed.

@cmeng-git
Copy link
Contributor Author

A bit of aTalk development history.

aTalk app development started way back in 2014, where the source was first forked from jitsi-android.
At that time, the jitsi-android is still an open source project where most of the class files headers contain the following if any.

/*
 * Jitsi, the OpenSource Java VoIP and Instant Messaging client.
 * 
 * Distributable under LGPL license. See terms of license at gnu.org.
 */

Later on in 2015 , the Jitsi project is taken over by
Copyright @ 2015 Atlassian Pty Ltd, and the latest I believe is 8x8
So I am not sure, under this case who is the rightful owner of the source that were forked back in 2014.

In all the released sources that are in the pull request, some of the classes are new and developed by aTalk developer; where they have used "Copyright xxxx Jive Software" as header statement, and the @author carries the developer(s) .

The other jingle classes in the pull request, that perform similar functions as the original jitsi-android, all have been rewritten as to confirm to smack standard implementation using builder approach i.e. java class that extends AbstractXmlElement. I am no sure whether 8x8 can claim as the rightful owner of these files. The situation may be similar to FMJ and JFM packages that implements same/similar functions.

@cmeng-git
Copy link
Contributor Author

cmeng-git commented Apr 3, 2022

I have done a local build on the latest pull-in sources. the whole build was OK, except in Checkstyle with errors in the import statements e.g.:

'org.jxmpp.jid.Jid' should be separated from previous import group by one line.
Import statement for 'javax.xml.namespace.QName' is in the wrong order. Should be in the 'STANDARD_JAVA_PACKAGE' group, expecting not assigned imports on this line.

The source are code formatted by android studio; user has not control or change settings to remove the above errors.
I also observed the following in the current smart class file header when doing the code format.
Again there is no settings to prevent this.

/** ### Dagling Javadoc comment (can only use i.e. /*) 
 * ### This line will always get removed after code format 
 * Copyright 2017-2022 Jive Software
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *  ### code format always replace multiple spaces with a single space in the following line
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */ 

In aTalk code format, it always use "next line" option for braces placement in
a. class
b. method
c. else

I found the above code format make the source easier for code review. No sure if Smack will consider this as well.
I see that at time smack source inserted a blank line immediately after class { and method { statements.

public class ThisIsASampleClass extends C1 implements I1, I2, I3, I4, I5
{  ### new line
    private int f1 = 1;
    private String field2 = "";

    public void foo1(int i1, int i2, int i3, int i4, int i5, int i6, int i7)
   {  ### new line
       // do something
    }

        if (2 < 3) {
            return;
        }  ### new line
        else {
          if (2 > 3)
             // do something
             return;
        }
}

Currently with the above deviations, there are a lot of manual works need to performed when raising a pull request for smack repository.

@damencho
Copy link
Contributor

damencho commented Apr 4, 2022

A bit of aTalk development history.

aTalk app development started way back in 2014, where the source was first forked from jitsi-android. At that time, the jitsi-android is still an open source project where most of the class files headers contain the following if any.

/*
 * Jitsi, the OpenSource Java VoIP and Instant Messaging client.
 * 
 * Distributable under LGPL license. See terms of license at gnu.org.
 */

Later on in 2015 , the Jitsi project is taken over by Copyright @ 2015 Atlassian Pty Ltd, and the latest I believe is 8x8 So I am not sure, under this case who is the rightful owner of the source that were forked back in 2014.

In all the released sources that are in the pull request, some of the classes are new and developed by aTalk developer; where they have used "Copyright xxxx Jive Software" as header statement, and the @author carries the developer(s) .

The other jingle classes in the pull request, that perform similar functions as the original jitsi-android, all have been rewritten as to confirm to smack standard implementation using builder approach i.e. java class that extends AbstractXmlElement. I am no sure whether 8x8 can claim as the rightful owner of these files. The situation may be similar to FMJ and JFM packages that implements same/similar functions.

I'm not a lawyer, but what I know is that you cannot change the license headers of the code, especially for the LGPL code. So it is up to the Smack developers to decide.

@cmeng-git
Copy link
Contributor Author

FYI: All the other jingle classes in jingle_rtp package that perform similar functions as the original jitsi-android, have been rewritten to confirm to smack standard implementation using builder approach. The code structure implementation made reference to smack source more than to jitsi.

The situation is similar to FMJ and JFM packages that implement same/similar functions. In fact all Java Documents for FMJ refer to JFM; and JFM has not copyright over FMJ.

@Flowdalic
Copy link
Member

I see two challenges with getting this into Smack as of now.

First, Jingle is a very complex topic, and most Jingle implementations I've seen so far where subpar. They certainly "get the job done" but lack proper error handling and reporting, extensibility, and clean code (not that Smack is much better in some areas here). Ultimately, this leads to them becoming a maintenance burden, if not nightmare.

The second issue is that unclear code pedigree. License regulations have to be obeyed and copyright statements have to be retained. Note that I am not saying that there are actual issues with the code in this PR. It is just that concerns have been raised, and I take licenses seriously and have not been able to find time to review the stated issues.

In general, I am quite busy for the foreseeable future, and while I am certain that I will be in the future able again to spend some more of my spare time on open source projects, now is not the right time for me to take over, or even thoroughly review, the proposed code.

But I think the situation is not so bleak as one may think. It would be perfectly fine, if not even advantageous, if we cultivate the Jingle code in an extra repository, and, in turn, library which is based on Smack. I think we could even get the Jingle Models, i.e., the IQ and ExtensionElement classes and their according Providers, into Smack. Especially considered that there is a large infrastructure of those already in Smack. While we let the Jingle logic, i.e., the Managers mature in that extra repo. I am also happy redirect users to this "Smack Jingle" repository by placing prominent notices in Smack's README and on Smack's homepage.

@cmeng-git
Copy link
Contributor Author

I strong agree JIngle implementation is a complex task. I have reservation aTalk pull request will fully accepted by smack team immediately; but hopefully give someone a reference base, and make enhancements when doing the final smack implementation; meantime allowing other xmpp client developers a reference source for his earlier jingle features implementation.

I feel the smack team has done an excellent job in providing the improved smack-jingle library even at this stage. aTalk heavily depends on smack libraries including the new Jingle support, these have helped in extending and accelerating aTalk app features implementation.

To my best knowledge in interpretation of the license clauses as a layman, I certainly respect copyrighted material and source contributors. If your team found otherwise in my source releases, I would more than happy to make the necessary changes.

You mentioned the "Smack Jingle" repository, is this in existence at the moment. Certainly this will help the xmpp community to share the work done; and later get included in the main smack library when it is matured. JIngle plays an importance role for xmpp client implementation. XMPP client implementation without jingle support, does limit its main function to be a text only communicator.

@Flowdalic
Copy link
Member

but hopefully give someone a reference base, and make enhancements when doing the final smack implementation; meantime allowing other xmpp client developers a reference source for his earlier jingle features implementation.

Thanks, that line of though is certainly sensible.

I feel the smack team has done an excellent job

Thanks again. :)

You mentioned the "Smack Jingle" repository, is this in existence at the moment?

No. This was a subliminal hint in an attempt to encourage you to create such a repository.

@iachimoe
Copy link
Contributor

iachimoe commented Jul 4, 2022

This looks super interesting! Curious as to whether there are plans to progress it further?

@Neustradamus
Copy link
Contributor

What is the progress about it?

@cmeng-git
Copy link
Contributor Author

The PR now contains all the souces to support rtp for media call, jingle file transfer with JET. The source is exact match 99% with aTalk, and is fully tested on aTalk.

@Neustradamus
Copy link
Contributor

@vanitasvitae, @gkfabs: Have you seen?
What do you think?

@guusdk
Copy link
Member

guusdk commented Feb 24, 2023

@Neustradamus - stop mentioning people in comments just to get their attention. It is incredibly annoying. I believe you've been asked to stop doing that repeatedly.

@guusdk
Copy link
Member

guusdk commented Feb 24, 2023

For what it's worth, I appreciate the enormous efforts that have been put into this implementation by the author. However, I strongly agree with Flowdalic's statement that reads:

License regulations have to be obeyed and copyright statements have to be retained.

While there are concerns regarding this, we should avoid merging these changes.

Sadly, reviewing issues like these is a time-consuming effort - one that few of us have much experience with. @Flowdalic if it helps, and as this wades into the territory of 'legalese', we can find out if the members of the board of the Ignite Realtime Foundation can help with this effort.

@cmeng-git
Copy link
Contributor Author

Copyright controversy is only related to source in jingle_rtp directory. Most of the other sources are actually from the smack team members. aTalk just ported the sources and made enhancements, so they are compatible with smack version 4.4, and work reliability for application integration. May be smack team can consider port in these source first, and leave jingle_rtp out until thing are clear up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants