release 2016.07.13

[ellentv] Improve extraction (Closes #10067 )
[bbc] extract more and better qulities from Unified Streaming Platform m3u8 manifests
2016-07-13 23:58:01 +07:00 · 2016-07-13 22:42:53 +07:00 · 2016-07-13 15:58:24 +01:00 · 2016-07-13 15:58:24 +01:00 · 2016-07-13 21:53:23 +07:00 · 2016-07-13 21:52:52 +07:00
36 changed files with 902 additions and 416 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -6,8 +6,8 @@

 ---

-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.07.09*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.07.09**
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.07.13*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.07.13**

 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
@@ -35,7 +35,7 @@ $ youtube-dl -v <your command line>
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2016.07.09
+[debug] youtube-dl version 2016.07.13
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,9 +7,6 @@ python:
  - "3.4"
  - "3.5"
 sudo: false
-install:
-  - bash ./devscripts/install_srelay.sh
-  - export PATH=$PATH:$(pwd)/tmp/srelay-0.4.8b6
 script: nosetests test --verbose
 notifications:
  email:
--- a/1
+++ b/1
@@ -177,3 +177,4 @@ Roman Tsiupa
 Artur Krysiak
 Jakub Adam Wieczorek
 Aleksandar Topuzović
+Nehal Patel
--- a/devscripts/gh-pages/generate-download.py
+++ b/devscripts/gh-pages/generate-download.py
@@ -15,13 +15,9 @@ data = urllib.request.urlopen(URL).read()
 with open('download.html.in', 'r', encoding='utf-8') as tmplf:
    template = tmplf.read()

-md5sum = hashlib.md5(data).hexdigest()
-sha1sum = hashlib.sha1(data).hexdigest()
 sha256sum = hashlib.sha256(data).hexdigest()
 template = template.replace('@PROGRAM_VERSION@', version)
 template = template.replace('@PROGRAM_URL@', URL)
-template = template.replace('@PROGRAM_MD5SUM@', md5sum)
-template = template.replace('@PROGRAM_SHA1SUM@', sha1sum)
 template = template.replace('@PROGRAM_SHA256SUM@', sha256sum)
 template = template.replace('@EXE_URL@', versions_info['versions'][version]['exe'][0])
 template = template.replace('@EXE_SHA256SUM@', versions_info['versions'][version]['exe'][1])
--- a/devscripts/install_srelay.sh
+++ b/devscripts/install_srelay.sh
@@ -1,8 +0,0 @@
-#!/bin/bash
-
-mkdir -p tmp && cd tmp
-wget -N http://downloads.sourceforge.net/project/socks-relay/socks-relay/srelay-0.4.8/srelay-0.4.8b6.tar.gz
-tar zxvf srelay-0.4.8b6.tar.gz
-cd srelay-0.4.8b6
-./configure
-make
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@@ -554,6 +554,7 @@
 - **RICE**
 - **RingTV**
 - **RockstarGames**
+ - **RoosterTeeth**
 - **RottenTomatoes**
 - **Roxwel**
 - **RTBF**
@@ -567,6 +568,7 @@
 - **rtve.es:infantil**: RTVE infantil
 - **rtve.es:live**: RTVE.es live streams
 - **RTVNH**
+ - **Rudo**
 - **RUHD**
 - **RulePorn**
 - **rutube**: Rutube videos
@@ -793,6 +795,7 @@
 - **vine:user**
 - **vk**: VK
 - **vk:uservideos**: VK - User's Videos
+ - **vk:wallpost**
 - **vlive**
 - **Vodlocker**
 - **VoiceRepublic**
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -81,6 +81,7 @@ from youtube_dl.utils import (
    cli_option,
    cli_valueless_option,
    cli_bool_option,
+    parse_codecs,
 )
 from youtube_dl.compat import (
    compat_chr,
@@ -608,6 +609,29 @@ class TestUtil(unittest.TestCase):
            limit_length('foo bar baz asd', 12).startswith('foo bar'))
        self.assertTrue('...' in limit_length('foo bar baz asd', 12))

+    def test_parse_codecs(self):
+        self.assertEqual(parse_codecs(''), {})
+        self.assertEqual(parse_codecs('avc1.77.30, mp4a.40.2'), {
+            'vcodec': 'avc1.77.30',
+            'acodec': 'mp4a.40.2',
+        })
+        self.assertEqual(parse_codecs('mp4a.40.2'), {
+            'vcodec': 'none',
+            'acodec': 'mp4a.40.2',
+        })
+        self.assertEqual(parse_codecs('mp4a.40.5,avc1.42001e'), {
+            'vcodec': 'avc1.42001e',
+            'acodec': 'mp4a.40.5',
+        })
+        self.assertEqual(parse_codecs('avc3.640028'), {
+            'vcodec': 'avc3.640028',
+            'acodec': 'none',
+        })
+        self.assertEqual(parse_codecs(', h264,,newcodec,aac'), {
+            'vcodec': 'h264',
+            'acodec': 'aac',
+        })
+
    def test_escape_rfc3986(self):
        reserved = "!*'();:@&=+$,/?#[]"
        unreserved = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~'
--- a/youtube_dl/extractor/animeondemand.py
+++ b/youtube_dl/extractor/animeondemand.py
@@ -203,7 +203,7 @@ class AnimeOnDemandIE(InfoExtractor):
                'formats': formats,
            }

-        def extract_entries(html, video_id, common_info, num):
+        def extract_entries(html, video_id, common_info, num=None):
            info = extract_info(html, video_id, num)

            if info['formats']:
--- a/youtube_dl/extractor/ard.py
+++ b/youtube_dl/extractor/ard.py
@@ -13,6 +13,7 @@ from ..utils import (
    parse_duration,
    unified_strdate,
    xpath_text,
+    update_url_query,
 )
 from ..compat import compat_etree_fromstring

@@ -34,6 +35,7 @@ class ARDMediathekIE(InfoExtractor):
            # m3u8 download
            'skip_download': True,
        },
+        'skip': 'HTTP Error 404: Not Found',
    }, {
        'url': 'http://www.ardmediathek.de/tv/Tatort/Tatort-Scheinwelten-H%C3%B6rfassung-Video/Das-Erste/Video?documentId=29522730&bcastId=602916',
        'md5': 'f4d98b10759ac06c0072bbcd1f0b9e3e',
@@ -44,6 +46,7 @@ class ARDMediathekIE(InfoExtractor):
            'description': 'md5:196392e79876d0ac94c94e8cdb2875f1',
            'duration': 5252,
        },
+        'skip': 'HTTP Error 404: Not Found',
    }, {
        # audio
        'url': 'http://www.ardmediathek.de/tv/WDR-H%C3%B6rspiel-Speicher/Tod-eines-Fu%C3%9Fballers/WDR-3/Audio-Podcast?documentId=28488308&bcastId=23074086',
@@ -55,6 +58,7 @@ class ARDMediathekIE(InfoExtractor):
            'description': 'md5:f6e39f3461f0e1f54bfa48c8875c86ef',
            'duration': 3240,
        },
+        'skip': 'HTTP Error 404: Not Found',
    }, {
        'url': 'http://mediathek.daserste.de/sendungen_a-z/328454_anne-will/22429276_vertrauen-ist-gut-spionieren-ist-besser-geht',
        'only_matching': True,
@@ -113,11 +117,14 @@ class ARDMediathekIE(InfoExtractor):
                        continue
                    if ext == 'f4m':
                        formats.extend(self._extract_f4m_formats(
-                            stream_url + '?hdcore=3.1.1&plugin=aasp-3.1.1.69.124',
-                            video_id, preference=-1, f4m_id='hds', fatal=False))
+                            update_url_query(stream_url, {
+                                'hdcore': '3.1.1',
+                                'plugin': 'aasp-3.1.1.69.124'
+                            }),
+                            video_id, f4m_id='hds', fatal=False))
                    elif ext == 'm3u8':
                        formats.extend(self._extract_m3u8_formats(
-                            stream_url, video_id, 'mp4', preference=1, m3u8_id='hls', fatal=False))
+                            stream_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
                    else:
                        if server and server.startswith('rtmp'):
                            f = {
@@ -231,7 +238,8 @@ class ARDIE(InfoExtractor):
            'title': 'Die Story im Ersten: Mission unter falscher Flagge',
            'upload_date': '20140804',
            'thumbnail': 're:^https?://.*\.jpg$',
-        }
+        },
+        'skip': 'HTTP Error 404: Not Found',
    }

    def _real_extract(self, url):
--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@@ -44,6 +44,8 @@ class BBCCoUkIE(InfoExtractor):

    _MEDIASELECTION_NS = 'http://bbc.co.uk/2008/mp/mediaselection'
    _EMP_PLAYLIST_NS = 'http://bbc.co.uk/2008/emp/playlist'
+    # Unified Streaming Platform
+    _USP_RE = r'/([^/]+)\.ism(?:\.hlsv2\.ism)?/[^/]+\.m3u8'

    _NAMESPACES = (
        _MEDIASELECTION_NS,
@@ -55,12 +57,11 @@ class BBCCoUkIE(InfoExtractor):
            'url': 'http://www.bbc.co.uk/programmes/b039g8p7',
            'info_dict': {
                'id': 'b039d07m',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Leonard Cohen, Kaleidoscope - BBC Radio 4',
                'description': 'The Canadian poet and songwriter reflects on his musical career.',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            }
        },
@@ -92,7 +93,7 @@ class BBCCoUkIE(InfoExtractor):
                # rtmp download
                'skip_download': True,
            },
-            'skip': 'Currently BBC iPlayer TV programmes are available to play in the UK only',
+            'skip': 'this episode is not currently available',
        },
        {
            'url': 'http://www.bbc.co.uk/iplayer/episode/p026c7jt/tomorrows-worlds-the-unearthly-history-of-science-fiction-2-invasion',
@@ -107,7 +108,7 @@ class BBCCoUkIE(InfoExtractor):
                # rtmp download
                'skip_download': True,
            },
-            'skip': 'Currently BBC iPlayer TV programmes are available to play in the UK only',
+            'skip': 'this episode is not currently available',
        }, {
            'url': 'http://www.bbc.co.uk/programmes/b04v20dw',
            'info_dict': {
@@ -127,13 +128,12 @@ class BBCCoUkIE(InfoExtractor):
            'note': 'Audio',
            'info_dict': {
                'id': 'p022h44j',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'BBC Proms Music Guides, Rachmaninov: Symphonic Dances',
                'description': "In this Proms Music Guide, Andrew McGregor looks at Rachmaninov's Symphonic Dances.",
                'duration': 227,
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            }
        }, {
@@ -141,13 +141,12 @@ class BBCCoUkIE(InfoExtractor):
            'note': 'Video',
            'info_dict': {
                'id': 'p025c103',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Reading and Leeds Festival, 2014, Rae Morris - Closer (Live on BBC Three)',
                'description': 'Rae Morris performs Closer for BBC Three at Reading 2014',
                'duration': 226,
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            }
        }, {
@@ -163,7 +162,7 @@ class BBCCoUkIE(InfoExtractor):
                # rtmp download
                'skip_download': True,
            },
-            'skip': 'geolocation',
+            'skip': 'this episode is not currently available',
        }, {
            'url': 'http://www.bbc.co.uk/iplayer/episode/b05zmgwn/royal-academy-summer-exhibition',
            'info_dict': {
@@ -177,7 +176,7 @@ class BBCCoUkIE(InfoExtractor):
                # rtmp download
                'skip_download': True,
            },
-            'skip': 'geolocation',
+            'skip': 'this episode is not currently available',
        }, {
            # iptv-all mediaset fails with geolocation however there is no geo restriction
            # for this programme at all
@@ -192,18 +191,17 @@ class BBCCoUkIE(InfoExtractor):
                # rtmp download
                'skip_download': True,
            },
-            'skip': 'Now it\'s really geo-restricted',
+            'skip': 'this episode is not currently available on BBC iPlayer Radio',
        }, {
            # compact player (https://github.com/rg3/youtube-dl/issues/8147)
            'url': 'http://www.bbc.co.uk/programmes/p028bfkf/player',
            'info_dict': {
                'id': 'p028bfkj',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Extract from BBC documentary Look Stranger - Giant Leeks and Magic Brews',
                'description': 'Extract from BBC documentary Look Stranger - Giant Leeks and Magic Brews',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        }, {
@@ -248,9 +246,15 @@ class BBCCoUkIE(InfoExtractor):
            elif transfer_format == 'dash':
                pass
            elif transfer_format == 'hls':
-                formats.extend(self._extract_m3u8_formats(
+                is_unified_streaming = re.search(self._USP_RE, href)
+                if is_unified_streaming:
+                    href = re.sub(self._USP_RE, r'/\1.ism/\1.m3u8', href)
+                m3u8_formats = self._extract_m3u8_formats(
                    href, programme_id, ext='mp4', entry_protocol='m3u8_native',
-                    m3u8_id=supplier, fatal=False))
+                    m3u8_id=supplier, fatal=False)
+                if is_unified_streaming:
+                    self._check_formats(m3u8_formats, programme_id)
+                formats.extend(m3u8_formats)
            # Direct link
            else:
                formats.append({
@@ -305,13 +309,14 @@ class BBCCoUkIE(InfoExtractor):
        for connection in self._extract_connections(media):
            conn_formats = self._extract_connection(connection, programme_id)
            for format in conn_formats:
-                format.update({
-                    'width': width,
-                    'height': height,
-                    'vbr': vbr,
-                    'vcodec': vcodec,
-                    'filesize': file_size,
-                })
+                if format.get('protocol') != 'm3u8_native':
+                    format.update({
+                        'width': width,
+                        'height': height,
+                        'vbr': vbr,
+                        'vcodec': vcodec,
+                        'filesize': file_size,
+                    })
                if service:
                    format['format_id'] = '%s_%s' % (service, format['format_id'])
            formats.extend(conn_formats)
--- a/youtube_dl/extractor/biobiochiletv.py
+++ b/youtube_dl/extractor/biobiochiletv.py
@@ -2,11 +2,15 @@
 from __future__ import unicode_literals

 from .common import InfoExtractor
-from ..utils import remove_end
+from ..utils import (
+    ExtractorError,
+    remove_end,
+)
+from .rudo import RudoIE


 class BioBioChileTVIE(InfoExtractor):
-    _VALID_URL = r'https?://tv\.biobiochile\.cl/notas/(?:[^/]+/)+(?P<id>[^/]+)\.shtml'
+    _VALID_URL = r'https?://(?:tv|www)\.biobiochile\.cl/(?:notas|noticias)/(?:[^/]+/)+(?P<id>[^/]+)\.shtml'

    _TESTS = [{
        'url': 'http://tv.biobiochile.cl/notas/2015/10/21/sobre-camaras-y-camarillas-parlamentarias.shtml',
@@ -18,6 +22,7 @@ class BioBioChileTVIE(InfoExtractor):
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': 'Fernando Atria',
        },
+        'skip': 'URL expired and redirected to http://www.biobiochile.cl/portada/bbtv/index.html',
    }, {
        # different uploader layout
        'url': 'http://tv.biobiochile.cl/notas/2016/03/18/natalia-valdebenito-repasa-a-diputado-hasbun-paso-a-la-categoria-de-hablar-brutalidades.shtml',
@@ -32,6 +37,16 @@ class BioBioChileTVIE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
+        'skip': 'URL expired and redirected to http://www.biobiochile.cl/portada/bbtv/index.html',
+    }, {
+        'url': 'http://www.biobiochile.cl/noticias/bbtv/comentarios-bio-bio/2016/07/08/edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos.shtml',
+        'info_dict': {
+            'id': 'edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos',
+            'ext': 'mp4',
+            'uploader': '(none)',
+            'upload_date': '20160708',
+            'title': 'Edecanes del Congreso: Figuras decorativas que le cuestan muy caro a los chilenos',
+        },
    }, {
        'url': 'http://tv.biobiochile.cl/notas/2015/10/22/ninos-transexuales-de-quien-es-la-decision.shtml',
        'only_matching': True,
@@ -45,42 +60,22 @@ class BioBioChileTVIE(InfoExtractor):

        webpage = self._download_webpage(url, video_id)

+        rudo_url = RudoIE._extract_url(webpage)
+        if not rudo_url:
+            raise ExtractorError('No videos found')
+
        title = remove_end(self._og_search_title(webpage), ' - BioBioChile TV')

-        file_url = self._search_regex(
-            r'loadFWPlayerVideo\([^,]+,\s*(["\'])(?P<url>.+?)\1',
-            webpage, 'file url', group='url')
-
-        base_url = self._search_regex(
-            r'file\s*:\s*(["\'])(?P<url>.+?)\1\s*\+\s*fileURL', webpage,
-            'base url', default='http://unlimited2-cl.digitalproserver.com/bbtv/',
-            group='url')
-
-        formats = self._extract_m3u8_formats(
-            '%s%s/playlist.m3u8' % (base_url, file_url), video_id, 'mp4',
-            entry_protocol='m3u8_native', m3u8_id='hls', fatal=False)
-        f = {
-            'url': '%s%s' % (base_url, file_url),
-            'format_id': 'http',
-            'protocol': 'http',
-            'preference': 1,
-        }
-        if formats:
-            f_copy = formats[-1].copy()
-            f_copy.update(f)
-            f = f_copy
-        formats.append(f)
-        self._sort_formats(formats)
-
        thumbnail = self._og_search_thumbnail(webpage)
        uploader = self._html_search_regex(
-            r'<a[^>]+href=["\']https?://busca\.biobiochile\.cl/author[^>]+>(.+?)</a>',
+            r'<a[^>]+href=["\']https?://(?:busca|www)\.biobiochile\.cl/(?:lista/)?(?:author|autor)[^>]+>(.+?)</a>',
            webpage, 'uploader', fatal=False)

        return {
+            '_type': 'url_transparent',
+            'url': rudo_url,
            'id': video_id,
            'title': title,
            'thumbnail': thumbnail,
            'uploader': uploader,
-            'formats': formats,
        }
--- a/youtube_dl/extractor/brightcove.py
+++ b/youtube_dl/extractor/brightcove.py
@@ -26,6 +26,7 @@ from ..utils import (
    unescapeHTML,
    unsmuggle_url,
    update_url_query,
+    clean_html,
 )


@@ -620,7 +621,7 @@ class BrightcoveNewIE(InfoExtractor):
        return {
            'id': video_id,
            'title': title,
-            'description': json_data.get('description'),
+            'description': clean_html(json_data.get('description')),
            'thumbnail': json_data.get('thumbnail') or json_data.get('poster'),
            'duration': float_or_none(json_data.get('duration'), 1000),
            'timestamp': parse_iso8601(json_data.get('published_at')),
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -55,6 +55,8 @@ from ..utils import (
    update_Request,
    update_url_query,
    parse_m3u8_attributes,
+    extract_attributes,
+    parse_codecs,
 )


@@ -1205,6 +1207,7 @@ class InfoExtractor(object):
                    'url': format_url(line.strip()),
                    'tbr': tbr,
                    'ext': ext,
+                    'fps': float_or_none(last_info.get('FRAME-RATE')),
                    'protocol': entry_protocol,
                    'preference': preference,
                }
@@ -1213,24 +1216,17 @@ class InfoExtractor(object):
                    width_str, height_str = resolution.split('x')
                    f['width'] = int(width_str)
                    f['height'] = int(height_str)
-                codecs = last_info.get('CODECS')
-                if codecs:
-                    vcodec, acodec = [None] * 2
-                    va_codecs = codecs.split(',')
-                    if len(va_codecs) == 1:
-                        # Audio only entries usually come with single codec and
-                        # no resolution. For more robustness we also check it to
-                        # be mp4 audio.
-                        if not resolution and va_codecs[0].startswith('mp4a'):
-                            vcodec, acodec = 'none', va_codecs[0]
-                        else:
-                            vcodec = va_codecs[0]
-                    else:
-                        vcodec, acodec = va_codecs[:2]
+                # Unified Streaming Platform
+                mobj = re.search(
+                    r'audio.*?(?:%3D|=)(\d+)(?:-video.*?(?:%3D|=)(\d+))?', f['url'])
+                if mobj:
+                    abr, vbr = mobj.groups()
+                    abr, vbr = float_or_none(abr, 1000), float_or_none(vbr, 1000)
                    f.update({
-                        'acodec': acodec,
-                        'vcodec': vcodec,
+                        'vbr': vbr,
+                        'abr': abr,
                    })
+                f.update(parse_codecs(last_info.get('CODECS')))
                if last_media is not None:
                    f['m3u8_media'] = last_media
                    last_media = None
@@ -1635,6 +1631,62 @@ class InfoExtractor(object):
                        self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
        return formats

+    def _parse_html5_media_entries(self, base_url, webpage):
+        def absolute_url(video_url):
+            return compat_urlparse.urljoin(base_url, video_url)
+
+        def parse_content_type(content_type):
+            if not content_type:
+                return {}
+            ctr = re.search(r'(?P<mimetype>[^/]+/[^;]+)(?:;\s*codecs="?(?P<codecs>[^"]+))?', content_type)
+            if ctr:
+                mimetype, codecs = ctr.groups()
+                f = parse_codecs(codecs)
+                f['ext'] = mimetype2ext(mimetype)
+                return f
+            return {}
+
+        entries = []
+        for media_tag, media_type, media_content in re.findall(r'(?s)(<(?P<tag>video|audio)[^>]*>)(.*?)</(?P=tag)>', webpage):
+            media_info = {
+                'formats': [],
+                'subtitles': {},
+            }
+            media_attributes = extract_attributes(media_tag)
+            src = media_attributes.get('src')
+            if src:
+                media_info['formats'].append({
+                    'url': absolute_url(src),
+                    'vcodec': 'none' if media_type == 'audio' else None,
+                })
+            media_info['thumbnail'] = media_attributes.get('poster')
+            if media_content:
+                for source_tag in re.findall(r'<source[^>]+>', media_content):
+                    source_attributes = extract_attributes(source_tag)
+                    src = source_attributes.get('src')
+                    if not src:
+                        continue
+                    f = parse_content_type(source_attributes.get('type'))
+                    f.update({
+                        'url': absolute_url(src),
+                        'vcodec': 'none' if media_type == 'audio' else None,
+                    })
+                    media_info['formats'].append(f)
+                for track_tag in re.findall(r'<track[^>]+>', media_content):
+                    track_attributes = extract_attributes(track_tag)
+                    kind = track_attributes.get('kind')
+                    if not kind or kind == 'subtitles':
+                        src = track_attributes.get('src')
+                        if not src:
+                            continue
+                        lang = track_attributes.get('srclang') or track_attributes.get('lang') or track_attributes.get('label')
+                        media_info['subtitles'].setdefault(lang, []).append({
+                            'url': absolute_url(src),
+                        })
+            if media_info['formats']:
+                entries.append(media_info)
+        return entries
+
    def _live_title(self, name):
        """ Generate the title for a live video """
        now = datetime.datetime.now()
--- a/youtube_dl/extractor/dbtv.py
+++ b/youtube_dl/extractor/dbtv.py
@@ -4,78 +4,47 @@ from __future__ import unicode_literals
 import re

 from .common import InfoExtractor
-from ..compat import compat_str
-from ..utils import (
-    float_or_none,
-    int_or_none,
-    clean_html,
-)


 class DBTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?dbtv\.no/(?:(?:lazyplayer|player)/)?(?P<id>[0-9]+)(?:#(?P<display_id>.+))?'
+    _VALID_URL = r'https?://(?:www\.)?dbtv\.no/(?:[^/]+/)?(?P<id>[0-9]+)(?:#(?P<display_id>.+))?'
    _TESTS = [{
        'url': 'http://dbtv.no/3649835190001#Skulle_teste_ut_fornøyelsespark,_men_kollegaen_var_bare_opptatt_av_bikinikroppen',
-        'md5': 'b89953ed25dacb6edb3ef6c6f430f8bc',
+        'md5': '2e24f67936517b143a234b4cadf792ec',
        'info_dict': {
-            'id': '33100',
+            'id': '3649835190001',
            'display_id': 'Skulle_teste_ut_fornøyelsespark,_men_kollegaen_var_bare_opptatt_av_bikinikroppen',
            'ext': 'mp4',
            'title': 'Skulle teste ut fornøyelsespark, men kollegaen var bare opptatt av bikinikroppen',
            'description': 'md5:1504a54606c4dde3e4e61fc97aa857e0',
-            'thumbnail': 're:https?://.*\.jpg$',
-            'timestamp': 1404039863.438,
+            'thumbnail': 're:https?://.*\.jpg',
+            'timestamp': 1404039863,
            'upload_date': '20140629',
            'duration': 69.544,
-            'view_count': int,
-            'categories': list,
-        }
+            'uploader_id': '1027729757001',
+        },
+        'add_ie': ['BrightcoveNew']
    }, {
        'url': 'http://dbtv.no/3649835190001',
        'only_matching': True,
    }, {
        'url': 'http://www.dbtv.no/lazyplayer/4631135248001',
        'only_matching': True,
+    }, {
+        'url': 'http://dbtv.no/vice/5000634109001',
+        'only_matching': True,
+    }, {
+        'url': 'http://dbtv.no/filmtrailer/3359293614001',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
-        display_id = mobj.group('display_id') or video_id
-
-        data = self._download_json(
-            'http://api.dbtv.no/discovery/%s' % video_id, display_id)
-
-        video = data['playlist'][0]
-
-        formats = [{
-            'url': f['URL'],
-            'vcodec': f.get('container'),
-            'width': int_or_none(f.get('width')),
-            'height': int_or_none(f.get('height')),
-            'vbr': float_or_none(f.get('rate'), 1000),
-            'filesize': int_or_none(f.get('size')),
-        } for f in video['renditions'] if 'URL' in f]
-
-        if not formats:
-            for url_key, format_id in [('URL', 'mp4'), ('HLSURL', 'hls')]:
-                if url_key in video:
-                    formats.append({
-                        'url': video[url_key],
-                        'format_id': format_id,
-                    })
-
-        self._sort_formats(formats)
+        video_id, display_id = re.match(self._VALID_URL, url).groups()

        return {
-            'id': compat_str(video['id']),
+            '_type': 'url_transparent',
+            'url': 'http://players.brightcove.net/1027729757001/default_default/index.html?videoId=%s' % video_id,
+            'id': video_id,
            'display_id': display_id,
-            'title': video['title'],
-            'description': clean_html(video['desc']),
-            'thumbnail': video.get('splash') or video.get('thumb'),
-            'timestamp': float_or_none(video.get('publishedAt'), 1000),
-            'duration': float_or_none(video.get('length'), 1000),
-            'view_count': int_or_none(video.get('views')),
-            'categories': video.get('tags'),
-            'formats': formats,
+            'ie_key': 'BrightcoveNew',
        }
--- a/youtube_dl/extractor/ellentv.py
+++ b/youtube_dl/extractor/ellentv.py
@@ -6,12 +6,13 @@ import json
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
+    NO_DEFAULT,
 )


 class EllenTVIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?(?:ellentv|ellentube)\.com/videos/(?P<id>[a-z0-9_-]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://www.ellentv.com/videos/0-ipq1gsai/',
        'md5': '4294cf98bc165f218aaa0b89e0fd8042',
        'info_dict': {
@@ -22,24 +23,47 @@ class EllenTVIE(InfoExtractor):
            'timestamp': 1428035648,
            'upload_date': '20150403',
            'uploader_id': 'batchUser',
-        }
-    }
+        },
+    }, {
+        # not available via http://widgets.ellentube.com/
+        'url': 'http://www.ellentv.com/videos/1-szkgu2m2/',
+        'info_dict': {
+            'id': '1_szkgu2m2',
+            'ext': 'flv',
+            'title': "Ellen's Amazingly Talented Audience",
+            'description': 'md5:86ff1e376ff0d717d7171590e273f0a5',
+            'timestamp': 1255140900,
+            'upload_date': '20091010',
+            'uploader_id': 'ellenkaltura@gmail.com',
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }]

    def _real_extract(self, url):
        video_id = self._match_id(url)

-        webpage = self._download_webpage(
-            'http://widgets.ellentube.com/videos/%s' % video_id,
-            video_id)
+        URLS = ('http://widgets.ellentube.com/videos/%s' % video_id, url)

-        partner_id = self._search_regex(
-            r"var\s+partnerId\s*=\s*'([^']+)", webpage, 'partner id')
+        for num, url_ in enumerate(URLS, 1):
+            webpage = self._download_webpage(
+                url_, video_id, fatal=num == len(URLS))

-        kaltura_id = self._search_regex(
-            [r'id="kaltura_player_([^"]+)"',
-             r"_wb_entry_id\s*:\s*'([^']+)",
-             r'data-kaltura-entry-id="([^"]+)'],
-            webpage, 'kaltura id')
+            default = NO_DEFAULT if num == len(URLS) else None
+
+            partner_id = self._search_regex(
+                r"var\s+partnerId\s*=\s*'([^']+)", webpage, 'partner id',
+                default=default)
+
+            kaltura_id = self._search_regex(
+                [r'id="kaltura_player_([^"]+)"',
+                 r"_wb_entry_id\s*:\s*'([^']+)",
+                 r'data-kaltura-entry-id="([^"]+)'],
+                webpage, 'kaltura id', default=default)
+
+            if partner_id and kaltura_id:
+                break

        return self.url_result('kaltura:%s:%s' % (partner_id, kaltura_id), 'Kaltura')

--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@@ -680,6 +680,7 @@ from .rice import RICEIE
 from .ringtv import RingTVIE
 from .ro220 import Ro220IE
 from .rockstargames import RockstarGamesIE
+from .roosterteeth import RoosterTeethIE
 from .rottentomatoes import RottenTomatoesIE
 from .roxwel import RoxwelIE
 from .rtbf import RTBFIE
@@ -690,6 +691,7 @@ from .rtp import RTPIE
 from .rts import RTSIE
 from .rtve import RTVEALaCartaIE, RTVELiveIE, RTVEInfantilIE
 from .rtvnh import RTVNHIE
+from .rudo import RudoIE
 from .ruhd import RUHDIE
 from .ruleporn import RulePornIE
 from .rutube import (
@@ -987,6 +989,7 @@ from .viki import (
 from .vk import (
    VKIE,
    VKUserVideosIE,
+    VKWallPostIE,
 )
 from .vlive import VLiveIE
 from .vodlocker import VodlockerIE
--- a/youtube_dl/extractor/facebook.py
+++ b/youtube_dl/extractor/facebook.py
@@ -231,11 +231,13 @@ class FacebookIE(InfoExtractor):
                    continue
                if isinstance(f, dict):
                    f = [f]
-                if isinstance(f, list):
+                if not isinstance(f, list):
                    continue
                if f[0].get('video_id') == video_id:
                    video_data = video_data_candidate
                    break
+            if video_data:
+                break

        def video_data_list2dict(video_data):
            ret = {}
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@@ -1314,22 +1314,37 @@ class GenericIE(InfoExtractor):
            'add_ie': ['Kaltura'],
        },
        {
-            # TODO: find another test
-            # http://schema.org/VideoObject
-            # 'url': 'https://flipagram.com/f/nyvTSJMKId',
-            # 'md5': '888dcf08b7ea671381f00fab74692755',
-            # 'info_dict': {
-            #     'id': 'nyvTSJMKId',
-            #     'ext': 'mp4',
-            #     'title': 'Flipagram by sjuria101 featuring Midnight Memories by One Direction',
-            #     'description': '#love for cats.',
-            #     'timestamp': 1461244995,
-            #     'upload_date': '20160421',
-            # },
-            # 'params': {
-            #     'force_generic_extractor': True,
-            # },
-        }
+            # Non-standard Vimeo embed
+            'url': 'https://openclassrooms.com/courses/understanding-the-web',
+            'md5': '64d86f1c7d369afd9a78b38cbb88d80a',
+            'info_dict': {
+                'id': '148867247',
+                'ext': 'mp4',
+                'title': 'Understanding the web - Teaser',
+                'description': 'This is "Understanding the web - Teaser" by openclassrooms on Vimeo, the home for high quality videos and the people who love them.',
+                'upload_date': '20151214',
+                'uploader': 'OpenClassrooms',
+                'uploader_id': 'openclassrooms',
+            },
+            'add_ie': ['Vimeo'],
+        },
+        # {
+        #     # TODO: find another test
+        #     # http://schema.org/VideoObject
+        #     'url': 'https://flipagram.com/f/nyvTSJMKId',
+        #     'md5': '888dcf08b7ea671381f00fab74692755',
+        #     'info_dict': {
+        #         'id': 'nyvTSJMKId',
+        #         'ext': 'mp4',
+        #         'title': 'Flipagram by sjuria101 featuring Midnight Memories by One Direction',
+        #         'description': '#love for cats.',
+        #         'timestamp': 1461244995,
+        #         'upload_date': '20160421',
+        #     },
+        #     'params': {
+        #         'force_generic_extractor': True,
+        #     },
+        # }
    ]

    def report_following_redirect(self, new_url):
--- a/youtube_dl/extractor/lynda.py
+++ b/youtube_dl/extractor/lynda.py
@@ -100,7 +100,7 @@ class LyndaIE(LyndaBaseIE):

    _TESTS = [{
        'url': 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html',
-        'md5': '679734f6786145da3546585de9a356be',
+        # md5 is unstable
        'info_dict': {
            'id': '114408',
            'ext': 'mp4',
--- a/youtube_dl/extractor/miomio.py
+++ b/youtube_dl/extractor/miomio.py
@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import random

 from .common import InfoExtractor
+from ..compat import compat_urlparse
 from ..utils import (
    xpath_text,
    int_or_none,
@@ -18,13 +19,16 @@ class MioMioIE(InfoExtractor):
    _TESTS = [{
        # "type=video" in flashvars
        'url': 'http://www.miomio.tv/watch/cc88912/',
-        'md5': '317a5f7f6b544ce8419b784ca8edae65',
        'info_dict': {
            'id': '88912',
            'ext': 'flv',
            'title': '【SKY】字幕 铠武昭和VS平成 假面骑士大战FEAT战队 魔星字幕组 字幕',
            'duration': 5923,
        },
+        'params': {
+            # The server provides broken file
+            'skip_download': True,
+        }
    }, {
        'url': 'http://www.miomio.tv/watch/cc184024/',
        'info_dict': {
@@ -32,7 +36,7 @@ class MioMioIE(InfoExtractor):
            'title': '《动漫同人插画绘制》',
        },
        'playlist_mincount': 86,
-        'skip': 'This video takes time too long for retrieving the URL',
+        'skip': 'Unable to load videos',
    }, {
        'url': 'http://www.miomio.tv/watch/cc173113/',
        'info_dict': {
@@ -40,20 +44,23 @@ class MioMioIE(InfoExtractor):
            'title': 'The New Macbook 2015 上手试玩与简评'
        },
        'playlist_mincount': 2,
+        'skip': 'Unable to load videos',
+    }, {
+        # new 'h5' player
+        'url': 'http://www.miomio.tv/watch/cc273295/',
+        'md5': '',
+        'info_dict': {
+            'id': '273295',
+            'ext': 'mp4',
+            'title': 'アウト×デラックス 20160526',
+        },
+        'params': {
+            # intermittent HTTP 500
+            'skip_download': True,
+        },
    }]

-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-
-        title = self._html_search_meta(
-            'description', webpage, 'title', fatal=True)
-
-        mioplayer_path = self._search_regex(
-            r'src="(/mioplayer/[^"]+)"', webpage, 'ref_path')
-
-        http_headers = {'Referer': 'http://www.miomio.tv%s' % mioplayer_path}
-
+    def _extract_mioplayer(self, webpage, video_id, title, http_headers):
        xml_config = self._search_regex(
            r'flashvars="type=(?:sina|video)&amp;(.+?)&amp;',
            webpage, 'xml config')
@@ -92,10 +99,34 @@ class MioMioIE(InfoExtractor):
                'http_headers': http_headers,
            })

+        return entries
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+
+        title = self._html_search_meta(
+            'description', webpage, 'title', fatal=True)
+
+        mioplayer_path = self._search_regex(
+            r'src="(/mioplayer(?:_h5)?/[^"]+)"', webpage, 'ref_path')
+
+        if '_h5' in mioplayer_path:
+            player_url = compat_urlparse.urljoin(url, mioplayer_path)
+            player_webpage = self._download_webpage(
+                player_url, video_id,
+                note='Downloading player webpage', headers={'Referer': url})
+            entries = self._parse_html5_media_entries(player_url, player_webpage)
+            http_headers = {'Referer': player_url}
+        else:
+            http_headers = {'Referer': 'http://www.miomio.tv%s' % mioplayer_path}
+            entries = self._extract_mioplayer(webpage, video_id, title, http_headers)
+
        if len(entries) == 1:
            segment = entries[0]
            segment['id'] = video_id
            segment['title'] = title
+            segment['http_headers'] = http_headers
            return segment

        return {
--- a/youtube_dl/extractor/playvid.py
+++ b/youtube_dl/extractor/playvid.py
@@ -15,7 +15,7 @@ from ..utils import (

 class PlayvidIE(InfoExtractor):
    _VALID_URL = r'https?://www\.playvid\.com/watch(\?v=|/)(?P<id>.+?)(?:#|$)'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://www.playvid.com/watch/RnmBNgtrrJu',
        'md5': 'ffa2f6b2119af359f544388d8c01eb6c',
        'info_dict': {
@@ -24,8 +24,19 @@ class PlayvidIE(InfoExtractor):
            'title': 'md5:9256d01c6317e3f703848b5906880dc8',
            'duration': 82,
            'age_limit': 18,
-        }
-    }
+        },
+        'skip': 'Video removed due to ToS',
+    }, {
+        'url': 'http://www.playvid.com/watch/hwb0GpNkzgH',
+        'md5': '39d49df503ad7b8f23a4432cbf046477',
+        'info_dict': {
+            'id': 'hwb0GpNkzgH',
+            'ext': 'mp4',
+            'title': 'Ellen Euro Cutie Blond Takes a Sexy Survey Get Facial in The Park',
+            'age_limit': 18,
+            'thumbnail': 're:^https?://.*\.jpg$',
+        },
+    }]

    def _real_extract(self, url):
        video_id = self._match_id(url)
--- a/youtube_dl/extractor/roosterteeth.py
+++ b/youtube_dl/extractor/roosterteeth.py
@@ -0,0 +1,148 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    int_or_none,
+    strip_or_none,
+    unescapeHTML,
+    urlencode_postdata,
+)
+
+
+class RoosterTeethIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:.+?\.)?roosterteeth\.com/episode/(?P<id>[^/?#&]+)'
+    _LOGIN_URL = 'https://roosterteeth.com/login'
+    _NETRC_MACHINE = 'roosterteeth'
+    _TESTS = [{
+        'url': 'http://roosterteeth.com/episode/million-dollars-but-season-2-million-dollars-but-the-game-announcement',
+        'md5': 'e2bd7764732d785ef797700a2489f212',
+        'info_dict': {
+            'id': '26576',
+            'display_id': 'million-dollars-but-season-2-million-dollars-but-the-game-announcement',
+            'ext': 'mp4',
+            'title': 'Million Dollars, But...: Million Dollars, But... The Game Announcement',
+            'description': 'md5:0cc3b21986d54ed815f5faeccd9a9ca5',
+            'thumbnail': 're:^https?://.*\.png$',
+            'series': 'Million Dollars, But...',
+            'episode': 'Million Dollars, But... The Game Announcement',
+            'comment_count': int,
+        },
+    }, {
+        'url': 'http://achievementhunter.roosterteeth.com/episode/off-topic-the-achievement-hunter-podcast-2016-i-didn-t-think-it-would-pass-31',
+        'only_matching': True,
+    }, {
+        'url': 'http://funhaus.roosterteeth.com/episode/funhaus-shorts-2016-austin-sucks-funhaus-shorts',
+        'only_matching': True,
+    }, {
+        'url': 'http://screwattack.roosterteeth.com/episode/death-battle-season-3-mewtwo-vs-shadow',
+        'only_matching': True,
+    }, {
+        'url': 'http://theknow.roosterteeth.com/episode/the-know-game-news-season-1-boring-steam-sales-are-better',
+        'only_matching': True,
+    }, {
+        # only available for FIRST members
+        'url': 'http://roosterteeth.com/episode/rt-docs-the-world-s-greatest-head-massage-the-world-s-greatest-head-massage-an-asmr-journey-part-one',
+        'only_matching': True,
+    }]
+
+    def _login(self):
+        (username, password) = self._get_login_info()
+        if username is None:
+            return
+
+        login_page = self._download_webpage(
+            self._LOGIN_URL, None,
+            note='Downloading login page',
+            errnote='Unable to download login page')
+
+        login_form = self._hidden_inputs(login_page)
+
+        login_form.update({
+            'username': username,
+            'password': password,
+        })
+
+        login_request = self._download_webpage(
+            self._LOGIN_URL, None,
+            note='Logging in as %s' % username,
+            data=urlencode_postdata(login_form),
+            headers={
+                'Referer': self._LOGIN_URL,
+            })
+
+        if not any(re.search(p, login_request) for p in (
+                r'href=["\']https?://(?:www\.)?roosterteeth\.com/logout"',
+                r'>Sign Out<')):
+            error = self._html_search_regex(
+                r'(?s)<div[^>]+class=(["\']).*?\balert-danger\b.*?\1[^>]*>(?:\s*<button[^>]*>.*?</button>)?(?P<error>.+?)</div>',
+                login_request, 'alert', default=None, group='error')
+            if error:
+                raise ExtractorError('Unable to login: %s' % error, expected=True)
+            raise ExtractorError('Unable to log in')
+
+    def _real_initialize(self):
+        self._login()
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, display_id)
+
+        episode = strip_or_none(unescapeHTML(self._search_regex(
+            (r'videoTitle\s*=\s*(["\'])(?P<title>(?:(?!\1).)+)\1',
+             r'<title>(?P<title>[^<]+)</title>'), webpage, 'title',
+            default=None, group='title')))
+
+        title = strip_or_none(self._og_search_title(
+            webpage, default=None)) or episode
+
+        m3u8_url = self._search_regex(
+            r'file\s*:\s*(["\'])(?P<url>http.+?\.m3u8.*?)\1',
+            webpage, 'm3u8 url', default=None, group='url')
+
+        if not m3u8_url:
+            if re.search(r'<div[^>]+class=["\']non-sponsor', webpage):
+                self.raise_login_required(
+                    '%s is only available for FIRST members' % display_id)
+
+            if re.search(r'<div[^>]+class=["\']golive-gate', webpage):
+                self.raise_login_required('%s is not available yet' % display_id)
+
+            raise ExtractorError('Unable to extract m3u8 URL')
+
+        formats = self._extract_m3u8_formats(
+            m3u8_url, display_id, ext='mp4',
+            entry_protocol='m3u8_native', m3u8_id='hls')
+        self._sort_formats(formats)
+
+        description = strip_or_none(self._og_search_description(webpage))
+        thumbnail = self._proto_relative_url(self._og_search_thumbnail(webpage))
+
+        series = self._search_regex(
+            (r'<h2>More ([^<]+)</h2>', r'<a[^>]+>See All ([^<]+) Videos<'),
+            webpage, 'series', fatal=False)
+
+        comment_count = int_or_none(self._search_regex(
+            r'>Comments \((\d+)\)<', webpage,
+            'comment count', fatal=False))
+
+        video_id = self._search_regex(
+            (r'containerId\s*=\s*["\']episode-(\d+)\1',
+             r'<div[^<]+id=["\']episode-(\d+)'), webpage,
+            'video id', default=display_id)
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+            'thumbnail': thumbnail,
+            'series': series,
+            'episode': episode,
+            'comment_count': comment_count,
+            'formats': formats,
+        }
--- a/youtube_dl/extractor/rudo.py
+++ b/youtube_dl/extractor/rudo.py
@@ -0,0 +1,53 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .jwplatform import JWPlatformBaseIE
+from ..utils import (
+    js_to_json,
+    get_element_by_class,
+    unified_strdate,
+)
+
+
+class RudoIE(JWPlatformBaseIE):
+    _VALID_URL = r'https?://rudo\.video/vod/(?P<id>[0-9a-zA-Z]+)'
+
+    _TEST = {
+        'url': 'http://rudo.video/vod/oTzw0MGnyG',
+        'md5': '2a03a5b32dd90a04c83b6d391cf7b415',
+        'info_dict': {
+            'id': 'oTzw0MGnyG',
+            'ext': 'mp4',
+            'title': 'Comentario Tomás Mosciatti',
+            'upload_date': '20160617',
+        },
+    }
+
+    @classmethod
+    def _extract_url(self, webpage):
+        mobj = re.search(
+            '<iframe[^>]+src=(?P<q1>[\'"])(?P<url>(?:https?:)?//rudo\.video/vod/[0-9a-zA-Z]+)(?P=q1)',
+            webpage)
+        if mobj:
+            return mobj.group('url')
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, video_id, encoding='iso-8859-1')
+
+        jwplayer_data = self._parse_json(self._search_regex(
+            r'(?s)playerInstance\.setup\(({.+?})\)', webpage, 'jwplayer data'), video_id,
+            transform_source=lambda s: js_to_json(re.sub(r'encodeURI\([^)]+\)', '""', s)))
+
+        info_dict = self._parse_jwplayer_data(
+            jwplayer_data, video_id, require_title=False, m3u8_id='hls')
+
+        info_dict.update({
+            'title': self._og_search_title(webpage),
+            'upload_date': unified_strdate(get_element_by_class('date', webpage)),
+        })
+
+        return info_dict
--- a/youtube_dl/extractor/shahid.py
+++ b/youtube_dl/extractor/shahid.py
@@ -2,11 +2,11 @@
 from __future__ import unicode_literals

 from .common import InfoExtractor
-from ..compat import compat_urllib_parse_urlencode
 from ..utils import (
    ExtractorError,
    int_or_none,
    parse_iso8601,
+    str_or_none,
 )


@@ -33,45 +33,27 @@ class ShahidIE(InfoExtractor):
        'only_matching': True
    }]

-    def _handle_error(self, response):
-        if not isinstance(response, dict):
-            return
-        error = response.get('error')
+    def _call_api(self, path, video_id, note):
+        data = self._download_json(
+            'http://api.shahid.net/api/v1_1/' + path, video_id, note, query={
+                'apiKey': 'sh@hid0nlin3',
+                'hash': 'b2wMCTHpSmyxGqQjJFOycRmLSex+BpTK/ooxy6vHaqs=',
+            }).get('data', {})
+
+        error = data.get('error')
        if error:
            raise ExtractorError(
                '%s returned error: %s' % (self.IE_NAME, '\n'.join(error.values())),
                expected=True)

-    def _download_json(self, url, video_id, note='Downloading JSON metadata'):
-        response = super(ShahidIE, self)._download_json(url, video_id, note)['data']
-        self._handle_error(response)
-        return response
+        return data

    def _real_extract(self, url):
        video_id = self._match_id(url)

-        webpage = self._download_webpage(url, video_id)
-
-        api_vars = {
-            'id': video_id,
-            'type': 'player',
-            'url': 'http://api.shahid.net/api/v1_1',
-            'playerType': 'episode',
-        }
-
-        flashvars = self._search_regex(
-            r'var\s+flashvars\s*=\s*({[^}]+})', webpage, 'flashvars', default=None)
-        if flashvars:
-            for key in api_vars.keys():
-                value = self._search_regex(
-                    r'\b%s\s*:\s*(?P<q>["\'])(?P<value>.+?)(?P=q)' % key,
-                    flashvars, 'type', default=None, group='value')
-                if value:
-                    api_vars[key] = value
-
-        player = self._download_json(
-            'https://shahid.mbc.net/arContent/getPlayerContent-param-.id-%s.type-%s.html'
-            % (video_id, api_vars['type']), video_id, 'Downloading player JSON')
+        player = self._call_api(
+            'Content/Episode/%s' % video_id,
+            video_id, 'Downloading player JSON')

        if player.get('drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)
@@ -79,22 +61,11 @@ class ShahidIE(InfoExtractor):
        formats = self._extract_m3u8_formats(player['url'], video_id, 'mp4')
        self._sort_formats(formats)

-        video = self._download_json(
-            '%s/%s/%s?%s' % (
-                api_vars['url'], api_vars['playerType'], api_vars['id'],
-                compat_urllib_parse_urlencode({
-                    'apiKey': 'sh@hid0nlin3',
-                    'hash': 'b2wMCTHpSmyxGqQjJFOycRmLSex+BpTK/ooxy6vHaqs=',
-                })),
-            video_id, 'Downloading video JSON')
-
-        video = video[api_vars['playerType']]
+        video = self._call_api(
+            'episode/%s' % video_id, video_id,
+            'Downloading video JSON')['episode']

        title = video['title']
-        description = video.get('description')
-        thumbnail = video.get('thumbnailUrl')
-        duration = int_or_none(video.get('duration'))
-        timestamp = parse_iso8601(video.get('referenceDate'))
        categories = [
            category['name']
            for category in video.get('genres', []) if 'name' in category]
@@ -102,10 +73,16 @@ class ShahidIE(InfoExtractor):
        return {
            'id': video_id,
            'title': title,
-            'description': description,
-            'thumbnail': thumbnail,
-            'duration': duration,
-            'timestamp': timestamp,
+            'description': video.get('description'),
+            'thumbnail': video.get('thumbnailUrl'),
+            'duration': int_or_none(video.get('duration')),
+            'timestamp': parse_iso8601(video.get('referenceDate')),
            'categories': categories,
+            'series': video.get('showTitle') or video.get('showName'),
+            'season': video.get('seasonTitle'),
+            'season_number': int_or_none(video.get('seasonNumber')),
+            'season_id': str_or_none(video.get('seasonId')),
+            'episode_number': int_or_none(video.get('number')),
+            'episode_id': video_id,
            'formats': formats,
        }
--- a/youtube_dl/extractor/srmediathek.py
+++ b/youtube_dl/extractor/srmediathek.py
@@ -11,7 +11,7 @@ from ..utils import (
 class SRMediathekIE(ARDMediathekIE):
    IE_NAME = 'sr:mediathek'
    IE_DESC = 'Saarländischer Rundfunk'
-    _VALID_URL = r'https?://sr-mediathek\.sr-online\.de/index\.php\?.*?&id=(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://sr-mediathek(?:\.sr-online)?\.de/index\.php\?.*?&id=(?P<id>[0-9]+)'

    _TESTS = [{
        'url': 'http://sr-mediathek.sr-online.de/index.php?seite=7&id=28455',
@@ -35,7 +35,9 @@ class SRMediathekIE(ARDMediathekIE):
            # m3u8 download
            'skip_download': True,
        },
-        'expected_warnings': ['Unable to download f4m manifest']
+    }, {
+        'url': 'http://sr-mediathek.de/index.php?seite=7&id=7480',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
--- a/youtube_dl/extractor/tmz.py
+++ b/youtube_dl/extractor/tmz.py
@@ -5,31 +5,27 @@ from .common import InfoExtractor


 class TMZIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?tmz\.com/videos/(?P<id>[^/]+)/?'
-    _TEST = {
+    _VALID_URL = r'https?://(?:www\.)?tmz\.com/videos/(?P<id>[^/?#]+)'
+    _TESTS = [{
        'url': 'http://www.tmz.com/videos/0_okj015ty/',
-        'md5': '791204e3bf790b1426cb2db0706184c0',
+        'md5': '4d22a51ef205b6c06395d8394f72d560',
        'info_dict': {
            'id': '0_okj015ty',
-            'url': 'http://tmz.vo.llnwd.net/o28/2014-03/13/0_okj015ty_0_rt8ro3si_2.mp4',
            'ext': 'mp4',
            'title': 'Kim Kardashian\'s Boobs Unlock a Mystery!',
            'description': 'Did Kim Kardasain try to one-up Khloe by one-upping Kylie???  Or is she just showing off her amazing boobs?',
-            'thumbnail': r're:http://cdnbakmi\.kaltura\.com/.*thumbnail.*',
+            'timestamp': 1394747163,
+            'uploader_id': 'batchUser',
+            'upload_date': '20140313',
        }
-    }
+    }, {
+        'url': 'http://www.tmz.com/videos/0-cegprt2p/',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-
-        return {
-            'id': video_id,
-            'url': self._html_search_meta('VideoURL', webpage, fatal=True),
-            'title': self._og_search_title(webpage),
-            'description': self._og_search_description(webpage),
-            'thumbnail': self._html_search_meta('ThumbURL', webpage),
-        }
+        video_id = self._match_id(url).replace('-', '_')
+        return self.url_result('kaltura:591531:%s' % video_id, 'Kaltura', video_id)


 class TMZArticleIE(InfoExtractor):
--- a/youtube_dl/extractor/tvplay.py
+++ b/youtube_dl/extractor/tvplay.py
@@ -8,43 +8,36 @@ from ..compat import compat_str
 from ..utils import (
    parse_iso8601,
    qualities,
+    determine_ext,
+    update_url_query,
+    int_or_none,
 )


 class TVPlayIE(InfoExtractor):
    IE_DESC = 'TV3Play and related services'
    _VALID_URL = r'''(?x)https?://(?:www\.)?
-        (?:tvplay\.lv/parraides|
-           tv3play\.lt/programos|
-           play\.tv3\.lt/programos|
-           tv3play\.ee/sisu|
-           tv3play\.se/program|
-           tv6play\.se/program|
-           tv8play\.se/program|
-           tv10play\.se/program|
-           tv3play\.no/programmer|
-           viasat4play\.no/programmer|
-           tv6play\.no/programmer|
-           tv3play\.dk/programmer|
+        (?:tvplay(?:\.skaties)?\.lv/parraides|
+           (?:tv3play|play\.tv3)\.lt/programos|
+           tv3play(?:\.tv3)?\.ee/sisu|
+           tv(?:3|6|8|10)play\.se/program|
+           (?:(?:tv3play|viasat4play|tv6play)\.no|tv3play\.dk)/programmer|
           play\.novatv\.bg/programi
        )/[^/]+/(?P<id>\d+)
        '''
    _TESTS = [
        {
            'url': 'http://www.tvplay.lv/parraides/vinas-melo-labak/418113?autostart=true',
+            'md5': 'a1612fe0849455423ad8718fe049be21',
            'info_dict': {
                'id': '418113',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Kādi ir īri? - Viņas melo labāk',
                'description': 'Baiba apsmej īrus, kādi tie ir un ko viņi dara.',
                'duration': 25,
                'timestamp': 1406097056,
                'upload_date': '20140723',
            },
-            'params': {
-                # rtmp download
-                'skip_download': True,
-            },
        },
        {
            'url': 'http://play.tv3.lt/programos/moterys-meluoja-geriau/409229?autostart=true',
@@ -82,7 +75,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.tv3play.se/program/husraddarna/395385?autostart=true',
            'info_dict': {
                'id': '395385',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Husräddarna S02E07',
                'description': 'md5:f210c6c89f42d4fc39faa551be813777',
                'duration': 2574,
@@ -90,7 +83,6 @@ class TVPlayIE(InfoExtractor):
                'upload_date': '20140520',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -98,7 +90,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.tv6play.se/program/den-sista-dokusapan/266636?autostart=true',
            'info_dict': {
                'id': '266636',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Den sista dokusåpan S01E08',
                'description': 'md5:295be39c872520221b933830f660b110',
                'duration': 1492,
@@ -107,7 +99,6 @@ class TVPlayIE(InfoExtractor):
                'age_limit': 18,
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -115,7 +106,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.tv8play.se/program/antikjakten/282756?autostart=true',
            'info_dict': {
                'id': '282756',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Antikjakten S01E10',
                'description': 'md5:1b201169beabd97e20c5ad0ad67b13b8',
                'duration': 2646,
@@ -123,7 +114,6 @@ class TVPlayIE(InfoExtractor):
                'upload_date': '20120925',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -131,7 +121,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.tv3play.no/programmer/anna-anka-soker-assistent/230898?autostart=true',
            'info_dict': {
                'id': '230898',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Anna Anka søker assistent - Ep. 8',
                'description': 'md5:f80916bf5bbe1c5f760d127f8dd71474',
                'duration': 2656,
@@ -139,7 +129,6 @@ class TVPlayIE(InfoExtractor):
                'upload_date': '20100628',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -147,7 +136,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.viasat4play.no/programmer/budbringerne/21873?autostart=true',
            'info_dict': {
                'id': '21873',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Budbringerne program 10',
                'description': 'md5:4db78dc4ec8a85bb04fd322a3ee5092d',
                'duration': 1297,
@@ -155,7 +144,6 @@ class TVPlayIE(InfoExtractor):
                'upload_date': '20090929',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -163,7 +151,7 @@ class TVPlayIE(InfoExtractor):
            'url': 'http://www.tv6play.no/programmer/hotelinspektor-alex-polizzi/361883?autostart=true',
            'info_dict': {
                'id': '361883',
-                'ext': 'flv',
+                'ext': 'mp4',
                'title': 'Hotelinspektør Alex Polizzi - Ep. 10',
                'description': 'md5:3ecf808db9ec96c862c8ecb3a7fdaf81',
                'duration': 2594,
@@ -171,7 +159,6 @@ class TVPlayIE(InfoExtractor):
                'upload_date': '20140224',
            },
            'params': {
-                # rtmp download
                'skip_download': True,
            },
        },
@@ -191,6 +178,14 @@ class TVPlayIE(InfoExtractor):
                'skip_download': True,
            },
        },
+        {
+            'url': 'http://tvplay.skaties.lv/parraides/vinas-melo-labak/418113?autostart=true',
+            'only_matching': True,
+        },
+        {
+            'url': 'http://tv3play.tv3.ee/sisu/kodu-keset-linna/238551?autostart=true',
+            'only_matching': True,
+        }
    ]

    def _real_extract(self, url):
@@ -199,7 +194,9 @@ class TVPlayIE(InfoExtractor):
        video = self._download_json(
            'http://playapi.mtgx.tv/v1/videos/%s' % video_id, video_id, 'Downloading video JSON')

-        if video['is_geo_blocked']:
+        title = video['title']
+
+        if video.get('is_geo_blocked'):
            self.report_warning(
                'This content might not be available in your country due to copyright reasons')

@@ -208,42 +205,50 @@ class TVPlayIE(InfoExtractor):

        quality = qualities(['hls', 'medium', 'high'])
        formats = []
-        for format_id, video_url in streams['streams'].items():
+        for format_id, video_url in streams.get('streams', {}).items():
            if not video_url or not isinstance(video_url, compat_str):
                continue
-            fmt = {
-                'format_id': format_id,
-                'preference': quality(format_id),
-            }
-            if video_url.startswith('rtmp'):
-                m = re.search(r'^(?P<url>rtmp://[^/]+/(?P<app>[^/]+))/(?P<playpath>.+)$', video_url)
-                if not m:
-                    continue
-                fmt.update({
-                    'ext': 'flv',
-                    'url': m.group('url'),
-                    'app': m.group('app'),
-                    'play_path': m.group('playpath'),
-                })
-            elif video_url.endswith('.f4m'):
+            ext = determine_ext(video_url)
+            if ext == 'f4m':
                formats.extend(self._extract_f4m_formats(
-                    video_url + '?hdcore=3.5.0&plugin=aasp-3.5.0.151.81', video_id))
-                continue
+                    update_url_query(video_url, {
+                        'hdcore': '3.5.0',
+                        'plugin': 'aasp-3.5.0.151.81'
+                    }), video_id, f4m_id='hds', fatal=False))
+            elif ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    video_url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id='hls', fatal=False))
            else:
-                fmt.update({
-                    'url': video_url,
-                })
-            formats.append(fmt)
-
+                fmt = {
+                    'format_id': format_id,
+                    'quality': quality(format_id),
+                    'ext': ext,
+                }
+                if video_url.startswith('rtmp'):
+                    m = re.search(r'^(?P<url>rtmp://[^/]+/(?P<app>[^/]+))/(?P<playpath>.+)$', video_url)
+                    if not m:
+                        continue
+                    fmt.update({
+                        'ext': 'flv',
+                        'url': m.group('url'),
+                        'app': m.group('app'),
+                        'play_path': m.group('playpath'),
+                    })
+                else:
+                    fmt.update({
+                        'url': video_url,
+                    })
+                formats.append(fmt)
        self._sort_formats(formats)

        return {
            'id': video_id,
-            'title': video['title'],
-            'description': video['description'],
-            'duration': video['duration'],
-            'timestamp': parse_iso8601(video['created_at']),
-            'view_count': video['views']['total'],
-            'age_limit': video.get('age_limit', 0),
+            'title': title,
+            'description': video.get('description'),
+            'duration': int_or_none(video.get('duration')),
+            'timestamp': parse_iso8601(video.get('created_at')),
+            'view_count': int_or_none(video.get('views', {}).get('total')),
+            'age_limit': int_or_none(video.get('age_limit', 0)),
            'formats': formats,
        }
--- a/youtube_dl/extractor/vidzi.py
+++ b/youtube_dl/extractor/vidzi.py
@@ -9,8 +9,8 @@ from ..utils import (


 class VidziIE(JWPlatformBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?vidzi\.tv/(?P<id>\w+)'
-    _TEST = {
+    _VALID_URL = r'https?://(?:www\.)?vidzi\.tv/(?:embed-)?(?P<id>[0-9a-zA-Z]+)'
+    _TESTS = [{
        'url': 'http://vidzi.tv/cghql9yq6emu.html',
        'md5': '4f16c71ca0c8c8635ab6932b5f3f1660',
        'info_dict': {
@@ -22,12 +22,16 @@ class VidziIE(JWPlatformBaseIE):
            # m3u8 download
            'skip_download': True,
        },
-    }
+    }, {
+        'url': 'http://vidzi.tv/embed-4z2yb0rzphe9-600x338.html',
+        'skip_download': True,
+    }]

    def _real_extract(self, url):
        video_id = self._match_id(url)

-        webpage = self._download_webpage(url, video_id)
+        webpage = self._download_webpage(
+            'http://vidzi.tv/%s' % video_id, video_id)
        title = self._html_search_regex(
            r'(?s)<h2 class="video-title">(.*?)</h2>', webpage, 'title')

--- a/youtube_dl/extractor/vimeo.py
+++ b/youtube_dl/extractor/vimeo.py
@@ -364,6 +364,11 @@ class VimeoIE(VimeoBaseInfoExtractor):
            r'<embed[^>]+?src="((?:https?:)?//(?:www\.)?vimeo\.com/moogaloop\.swf.+?)"', webpage)
        if mobj:
            return mobj.group(1)
+        # Look more for non-standard embedded Vimeo player
+        mobj = re.search(
+            r'<video[^>]+src=(?P<q1>[\'"])(?P<url>(?:https?:)?//(?:www\.)?vimeo\.com/[0-9]+)(?P=q1)', webpage)
+        if mobj:
+            return mobj.group('url')

    def _verify_player_video_password(self, url, video_id):
        password = self._downloader.params.get('videopassword')
--- a/youtube_dl/extractor/vk.py
+++ b/youtube_dl/extractor/vk.py
@@ -6,11 +6,18 @@ import json
 import sys

 from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
 from ..utils import (
+    clean_html,
    ExtractorError,
+    get_element_by_class,
    int_or_none,
    orderedSet,
+    parse_duration,
+    remove_start,
    str_to_int,
    unescapeHTML,
    unified_strdate,
@@ -20,7 +27,55 @@ from .vimeo import VimeoIE
 from .pladform import PladformIE


-class VKIE(InfoExtractor):
+class VKBaseIE(InfoExtractor):
+    _NETRC_MACHINE = 'vk'
+
+    def _login(self):
+        (username, password) = self._get_login_info()
+        if username is None:
+            return
+
+        login_page, url_handle = self._download_webpage_handle(
+            'https://vk.com', None, 'Downloading login page')
+
+        login_form = self._hidden_inputs(login_page)
+
+        login_form.update({
+            'email': username.encode('cp1251'),
+            'pass': password.encode('cp1251'),
+        })
+
+        # https://new.vk.com/ serves two same remixlhk cookies in Set-Cookie header
+        # and expects the first one to be set rather than second (see
+        # https://github.com/rg3/youtube-dl/issues/9841#issuecomment-227871201).
+        # As of RFC6265 the newer one cookie should be set into cookie store
+        # what actually happens.
+        # We will workaround this VK issue by resetting the remixlhk cookie to
+        # the first one manually.
+        cookies = url_handle.headers.get('Set-Cookie')
+        if cookies:
+            if sys.version_info[0] >= 3:
+                cookies = cookies.encode('iso-8859-1')
+            cookies = cookies.decode('utf-8')
+            remixlhk = re.search(r'remixlhk=(.+?);.*?\bdomain=(.+?)(?:[,;]|$)', cookies)
+            if remixlhk:
+                value, domain = remixlhk.groups()
+                self._set_cookie(domain, 'remixlhk', value)
+
+        login_page = self._download_webpage(
+            'https://login.vk.com/?act=login', None,
+            note='Logging in as %s' % username,
+            data=urlencode_postdata(login_form))
+
+        if re.search(r'onLoginFailed', login_page):
+            raise ExtractorError(
+                'Unable to login, incorrect username and/or password', expected=True)
+
+    def _real_initialize(self):
+        self._login()
+
+
+class VKIE(VKBaseIE):
    IE_NAME = 'vk'
    IE_DESC = 'VK'
    _VALID_URL = r'''(?x)
@@ -38,8 +93,6 @@ class VKIE(InfoExtractor):
                            (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+))?
                        )
                    '''
-    _NETRC_MACHINE = 'vk'
-
    _TESTS = [
        {
            'url': 'http://vk.com/videos-77521?z=video-77521_162222515%2Fclub77521',
@@ -189,49 +242,6 @@ class VKIE(InfoExtractor):
        }
    ]

-    def _login(self):
-        (username, password) = self._get_login_info()
-        if username is None:
-            return
-
-        login_page, url_handle = self._download_webpage_handle(
-            'https://vk.com', None, 'Downloading login page')
-
-        login_form = self._hidden_inputs(login_page)
-
-        login_form.update({
-            'email': username.encode('cp1251'),
-            'pass': password.encode('cp1251'),
-        })
-
-        # https://new.vk.com/ serves two same remixlhk cookies in Set-Cookie header
-        # and expects the first one to be set rather than second (see
-        # https://github.com/rg3/youtube-dl/issues/9841#issuecomment-227871201).
-        # As of RFC6265 the newer one cookie should be set into cookie store
-        # what actually happens.
-        # We will workaround this VK issue by resetting the remixlhk cookie to
-        # the first one manually.
-        cookies = url_handle.headers.get('Set-Cookie')
-        if sys.version_info[0] >= 3:
-            cookies = cookies.encode('iso-8859-1')
-        cookies = cookies.decode('utf-8')
-        remixlhk = re.search(r'remixlhk=(.+?);.*?\bdomain=(.+?)(?:[,;]|$)', cookies)
-        if remixlhk:
-            value, domain = remixlhk.groups()
-            self._set_cookie(domain, 'remixlhk', value)
-
-        login_page = self._download_webpage(
-            'https://login.vk.com/?act=login', None,
-            note='Logging in as %s' % username,
-            data=urlencode_postdata(login_form))
-
-        if re.search(r'onLoginFailed', login_page):
-            raise ExtractorError(
-                'Unable to login, incorrect username and/or password', expected=True)
-
-    def _real_initialize(self):
-        self._login()
-
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('videoid')
@@ -355,7 +365,7 @@ class VKIE(InfoExtractor):
        }


-class VKUserVideosIE(InfoExtractor):
+class VKUserVideosIE(VKBaseIE):
    IE_NAME = 'vk:uservideos'
    IE_DESC = "VK - User's Videos"
    _VALID_URL = r'https?://(?:(?:m|new)\.)?vk\.com/videos(?P<id>-?[0-9]+)(?!\?.*\bz=video)(?:[/?#&]|$)'
@@ -396,3 +406,121 @@ class VKUserVideosIE(InfoExtractor):
            webpage, 'title', default=page_id))

        return self.playlist_result(entries, page_id, title)
+
+
+class VKWallPostIE(VKBaseIE):
+    IE_NAME = 'vk:wallpost'
+    _VALID_URL = r'https?://(?:(?:(?:(?:m|new)\.)?vk\.com/(?:[^?]+\?.*\bw=)?wall(?P<id>-?\d+_\d+)))'
+    _TESTS = [{
+        # public page URL, audio playlist
+        'url': 'https://vk.com/bs.official?w=wall-23538238_35',
+        'info_dict': {
+            'id': '23538238_35',
+            'title': 'Black Shadow - Wall post 23538238_35',
+            'description': 'md5:3f84b9c4f9ef499731cf1ced9998cc0c',
+        },
+        'playlist': [{
+            'md5': '5ba93864ec5b85f7ce19a9af4af080f6',
+            'info_dict': {
+                'id': '135220665_111806521',
+                'ext': 'mp3',
+                'title': 'Black Shadow - Слепое Верование',
+                'duration': 370,
+                'uploader': 'Black Shadow',
+                'artist': 'Black Shadow',
+                'track': 'Слепое Верование',
+            },
+        }, {
+            'md5': '4cc7e804579122b17ea95af7834c9233',
+            'info_dict': {
+                'id': '135220665_111802303',
+                'ext': 'mp3',
+                'title': 'Black Shadow - Война - Негасимое Бездны Пламя!',
+                'duration': 423,
+                'uploader': 'Black Shadow',
+                'artist': 'Black Shadow',
+                'track': 'Война - Негасимое Бездны Пламя!',
+            },
+            'params': {
+                'skip_download': True,
+            },
+        }],
+        'skip': 'Requires vk account credentials',
+    }, {
+        # single YouTube embed, no leading -
+        'url': 'https://vk.com/wall85155021_6319',
+        'info_dict': {
+            'id': '85155021_6319',
+            'title': 'Sergey Gorbunov - Wall post 85155021_6319',
+        },
+        'playlist_count': 1,
+        'skip': 'Requires vk account credentials',
+    }, {
+        # wall page URL
+        'url': 'https://vk.com/wall-23538238_35',
+        'only_matching': True,
+    }, {
+        # mobile wall page URL
+        'url': 'https://m.vk.com/wall-23538238_35',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        post_id = self._match_id(url)
+
+        wall_url = 'https://vk.com/wall%s' % post_id
+
+        post_id = remove_start(post_id, '-')
+
+        webpage = self._download_webpage(wall_url, post_id)
+
+        error = self._html_search_regex(
+            r'>Error</div>\s*<div[^>]+class=["\']body["\'][^>]*>([^<]+)',
+            webpage, 'error', default=None)
+        if error:
+            raise ExtractorError('VK said: %s' % error, expected=True)
+
+        description = clean_html(get_element_by_class('wall_post_text', webpage))
+        uploader = clean_html(get_element_by_class(
+            'fw_post_author', webpage)) or self._og_search_description(webpage)
+        thumbnail = self._og_search_thumbnail(webpage)
+
+        entries = []
+
+        for audio in re.finditer(r'''(?sx)
+                            <input[^>]+
+                                id=(?P<q1>["\'])audio_info(?P<id>\d+_\d+).*?(?P=q1)[^>]+
+                                value=(?P<q2>["\'])(?P<url>http.+?)(?P=q2)
+                                .+?
+                            </table>''', webpage):
+            audio_html = audio.group(0)
+            audio_id = audio.group('id')
+            duration = parse_duration(get_element_by_class('duration', audio_html))
+            track = self._html_search_regex(
+                r'<span[^>]+id=["\']title%s[^>]*>([^<]+)' % audio_id,
+                audio_html, 'title', default=None)
+            artist = self._html_search_regex(
+                r'>([^<]+)</a></b>\s*&ndash', audio_html,
+                'artist', default=None)
+            entries.append({
+                'id': audio_id,
+                'url': audio.group('url'),
+                'title': '%s - %s' % (artist, track) if artist and track else audio_id,
+                'thumbnail': thumbnail,
+                'duration': duration,
+                'uploader': uploader,
+                'artist': artist,
+                'track': track,
+            })
+
+        for video in re.finditer(
+                r'<a[^>]+href=(["\'])(?P<url>/video(?:-?[\d_]+).*?)\1', webpage):
+            entries.append(self.url_result(
+                compat_urlparse.urljoin(url, video.group('url')), VKIE.ie_key()))
+
+        title = 'Wall post %s' % post_id
+
+        return self.playlist_result(
+            orderedSet(entries), post_id,
+            '%s - %s' % (uploader, title) if uploader else title,
+            description)
--- a/youtube_dl/extractor/vuclip.py
+++ b/youtube_dl/extractor/vuclip.py
@@ -9,7 +9,7 @@ from ..compat import (
 from ..utils import (
    ExtractorError,
    parse_duration,
-    qualities,
+    remove_end,
 )


@@ -22,7 +22,7 @@ class VuClipIE(InfoExtractor):
            'id': '922692425',
            'ext': '3gp',
            'title': 'The Toy Soldiers - Hollywood Movie Trailer',
-            'duration': 180,
+            'duration': 177,
        }
    }

@@ -46,34 +46,21 @@ class VuClipIE(InfoExtractor):
                '%s said: %s' % (self.IE_NAME, error_msg), expected=True)

        # These clowns alternate between two page types
-        links_code = self._search_regex(
-            r'''(?xs)
-                (?:
-                    <img\s+src="[^"]*/play.gif".*?>|
-                    <!--\ player\ end\ -->\s*</div><!--\ thumb\ end-->
-                )
-                (.*?)
-                (?:
-                    <a\s+href="fblike|<div\s+class="social">
-                )
-            ''', webpage, 'links')
-        title = self._html_search_regex(
-            r'<title>(.*?)-\s*Vuclip</title>', webpage, 'title').strip()
+        video_url = self._search_regex(
+            r'<a[^>]+href="([^"]+)"[^>]*><img[^>]+src="[^"]*/play\.gif',
+            webpage, 'video URL', default=None)
+        if video_url:
+            formats = [{
+                'url': video_url,
+            }]
+        else:
+            formats = self._parse_html5_media_entries(url, webpage)[0]['formats']

-        quality_order = qualities(['Reg', 'Hi'])
-        formats = []
-        for url, q in re.findall(
-                r'<a\s+href="(?P<url>[^"]+)".*?>(?:<button[^>]*>)?(?P<q>[^<]+)(?:</button>)?</a>', links_code):
-            format_id = compat_urllib_parse_urlparse(url).scheme + '-' + q
-            formats.append({
-                'format_id': format_id,
-                'url': url,
-                'quality': quality_order(q),
-            })
-        self._sort_formats(formats)
+        title = remove_end(self._html_search_regex(
+            r'<title>(.*?)-\s*Vuclip</title>', webpage, 'title').strip(), ' - Video')

-        duration = parse_duration(self._search_regex(
-            r'\(([0-9:]+)\)</span>', webpage, 'duration', fatal=False))
+        duration = parse_duration(self._html_search_regex(
+            r'[(>]([0-9]+:[0-9]+)(?:<span|\))', webpage, 'duration', fatal=False))

        return {
            'id': video_id,
--- a/youtube_dl/extractor/wat.py
+++ b/youtube_dl/extractor/wat.py
@@ -9,7 +9,6 @@ from ..utils import (
    ExtractorError,
    unified_strdate,
    HEADRequest,
-    float_or_none,
 )


@@ -95,16 +94,7 @@ class WatIE(InfoExtractor):
            m3u8_url.replace('ios.', 'web.').replace('.m3u8', '.f4m'),
            video_id, f4m_id='hds', fatal=False))
        for m3u8_format in m3u8_formats:
-            mobj = re.search(
-                r'audio.*?%3D(\d+)(?:-video.*?%3D(\d+))?', m3u8_format['url'])
-            if not mobj:
-                continue
-            abr, vbr = mobj.groups()
-            abr, vbr = float_or_none(abr, 1000), float_or_none(vbr, 1000)
-            m3u8_format.update({
-                'vbr': vbr,
-                'abr': abr,
-            })
+            vbr, abr = m3u8_format.get('vbr'), m3u8_format.get('abr')
            if not vbr or not abr:
                continue
            f = m3u8_format.copy()
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -137,7 +137,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
        # Two-Factor
        # TODO add SMS and phone call support - these require making a request and then prompting the user

-        if re.search(r'(?i)<form[^>]* id="challenge"', login_results) is not None:
+        if re.search(r'(?i)<form[^>]+id="challenge"', login_results) is not None:
            tfa_code = self._get_tfa_info('2-step verification code')

            if not tfa_code:
@@ -165,17 +165,17 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
            if tfa_results is False:
                return False

-            if re.search(r'(?i)<form[^>]* id="challenge"', tfa_results) is not None:
+            if re.search(r'(?i)<form[^>]+id="challenge"', tfa_results) is not None:
                self._downloader.report_warning('Two-factor code expired or invalid. Please try again, or use a one-use backup code instead.')
                return False
-            if re.search(r'(?i)<form[^>]* id="gaia_loginform"', tfa_results) is not None:
+            if re.search(r'(?i)<form[^>]+id="gaia_loginform"', tfa_results) is not None:
                self._downloader.report_warning('unable to log in - did the page structure change?')
                return False
            if re.search(r'smsauth-interstitial-reviewsettings', tfa_results) is not None:
                self._downloader.report_warning('Your Google account has a security notice. Please log in on your web browser, resolve the notice, and try again.')
                return False

-        if re.search(r'(?i)<form[^>]* id="gaia_loginform"', login_results) is not None:
+        if re.search(r'(?i)<form[^>]+id="gaia_loginform"', login_results) is not None:
            self._downloader.report_warning('unable to log in: bad username or password')
            return False
        return True
@@ -858,6 +858,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        {
            'url': 'https://www.youtube.com/watch?feature=player_embedded&amp;amp;v=V36LpHqtcDY',
            'only_matching': True,
+        },
+        {
+            # YouTube Red paid video (https://github.com/rg3/youtube-dl/issues/10059)
+            'url': 'https://www.youtube.com/watch?v=i1Ko8UG-Tdo',
+            'only_matching': True,
        }
    ]

@@ -1978,10 +1983,13 @@ class YoutubeChannelIE(YoutubePlaylistBaseInfoExtractor):
        return (False if YoutubePlaylistsIE.suitable(url) or YoutubeLiveIE.suitable(url)
                else super(YoutubeChannelIE, cls).suitable(url))

+    def _build_template_url(self, url, channel_id):
+        return self._TEMPLATE_URL % channel_id
+
    def _real_extract(self, url):
        channel_id = self._match_id(url)

-        url = self._TEMPLATE_URL % channel_id
+        url = self._build_template_url(url, channel_id)

        # Channel by page listing is restricted to 35 pages of 30 items, i.e. 1050 videos total (see #5778)
        # Workaround by extracting as a playlist if managed to obtain channel playlist URL
@@ -2038,8 +2046,8 @@ class YoutubeChannelIE(YoutubePlaylistBaseInfoExtractor):

 class YoutubeUserIE(YoutubeChannelIE):
    IE_DESC = 'YouTube.com user videos (URL or "ytuser" keyword)'
-    _VALID_URL = r'(?:(?:https?://(?:\w+\.)?youtube\.com/(?:user/|c/)?(?!(?:attribution_link|watch|results)(?:$|[^a-z_A-Z0-9-])))|ytuser:)(?!feed/)(?P<id>[A-Za-z0-9_-]+)'
-    _TEMPLATE_URL = 'https://www.youtube.com/user/%s/videos'
+    _VALID_URL = r'(?:(?:https?://(?:\w+\.)?youtube\.com/(?:(?P<user>user|c)/)?(?!(?:attribution_link|watch|results)(?:$|[^a-z_A-Z0-9-])))|ytuser:)(?!feed/)(?P<id>[A-Za-z0-9_-]+)'
+    _TEMPLATE_URL = 'https://www.youtube.com/%s/%s/videos'
    IE_NAME = 'youtube:user'

    _TESTS = [{
@@ -2049,12 +2057,24 @@ class YoutubeUserIE(YoutubeChannelIE):
            'id': 'UUfX55Sx5hEFjoC3cNs6mCUQ',
            'title': 'Uploads from The Linux Foundation',
        }
+    }, {
+        # Only available via https://www.youtube.com/c/12minuteathlete/videos
+        # but not https://www.youtube.com/user/12minuteathlete/videos
+        'url': 'https://www.youtube.com/c/12minuteathlete/videos',
+        'playlist_mincount': 249,
+        'info_dict': {
+            'id': 'UUVjM-zV6_opMDx7WYxnjZiQ',
+            'title': 'Uploads from 12 Minute Athlete',
+        }
    }, {
        'url': 'ytuser:phihag',
        'only_matching': True,
    }, {
        'url': 'https://www.youtube.com/c/gametrailers',
        'only_matching': True,
+    }, {
+        'url': 'https://www.youtube.com/gametrailers',
+        'only_matching': True,
    }, {
        # This channel is not available.
        'url': 'https://www.youtube.com/user/kananishinoSMEJ/videos',
@@ -2071,6 +2091,10 @@ class YoutubeUserIE(YoutubeChannelIE):
        else:
            return super(YoutubeUserIE, cls).suitable(url)

+    def _build_template_url(self, url, channel_id):
+        mobj = re.match(self._VALID_URL, url)
+        return self._TEMPLATE_URL % (mobj.group('user') or 'user', mobj.group('id'))
+

 class YoutubeLiveIE(YoutubeBaseInfoExtractor):
    IE_DESC = 'YouTube.com live streams'
--- a/youtube_dl/postprocessor/ffmpeg.py
+++ b/youtube_dl/postprocessor/ffmpeg.py
@@ -363,8 +363,10 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
        input_files = [filename] + sub_filenames

        opts = [
-            '-map', '0',
-            '-c', 'copy',
+            '-map', '0:v',
+            '-c:v', 'copy',
+            '-map', '0:a',
+            '-c:a', 'copy',
            # Don't copy the existing subtitles, we may be running the
            # postprocessor a second time
            '-map', '-0:s',
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -2126,6 +2126,42 @@ def mimetype2ext(mt):
    }.get(res, res)


+def parse_codecs(codecs_str):
+    # http://tools.ietf.org/html/rfc6381
+    if not codecs_str:
+        return {}
+    splited_codecs = list(filter(None, map(
+        lambda str: str.strip(), codecs_str.strip().strip(',').split(','))))
+    vcodec, acodec = None, None
+    for full_codec in splited_codecs:
+        codec = full_codec.split('.')[0]
+        if codec in ('avc1', 'avc2', 'avc3', 'avc4', 'vp9', 'vp8', 'hev1', 'hev2', 'h263', 'h264', 'mp4v'):
+            if not vcodec:
+                vcodec = full_codec
+        elif codec in ('mp4a', 'opus', 'vorbis', 'mp3', 'aac'):
+            if not acodec:
+                acodec = full_codec
+        else:
+            write_string('WARNING: Unknown codec %s' % full_codec, sys.stderr)
+    if not vcodec and not acodec:
+        if len(splited_codecs) == 2:
+            return {
+                'vcodec': vcodec,
+                'acodec': acodec,
+            }
+        elif len(splited_codecs) == 1:
+            return {
+                'vcodec': 'none',
+                'acodec': vcodec,
+            }
+    else:
+        return {
+            'vcodec': vcodec or 'none',
+            'acodec': acodec or 'none',
+        }
+    return {}
+
+
 def urlhandle_detect_ext(url_handle):
    getheader = url_handle.headers.get

--- a/youtube_dl/version.py
+++ b/youtube_dl/version.py
@@ -1,3 +1,3 @@
 from __future__ import unicode_literals

-__version__ = '2016.07.09'
+__version__ = '2016.07.13'
Author	SHA1	Message	Date
Sergey M․	c485959034	release 2016.07.13	2016-07-13 23:58:01 +07:00
Sergey M․	a0560d8ab8	[ellentv] Improve extraction (Closes #10067 )	2016-07-13 22:42:53 +07:00
Remita Amine	0385aa6199	[bbc] extract more and better qulities from Unified Streaming Platform m3u8 manifests	2016-07-13 15:58:24 +01:00
Remita Amine	00f4764cb7	[common] extract vbr, abr and fps for Unified Streaming Platform m3u8 manifests	2016-07-13 15:58:24 +01:00
Sergey M․	51c2cd0b83	[extractors] Add vk:wallpost extractor import	2016-07-13 21:53:23 +07:00
Sergey M․	5f5a9d6158	[vk] Improve login	2016-07-13 21:52:52 +07:00
Sergey M․	2d19fb5072	[vk:wallpost] Add extractor	2016-07-13 21:51:44 +07:00
Yen Chi Hsuan	9d865a1af6	[travis] Skip downloading srelay SOCKS tests never run on Travis CI due to unknown reasons, and downloading them broke some tests (e.g. https://travis-ci.org/rg3/youtube-dl/builds/144306425)	2016-07-13 14:27:14 +08:00
Remita Amine	41aa44259d	[shahid] try to bypass geo restriction and extract more metadata(closes #10062 )	2016-07-12 23:15:38 +01:00
Philipp Hagemeister	381ff44756	[devscripts/generate-download] Remove MD5 and SHA1	2016-07-12 09:09:54 +02:00
Sergey M․	7f29cf545a	[youtube] Add YouTube Red paid video reference test (#10059 )	2016-07-12 02:10:35 +07:00
Remita Amine	7d1219f3e0	[tmz] delegate extraction to KalturaIE	2016-07-11 19:08:22 +01:00
Remita Amine	f1b4af7d79	[beightcove:new] remove html tags from description	2016-07-11 19:06:50 +01:00
Remita Amine	8a8590a617	[dbtv] delegate extraction to BrightcoveNewIE	2016-07-11 16:30:24 +01:00
Remita Amine	4a7a5e41f7	[tvplay] improve extraction	2016-07-11 14:51:44 +01:00
Yen Chi Hsuan	2a49d01600	[playvid] Update _TESTS Blocks https://travis-ci.org/rg3/youtube-dl/jobs/143809100	2016-07-11 15:15:28 +08:00
Yen Chi Hsuan	b99af8a51c	[biobiochiletv] Fix extraction and update _TESTS	2016-07-11 13:23:57 +08:00
Yen Chi Hsuan	8e7020daef	[rudo] Add new extractor Used in biobiochile.tv	2016-07-11 13:19:25 +08:00
Sergey M․	a26bcc61c1	release 2016.07.11	2016-07-11 03:17:12 +07:00
Sergey M․	5c4dcf8172	[vidzi] Add support for embed URLs (Closes #10058 )	2016-07-11 03:14:39 +07:00
Sergey M․	e9fb6a4bbe	[youtube] Relax TFA regexes	2016-07-11 03:08:38 +07:00
Yen Chi Hsuan	e2dbcaa1bf	[vuclip] Fix extraction	2016-07-11 00:52:25 +08:00
Yen Chi Hsuan	ae01850165	[miomio] Fix _TESTS	2016-07-11 00:03:24 +08:00
Yen Chi Hsuan	c3baaedfc8	[miomio] Support new 'h5' player (closes #9605 ) Depends on #8876	2016-07-10 23:46:48 +08:00
Yen Chi Hsuan	0b68de3cc1	Merge pull request #8876 from remitamine/html5_media [extractor/common] add helper method to extract html5 media entries	2016-07-10 23:40:45 +08:00
Sergey M․	39e9d524e5	Credit @nehalvpatel for roosterteeth (#9864 )	2016-07-10 01:30:12 +07:00
Sergey M․	865b087224	[roosterteeth] Improve (Closes #9864 )	2016-07-10 01:30:12 +07:00
Nehal Patel	3121b25639	[roosterteeth] Add extractor	2016-07-10 01:30:12 +07:00
Sergey M․	0286b85c79	release 2016.07.09.2	2016-07-09 22:22:24 +07:00
Sergey M․	ab52bb5137	[animeondemand] Fix typo	2016-07-09 22:20:34 +07:00
Sergey M․	61a98b8623	[lynda] Remove md5 from test (Closes #10047 )	2016-07-09 21:29:11 +07:00
Sergey M․	6daf34a045	[facebook] Fix typo and break when found video_data (Closes #10048 )	2016-07-09 21:25:07 +07:00
Yen Chi Hsuan	c03adf90bd	[generic] Add the test. Closes #1638	2016-07-09 14:39:01 +08:00
Yen Chi Hsuan	0ece114b7b	[vimeo] Recognize non-standard embeds (#1638 )	2016-07-09 14:38:27 +08:00
Yen Chi Hsuan	5b6a74856b	Merge pull request #9288 from reyyed/issue#9063fix [ffmpeg] Fix embedding subtitles (#9063)	2016-07-09 14:29:53 +08:00
Sergey M․	ce43100a01	release 2016.07.09.1	2016-07-09 10:06:40 +07:00
Remita Amine	8cc9b4016d	[srmediathek] extend _VALID_URL(closes #9373 )	2016-07-09 03:22:09 +01:00
Remita Amine	31eeab9f41	[ard] fix f4m extraction and skip tests with 404 errors	2016-07-09 03:22:09 +01:00
Sergey M․	9558dcec9c	[youtube:user] Preserve user/c path segment	2016-07-09 08:37:19 +07:00
Sergey M․	6e6b70d65f	[extractor/generic] Properly comment out a test	2016-07-09 08:37:19 +07:00
remitamine	59bbe4911a	[extractor/common] add helper method to extract html5 media entries	2016-06-26 14:04:08 +01:00
remitamine	4f3c5e0627	[utils] add helper function for parsing codecs	2016-06-26 14:03:58 +01:00
Wang Jun Tham	ccff2c404d	[ffmpeg] Fix embedding subtitles (#9063 ) Changed command line parameters for ffmpeg when embedding subtitles. Changed to ‘-map 0:v -c:v copy -map 0:a -c:a copy’	2016-04-24 00:08:02 +08:00