release 2016.06.05

.github/ISSUE_TEMPLATE.md

@@ -6,8 +6,8 @@
 
 ---
 
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.09.04.1*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
-- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.09.04.1**
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.06.05*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.06.05**
 
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
@@ -35,7 +35,7 @@ $ youtube-dl -v <your command line>
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2016.09.04.1
+[debug] youtube-dl version 2016.06.05
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,22 +0,0 @@
-## Please follow the guide below
-
-- You will be asked some questions, please read them **carefully** and answer honestly
-- Put an `x` into all the boxes [ ] relevant to your *pull request* (like that [x])
-- Use *Preview* tab to see how your *pull request* will actually look like
-
----
-
-### Before submitting a *pull request* make sure you have:
-- [ ] At least skimmed through [adding new extractor tutorial](https://github.com/rg3/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/rg3/youtube-dl#youtube-dl-coding-conventions) sections
-- [ ] [Searched](https://github.com/rg3/youtube-dl/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
-
-### What is the purpose of your *pull request*?
-- [ ] Bug fix
-- [ ] New extractor
-- [ ] New feature
-
----
-
-### Description of your *pull request* and other information
-
-Explanation of your *pull request* in arbitrary form goes here. Please make sure the description explains the purpose and effect of your *pull request* and is worded well enough to be understood. Provide as much context and examples as possible.

.travis.yml

@@ -7,6 +7,9 @@ python:
   - "3.4"
   - "3.5"
 sudo: false
+install:
+  - bash ./devscripts/install_srelay.sh
+  - export PATH=$PATH:$(pwd)/tmp/srelay-0.4.8b6
 script: nosetests test --verbose
 notifications:
   email:

AUTHORS

@@ -173,13 +173,3 @@ Kevin Deldycke
 inondle
 Tomáš Čech
 Déstin Reed
-Roman Tsiupa
-Artur Krysiak
-Jakub Adam Wieczorek
-Aleksandar Topuzović
-Nehal Patel
-Rob van Bekkum
-Petr Zvoníček
-Pratyush Singh
-Aleksander Nitecki
-Sebastian Blunt

CONTRIBUTING.md

@@ -46,7 +46,7 @@ Make sure that someone has not already opened the issue you're trying to open. S
 
 ###  Why are existing options not enough?
 
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#synopsis). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
 
 ###  Is there enough context in your bug report?
 
@@ -97,17 +97,9 @@ If you want to add support for a new site, first of all **make sure** this site
 After you have ensured this site is distributing it's content legally, you can follow this quick list (assuming your service is called `yourextractor`):
 
 1. [Fork this repository](https://github.com/rg3/youtube-dl/fork)
-2. Check out the source code with:
-
-        git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git
-
-3. Start a new git branch with
-
-        cd youtube-dl
-        git checkout -b yourextractor
-
+2. Check out the source code with `git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git`
+3. Start a new git branch with `cd youtube-dl; git checkout -b yourextractor`
 4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`:
-
     ```python
     # coding: utf-8
     from __future__ import unicode_literals
@@ -150,149 +142,17 @@ After you have ensured this site is distributing it's content legally, you can f
     ```
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
+7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L68-L226). Add tests and code for as many as you want.
+8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L138-L226) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
+9. Check the code with [flake8](https://pypi.python.org/pypi/flake8).
+10. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
 
         $ git add youtube_dl/extractor/extractors.py
         $ git add youtube_dl/extractor/yourextractor.py
         $ git commit -m '[yourextractor] Add new extractor'
         $ git push origin yourextractor
 
-10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
-## youtube-dl coding conventions
-
-This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
-
-Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hoster out of your control and this layout tend to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize code dependency on source's layout changes and even to make the code foresee potential future changes and be ready for that. This is important because it will allow extractor not to break on minor layout changes thus keeping old youtube-dl versions working. Even though this breakage issue is easily fixed by emitting a new version of youtube-dl with fix incorporated all the previous version become broken in all repositories and distros' packages that may not be so prompt in fetching the update from us. Needless to say some may never receive an update at all that is possible for non rolling release distros.
-
-### Mandatory and optional metafields
-
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by [information dictionary](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L75-L257) or simply *info dict*. Only the following meta fields in *info dict* are considered mandatory for successful extraction process by youtube-dl:
-
- - `id` (media identifier)
- - `title` (media title)
- - `url` (media download URL) or `formats`
-
-In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` to be mandatory. Thus aforementioned metafields are the critical data the extraction does not make any sense without and if any of them fail to be extracted then extractor is considered completely broken.
-
-[Any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L149-L257) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
-
-#### Example
-
-Say you have some source dictionary `meta` that you've fetched as JSON with HTTP request and it has a key `summary`:
-
-```python
-meta = self._download_json(url, video_id)
-```
-    
-Assume at this point `meta`'s layout is:
-
-```python
-{
-    ...
-    "summary": "some fancy summary text",
-    ...
-}
-```
-
-Assume you want to extract `summary` and put into resulting info dict as `description`. Since `description` is optional metafield you should be ready that this key may be missing from the `meta` dict, so that you should extract it like:
-
-```python
-description = meta.get('summary')  # correct
-```
-
-and not like:
-
-```python
-description = meta['summary']  # incorrect
-```
-
-The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some time later but with former approach extraction will just go ahead with `description` set to `None` that is perfectly fine (remember `None` is equivalent for absence of data). 
-
-Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance:
-
-```python
-description = self._search_regex(
-    r'<span[^>]+id="title"[^>]*>([^<]+)<',
-    webpage, 'description', fatal=False)
-```
-
-With `fatal` set to `False` if `_search_regex` fails to extract `description` it will emit a warning and continue extraction.
-
-You can also pass `default=<some fallback value>`, for example:
-
-```python
-description = self._search_regex(
-    r'<span[^>]+id="title"[^>]*>([^<]+)<',
-    webpage, 'description', default=None)
-```
-
-On failure this code will silently continue the extraction with `description` set to `None`. That is useful for metafields that are known to may or may not be present.
- 
-### Provide fallbacks
-
-When extracting metadata try to provide several scenarios for that. For example if `title` is present in several places/sources try extracting from at least some of them. This would make it more future-proof in case some of the sources became unavailable.
-
-#### Example
-
-Say `meta` from previous example has a `title` and you are about to extract it. Since `title` is mandatory meta field you should end up with something like:
-
-```python
-title = meta['title']
-```
-
-If `title` disappeares from `meta` in future due to some changes on hoster's side the extraction would fail since `title` is mandatory. That's expected.
-
-Assume that you have some another source you can extract `title` from, for example `og:title` HTML meta of a `webpage`. In this case you can provide a fallback scenario:
-
-```python
-title = meta.get('title') or self._og_search_title(webpage)
-```
-
-This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
-
-### Make regular expressions flexible
-
-When using regular expressions try to write them fuzzy and flexible.
- 
-#### Example
-
-Say you need to extract `title` from the following HTML code:
-
-```html
-<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">some fancy title</span>
-```
-
-The code for that task should look similar to:
-
-```python
-title = self._search_regex(
-    r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title')
-```
-
-Or even better:
-
-```python
-title = self._search_regex(
-    r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)',
-    webpage, 'title', group='title')
-```
-
-Note how you tolerate potential changes in `style` attribute's value or switch from using double quotes to single for `class` attribute: 
-
-The code definitely should not look like:
-
-```python
-title = self._search_regex(
-    r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>',
-    webpage, 'title', group='title')
-```
-
-### Use safe conversion functions
-
-Wrap all extracted numeric data into safe functions from `utils`: `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
-

ChangeLog

@@ -1,587 +0,0 @@
-version 2016.09.04.1
-
-Core
-* In DASH downloader if the first segment fails, abort the whole download
-  process to prevent throttling (#10497)
-+ Add support for --skip-unavailable-fragments and --fragment retries in
-  hlsnative downloader (#10165, #10448).
-+ Add support for --skip-unavailable-fragments in DASH downloader
-+ Introduce --skip-unavailable-fragments option for fragment based downloaders
-  that allows to skip fragments unavailable due to a HTTP error
-* Fix extraction of video/audio entries with src attribute in
-  _parse_html5_media_entries (#10540)
-
-Extractors
-* [theplatform] Relax URL regular expression (#10546)
-* [youtube:playlist] Extend URL regular expression
-* [rottentomatoes] Delegate extraction to internetvideoarchive extractor
-* [internetvideoarchive] Extract all formats
-* [pornvoisines] Fix extraction (#10469)
-* [rottentomatoes] Fix extraction (#10467)
-* [espn] Extend URL regular expression (#10549)
-* [vimple] Extend URL regular expression (#10547)
-* [youtube:watchlater] Fix extraction (#10544)
-* [youjizz] Fix extraction (#10437)
-+ [foxnews] Add support for FoxNews Insider (#10445)
-+ [fc2] Recognize Flash player URLs (#10512)
-
-
-version 2016.09.03
-
-Core
-* Restore usage of NAME attribute from EXT-X-MEDIA tag for formats codes in
-  _extract_m3u8_formats (#10522)
-* Handle semicolon in mimetype2ext
-
-Extractors
-+ [youtube] Add support for rental videos' previews (#10532)
-* [youtube:playlist] Fallback to video extraction for video/playlist URLs when
-  no playlist is actually served (#10537)
-+ [drtv] Add support for dr.dk/nyheder (#10536)
-+ [facebook:plugins:video] Add extractor (#10530)
-+ [go] Add extractor for *.go.com sites
-* [adobepass] Check for authz_token expiration (#10527)
-* [nytimes] improve extraction
-* [thestar] Fix extraction (#10465)
-* [glide] Fix extraction (#10478)
-- [exfm] Remove extractor (#10482)
-* [youporn] Fix categories and tags extraction (#10521)
-+ [curiositystream] Add extractor for app.curiositystream.com
-- [thvideo] Remove extractor (#10464)
-* [movingimage] Fix for the new site name (#10466)
-+ [cbs] Add support for once formats (#10515)
-* [limelight] Skip ism snd duplicate manifests
-+ [porncom] Extract categories and tags (#10510)
-+ [facebook] Extract timestamp (#10508)
-+ [yahoo] Extract more formats
-
-
-version 2016.08.31
-
-Extractors
-* [soundcloud] Fix URL regular expression to avoid clashes with sets (#10505)
-* [bandcamp:album] Fix title extraction (#10455)
-* [pyvideo] Fix extraction (#10468)
-+ [ctv] Add support for tsn.ca, bnn.ca and thecomedynetwork.ca (#10016)
-* [9c9media] Extract more metadata
-* [9c9media] Fix multiple stacks extraction (#10016)
-* [adultswim] Improve video info extraction (#10492)
-* [vodplatform] Improve embed regular expression
-- [played] Remove extractor (#10470)
-+ [tbs] Add extractor for tbs.com and tntdrama.com (#10222)
-+ [cartoonnetwork] Add extractor for cartoonnetwork.com (#10110)
-* [adultswim] Rework in terms of turner extractor
-* [cnn] Rework in terms of turner extractor
-* [nba] Rework in terms of turner extractor
-+ [turner] Add base extractor for Turner Broadcasting System based sites
-* [bilibili] Fix extraction (#10375)
-* [openload] Fix extraction (#10408)
-
-
-version 2016.08.28
-
-Core
-+ Add warning message that ffmpeg doesn't support SOCKS
-* Improve thumbnail sorting
-+ Extract formats from #EXT-X-MEDIA tags in _extract_m3u8_formats
-* Fill IV with leading zeros for IVs shorter than 16 octets in hlsnative
-+ Add ac-3 to the list of audio codecs in parse_codecs
-
-Extractors
-* [periscope:user] Fix extraction (#10453)
-* [douyutv] Fix extraction (#10153, #10318, #10444)
-+ [nhk:vod] Add extractor for www3.nhk.or.jp on demand (#4437, #10424)
-- [trutube] Remove extractor (#10438)
-+ [usanetwork] Add extractor for usanetwork.com
-* [crackle] Fix extraction (#10333)
-* [spankbang] Fix description and uploader extraction (#10339)
-* [discoverygo] Detect cable provider restricted videos (#10425)
-+ [cbc] Add support for watch.cbc.ca
-* [kickstarter] Silent the warning for og:description (#10415)
-* [mtvservices:embedded] Fix extraction for the new 'edge' player (#10363)
-
-
-version 2016.08.24.1
-
-Extractors
-+ [pluralsight] Add support for subtitles (#9681)
-
-
-version 2016.08.24
-
-Extractors
-* [youtube] Fix authentication (#10392)
-* [openload] Fix extraction (#10408)
-+ [bravotv] Add support for Adobe Pass (#10407)
-* [bravotv] Fix clip info extraction (#10407)
-* [eagleplatform] Improve embedded videos detection (#10409)
-* [awaan] Fix extraction
-* [mtvservices:embedded] Update config URL
-+ [abc:iview] Add extractor (#6148)
-
-
-version 2016.08.22
-
-Core
-* Improve formats and subtitles extension auto calculation
-+ Recognize full unit names in parse_filesize
-+ Add support for m3u8 manifests in HTML5 multimedia tags
-* Fix octal/hexadecimal number detection in js_to_json
-
-Extractors
-+ [ivi] Add support for 720p and 1080p
-+ [charlierose] Add new extractor (#10382)
-* [1tv] Fix extraction (#9249)
-* [twitch] Renew authentication
-* [kaltura] Improve subtitles extension calculation
-+ [zingmp3] Add support for video clips
-* [zingmp3] Fix extraction (#10041)
-* [kaltura] Improve subtitles extraction (#10279)
-* [cultureunplugged] Fix extraction (#10330)
-+ [cnn] Add support for money.cnn.com (#2797)
-* [cbsnews] Fix extraction (#10362)
-* [cbs] Fix extraction (#10393)
-+ [litv] Support 'promo' URLs (#10385)
-* [snotr] Fix extraction (#10338)
-* [n-tv.de] Fix extraction (#10331)
-* [globo:article] Relax URL and video id regular expressions (#10379)
-
-
-version 2016.08.19
-
-Core
-- Remove output template description from --help
-* Recognize lowercase units in parse_filesize
-
-Extractors
-+ [porncom] Add extractor for porn.com (#2251, #10251)
-+ [generic] Add support for DBTV embeds
-* [vk:wallpost] Fix audio extraction for new site layout
-* [vk] Fix authentication
-+ [hgtvcom:show] Add extractor for hgtv.com shows (#10365)
-+ [discoverygo] Add support for another GO network sites
-
-
-version 2016.08.17
-
-Core
-+ Add _get_netrc_login_info
-
-Extractors
-* [mofosex] Extract all formats (#10335)
-+ [generic] Add support for vbox7 embeds
-+ [vbox7] Add support for embed URLs
-+ [viafree] Add extractor (#10358)
-+ [mtg] Add support for viafree URLs (#10358)
-* [theplatform] Extract all subtitles per language
-+ [xvideos] Fix HLS extraction (#10356)
-+ [amcnetworks] Add extractor
-+ [bbc:playlist] Add support for pagination (#10349)
-+ [fxnetworks] Add extractor (#9462)
-* [cbslocal] Fix extraction for SendtoNews-based videos
-* [sendtonews] Fix extraction
-* [jwplatform] Extract video id from JWPlayer data
-- [zippcast] Remove extractor (#10332)
-+ [viceland] Add extractor (#8799)
-+ [adobepass] Add base extractor for Adobe Pass Authentication
-* [life:embed] Improve extraction
-* [vgtv] Detect geo restricted videos (#10348)
-+ [uplynk] Add extractor
-* [xiami] Fix extraction (#10342)
-
-
-version 2016.08.13
-
-Core
-* Show progress for curl external downloader
-* Forward more options to curl external downloader
-
-Extractors
-* [pbs] Fix description extraction
-* [franceculture] Fix extraction (#10324)
-* [pornotube] Fix extraction (#10322)
-* [4tube] Fix metadata extraction (#10321)
-* [imgur] Fix width and height extraction (#10325)
-* [expotv] Improve extraction
-+ [vbox7] Fix extraction (#10309)
-- [tapely] Remove extractor (#10323)
-* [muenchentv] Fix extraction (#10313)
-+ [24video] Add support for .me and .xxx TLDs
-* [24video] Fix comment count extraction
-* [sunporno] Add support for embed URLs
-* [sunporno] Fix metadata extraction (#10316)
-+ [hgtv] Add extractor for hgtv.ca (#3999)
-- [pbs] Remove request to unavailable API
-+ [pbs] Add support for high quality HTTP formats
-+ [crunchyroll] Add support for HLS formats (#10301)
-
-
-version 2016.08.12
-
-Core
-* Subtitles are now written as is. Newline conversions are disabled. (#10268)
-+ Recognize more formats in unified_timestamp
-
-Extractors
-- [goldenmoustache] Remove extractor (#10298)
-* [drtuber] Improve title extraction
-* [drtuber] Make dislike count optional (#10297)
-* [chirbit] Fix extraction (#10296)
-* [francetvinfo] Relax URL regular expression
-* [rtlnl] Relax URL regular expression (#10282)
-* [formula1] Relax URL regular expression (#10283)
-* [wat] Improve extraction (#10281)
-* [ctsnews] Fix extraction
-
-
-version 2016.08.10
-
-Core
-* Make --metadata-from-title non fatal when title does not match the pattern
-* Introduce options for randomized sleep before each download
-  --min-sleep-interval and --max-sleep-interval (#9930)
-* Respect default in _search_json_ld
-
-Extractors
-+ [uol] Add extractor for uol.com.br (#4263)
-* [rbmaradio] Fix extraction and extract all formats (#10242)
-+ [sonyliv] Add extractor for sonyliv.com (#10258)
-* [aparat] Fix extraction
-* [cwtv] Extract HTTP formats
-+ [rozhlas] Add extractor for prehravac.rozhlas.cz (#10253)
-* [kuwo:singer] Fix extraction
-
-
-version 2016.08.07
-
-Core
-+ Add support for TV Parental Guidelines ratings in parse_age_limit
-+ Add decode_png (#9706)
-+ Add support for partOfTVSeries in JSON-LD
-* Lower master M3U8 manifest preference for better format sorting
-
-Extractors
-+ [discoverygo] Add extractor (#10245)
-* [flipagram] Make JSON-LD extraction non fatal
-* [generic] Make JSON-LD extraction non fatal
-+ [bbc] Add support for morph embeds (#10239)
-* [tnaflixnetworkbase] Improve title extraction
-* [tnaflix] Fix metadata extraction (#10249)
-* [fox] Fix theplatform release URL query
-* [openload] Fix extraction (#9706)
-* [bbc] Skip duplicate manifest URLs
-* [bbc] Improve format code
-+ [bbc] Add support for DASH and F4M
-* [bbc] Improve format sorting and listing
-* [bbc] Improve playlist extraction
-+ [pokemon] Add extractor (#10093)
-+ [condenast] Add fallback scenario for video info extraction
-
-
-version 2016.08.06
-
-Core
-* Add support for JSON-LD root list entries (#10203)
-* Improve unified_timestamp
-* Lower preference of RTSP formats in generic sorting
-+ Add support for multiple properties in _og_search_property
-* Improve password hiding from verbose output
-
-Extractors
-+ [adultswim] Add support for trailers (#10235)
-* [archiveorg] Improve extraction (#10219)
-+ [jwplatform] Add support for playlists
-+ [jwplatform] Add support for relative URLs
-* [jwplatform] Improve audio detection
-+ [tvplay] Capture and output native error message
-+ [tvplay] Extract series metadata
-+ [tvplay] Add support for subtitles (#10194)
-* [tvp] Improve extraction (#7799)
-* [cbslocal] Fix timestamp parsing (#10213)
-+ [naver] Add support for subtitles (#8096)
-* [naver] Improve extraction
-* [condenast] Improve extraction
-* [engadget] Relax URL regular expression
-* [5min] Fix extraction
-+ [nationalgeographic] Add support for Episode Guide
-+ [kaltura] Add support for subtitles
-* [kaltura] Optimize network requests
-+ [vodplatform] Add extractor for vod-platform.net
-- [gamekings] Remove extractor
-* [limelight] Extract HTTP formats
-* [ntvru] Fix extraction
-+ [comedycentral] Re-add :tds and :thedailyshow shortnames
-
-
-version 2016.08.01
-
-Fixed/improved extractors
-- [yandexmusic:track] Adapt to changes in track location JSON (#10193)
-- [bloomberg] Support another form of player (#10187)
-- [limelight] Skip DRM protected videos
-- [safari] Relax regular expressions for URL matching (#10202)
-- [cwtv] Add support for cwtvpr.com (#10196)
-
-
-version 2016.07.30
-
-Fixed/improved extractors
-- [twitch:clips] Sort formats
-- [tv2] Use m3u8_native
-- [tv2:article] Fix video detection (#10188)
-- rtve (#10076)
-- [dailymotion:playlist] Optimize download archive processing (#10180)
-
-
-version 2016.07.28
-
-Fixed/improved extractors
-- shared (#10170)
-- soundcloud (#10179)
-- twitch (#9767)
-
-
-version 2016.07.26.2
-
-Fixed/improved extractors
-- smotri
-- camdemy
-- mtv
-- comedycentral
-- cmt
-- cbc
-- mgtv
-- orf
-
-
-version 2016.07.24
-
-New extractors
-- arkena (#8682)
-- lcp (#8682)
-
-Fixed/improved extractors
-- facebook (#10151)
-- dailymail
-- telegraaf
-- dcn
-- onet
-- tvp
-
-Miscellaneous
-- Support $Time$ in DASH manifests
-
-
-version 2016.07.22
-
-New extractors
-- odatv (#9285)
-
-Fixed/improved extractors
-- bbc
-- youjizz (#10131)
-- youtube (#10140)
-- pornhub (#10138)
-- eporner (#10139)
-
-
-version 2016.07.17
-
-New extractors
-- nintendo (#9986)
-- streamable (#9122)
-
-Fixed/improved extractors
-- ard (#10095)
-- mtv
-- comedycentral (#10101)
-- viki (#10098)
-- spike (#10106)
-
-Miscellaneous
-- Improved twitter player detection (#10090)
-
-
-version 2016.07.16
-
-New extractors
-- ninenow (#5181)
-
-Fixed/improved extractors
-- rtve (#10076)
-- brightcove
-- 3qsdn
-- syfy (#9087, #3820, #2388)
-- youtube (#10083)
-
-Miscellaneous
-- Fix subtitle embedding for video-only and audio-only files (#10081)
-
-
-version 2016.07.13
-
-New extractors
-- rudo
-
-Fixed/improved extractors
-- biobiochiletv
-- tvplay
-- dbtv
-- brightcove
-- tmz
-- youtube (#10059)
-- shahid (#10062)
-- vk
-- ellentv (#10067)
-
-
-version 2016.07.11
-
-New Extractors
-- roosterteeth (#9864)
-
-Fixed/improved extractors
-- miomio (#9605)
-- vuclip
-- youtube
-- vidzi (#10058)
-
-
-version 2016.07.09.2
-
-Fixed/improved extractors
-- vimeo (#1638)
-- facebook (#10048)
-- lynda (#10047)
-- animeondemand
-
-Fixed/improved features
-- Embedding subtitles no longer throws an error with problematic inputs (#9063)
-
-
-version 2016.07.09.1
-
-Fixed/improved extractors
-- youtube
-- ard
-- srmediatek (#9373)
-
-
-version 2016.07.09
-
-New extractors
-- Flipagram (#9898)
-
-Fixed/improved extractors
-- telecinco
-- toutv
-- radiocanada
-- tweakers (#9516)
-- lynda
-- nick (#7542)
-- polskieradio (#10028)
-- le
-- facebook (#9851)
-- mgtv
-- animeondemand (#10031)
-
-Fixed/improved features
-- `--postprocessor-args` and `--downloader-args` now accepts non-ASCII inputs
-  on non-Windows systems
-
-
-version 2016.07.07
-
-New extractors
-- kamcord (#10001)
-
-Fixed/improved extractors
-- spiegel (#10018)
-- metacafe (#8539, #3253)
-- onet (#9950)
-- francetv (#9955)
-- brightcove (#9965)
-- daum (#9972)
-
-
-version 2016.07.06
-
-Fixed/improved extractors
-- youtube (#10007, #10009)
-- xuite
-- stitcher
-- spiegel
-- slideshare
-- sandia
-- rtvnh
-- prosiebensat1
-- onionstudios
-
-
-version 2016.07.05
-
-Fixed/improved extractors
-- brightcove
-- yahoo (#9995)
-- pornhub (#9997)
-- iqiyi
-- kaltura (#5557)
-- la7
-- Changed features
-- Rename --cn-verfication-proxy to --geo-verification-proxy
-Miscellaneous
-- Add script for displaying downloads statistics
-
-
-version 2016.07.03.1
-
-Fixed/improved extractors
-- theplatform
-- aenetworks
-- nationalgeographic
-- hrti (#9482)
-- facebook (#5701)
-- buzzfeed (#5701)
-- rai (#8617, #9157, #9232, #8552, #8551)
-- nationalgeographic (#9991)
-- iqiyi
-
-
-version 2016.07.03
-
-New extractors
-- hrti (#9482)
-
-Fixed/improved extractors
-- vk (#9981)
-- facebook (#9938)
-- xtube (#9953, #9961)
-
-
-version 2016.07.02
-
-New extractors
-- fusion (#9958)
-
-Fixed/improved extractors
-- twitch (#9975)
-- vine (#9970)
-- periscope (#9967)
-- pornhub (#8696)
-
-
-version 2016.07.01
-
-New extractors
-- 9c9media
-- ctvnews (#2156)
-- ctv (#4077)
-
-Fixed/Improved extractors
-- rds
-- meta (#8789)
-- pornhub (#9964)
-- sixplay (#2183)
-
-New features
-- Accept quoted strings across multiple lines (#9940)

Makefile

@@ -94,7 +94,7 @@ _EXTRACTOR_FILES != find youtube_dl/extractor -iname '*.py' -and -not -iname 'la
 youtube_dl/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
 	$(PYTHON) devscripts/make_lazy_extractors.py $@
 
-youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish ChangeLog
+youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
 	@tar -czf youtube-dl.tar.gz --transform "s|^|youtube-dl/|" --owner 0 --group 0 \
 		--exclude '*.DS_Store' \
 		--exclude '*.kate-swp' \
@@ -107,7 +107,7 @@ youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-
 		--exclude 'docs/_build' \
 		-- \
 		bin devscripts test youtube_dl docs \
-		ChangeLog LICENSE README.md README.txt \
+		LICENSE README.md README.txt \
 		Makefile MANIFEST.in youtube-dl.1 youtube-dl.bash-completion \
 		youtube-dl.zsh youtube-dl.fish setup.py \
 		youtube-dl

README.md

@@ -17,7 +17,7 @@ youtube-dl - download videos from youtube.com or other video platforms
 
 To install it right away for all UNIX users (Linux, OS X, etc.), type:
 
-    sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
+    sudo curl https://yt-dl.org/latest/youtube-dl -o /usr/local/bin/youtube-dl
     sudo chmod a+rx /usr/local/bin/youtube-dl
 
 If you do not have curl, you can alternatively use a recent wget:
@@ -27,24 +27,18 @@ If you do not have curl, you can alternatively use a recent wget:
 
 Windows users can [download an .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in any location on their [PATH](http://en.wikipedia.org/wiki/PATH_%28variable%29) except for `%SYSTEMROOT%\System32` (e.g. **do not** put in `C:\Windows\System32`).
 
-You can also use pip:
-
-    sudo pip install --upgrade youtube-dl
-    
-This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
-
-OS X users can install youtube-dl with [Homebrew](http://brew.sh/):
+OS X users can install **youtube-dl** with [Homebrew](http://brew.sh/).
 
     brew install youtube-dl
 
-Or with [MacPorts](https://www.macports.org/):
+You can also use pip:
 
-    sudo port install youtube-dl
+    sudo pip install youtube-dl
 
 Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see the [youtube-dl Download Page](https://rg3.github.io/youtube-dl/download.html).
 
 # DESCRIPTION
-**youtube-dl** is a command-line program to download videos from
+**youtube-dl** is a small command-line program to download videos from
 YouTube.com and a few more sites. It requires the Python interpreter, version
 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on
 your Unix box, on Windows or on Mac OS X. It is released to the public domain,
@@ -89,8 +83,6 @@ which means you can modify it, redistribute it or use it however you like.
     --mark-watched                   Mark videos watched (YouTube only)
     --no-mark-watched                Do not mark videos watched (YouTube only)
     --no-color                       Do not emit color codes in output
-    --abort-on-unavailable-fragment  Abort downloading when some fragment is not
-                                     available
 
 ## Network Options:
     --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
@@ -105,9 +97,9 @@ which means you can modify it, redistribute it or use it however you like.
                                      (experimental)
     -6, --force-ipv6                 Make all connections via IPv6
                                      (experimental)
-    --geo-verification-proxy URL     Use this proxy to verify the IP address for
-                                     some geo-restricted sites. The default
-                                     proxy specified by --proxy (or none, if the
+    --cn-verification-proxy URL      Use this proxy to verify the IP address for
+                                     some Chinese sites. The default proxy
+                                     specified by --proxy (or none, if the
                                      options is not present) is used for the
                                      actual downloading. (experimental)
 
@@ -175,10 +167,7 @@ which means you can modify it, redistribute it or use it however you like.
     -R, --retries RETRIES            Number of retries (default is 10), or
                                      "infinite".
     --fragment-retries RETRIES       Number of retries for a fragment (default
-                                     is 10), or "infinite" (DASH and hlsnative
-                                     only)
-    --skip-unavailable-fragments     Skip unavailable fragments (DASH and
-                                     hlsnative only)
+                                     is 10), or "infinite" (DASH only)
     --buffer-size SIZE               Size of download buffer (e.g. 1024 or 16K)
                                      (default is 1024)
     --no-resize-buffer               Do not automatically adjust the buffer
@@ -206,8 +195,32 @@ which means you can modify it, redistribute it or use it however you like.
     -a, --batch-file FILE            File containing URLs to download ('-' for
                                      stdin)
     --id                             Use only video ID in file name
-    -o, --output TEMPLATE            Output filename template, see the "OUTPUT
-                                     TEMPLATE" for all the info
+    -o, --output TEMPLATE            Output filename template. Use %(title)s to
+                                     get the title, %(uploader)s for the
+                                     uploader name, %(uploader_id)s for the
+                                     uploader nickname if different,
+                                     %(autonumber)s to get an automatically
+                                     incremented number, %(ext)s for the
+                                     filename extension, %(format)s for the
+                                     format description (like "22 - 1280x720" or
+                                     "HD"), %(format_id)s for the unique id of
+                                     the format (like YouTube's itags: "137"),
+                                     %(upload_date)s for the upload date
+                                     (YYYYMMDD), %(extractor)s for the provider
+                                     (youtube, metacafe, etc), %(id)s for the
+                                     video id, %(playlist_title)s,
+                                     %(playlist_id)s, or %(playlist)s (=title if
+                                     present, ID otherwise) for the playlist the
+                                     video is in, %(playlist_index)s for the
+                                     position in the playlist. %(height)s and
+                                     %(width)s for the width and height of the
+                                     video format. %(resolution)s for a textual
+                                     description of the resolution of the video
+                                     format. %% for a literal percent. Use - to
+                                     output to stdout. Can also be used to
+                                     download to a different directory, for
+                                     example with -o '/my/downloads/%(uploader)s
+                                     /%(title)s-%(id)s.%(ext)s' .
     --autonumber-size NUMBER         Specify the number of digits in
                                      %(autonumber)s when it is present in output
                                      filename template or --auto-number option
@@ -236,7 +249,7 @@ which means you can modify it, redistribute it or use it however you like.
     --write-info-json                Write video metadata to a .info.json file
     --write-annotations              Write video annotations to a
                                      .annotations.xml file
-    --load-info-json FILE            JSON file containing the video information
+    --load-info FILE                 JSON file containing the video information
                                      (created with the "--write-info-json"
                                      option)
     --cookies FILE                   File to read cookies from and dump cookie
@@ -311,15 +324,7 @@ which means you can modify it, redistribute it or use it however you like.
                                      bidirectional text support. Requires bidiv
                                      or fribidi executable in PATH
     --sleep-interval SECONDS         Number of seconds to sleep before each
-                                     download when used alone or a lower bound
-                                     of a range for randomized sleep before each
-                                     download (minimum possible number of
-                                     seconds to sleep) when used along with
-                                     --max-sleep-interval.
-    --max-sleep-interval SECONDS     Upper bound of a range for randomized sleep
-                                     before each download (maximum possible
-                                     number of seconds to sleep). Must only be
-                                     used along with --min-sleep-interval.
+                                     download.
 
 ## Video Format Options:
     -f, --format FORMAT              Video format code, see the "FORMAT
@@ -413,22 +418,13 @@ which means you can modify it, redistribute it or use it however you like.
 
 # CONFIGURATION
 
-You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and OS X, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
+You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and OS X, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`.
 
 For example, with the following configuration file youtube-dl will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
 ```
-# Lines starting with # are comments
-
-# Always extract audio
 -x
-
-# Do not copy the mtime
 --no-mtime
-
-# Use this proxy
 --proxy 127.0.0.1:3128
-
-# Save all videos under Movies directory in your home directory
 -o ~/Movies/%(title)s.%(ext)s
 ```
 
@@ -509,9 +505,6 @@ The basic usage is not to set any template arguments when downloading a single f
  - `autonumber`: Five-digit number that will be increased with each download, starting at zero
  - `playlist`: Name or id of the playlist that contains the video
  - `playlist_index`: Index of the video in the playlist padded with leading zeros according to the total length of the playlist
- - `playlist_id`: Playlist identifier
- - `playlist_title`: Playlist title
-
 
 Available for the video that belongs to some logical chapter or section:
  - `chapter`: Name or title of the chapter the video belongs to
@@ -551,10 +544,6 @@ The current default template is `%(title)s-%(id)s.%(ext)s`.
 
 In some cases, you don't want special characters such as 中, spaces, or &, such as when transferring the downloaded filename to a Windows system or the filename through an 8bit-unsafe channel. In these cases, add the `--restrict-filenames` flag to get a shorter title:
 
-#### Output template and Windows batch files
-
-If you are using output template inside a Windows batch file then you must escape plain percent characters (`%`) by doubling, so that `-o "%(title)s-%(id)s.%(ext)s"` should become `-o "%%(title)s-%%(id)s.%%(ext)s"`. However you should not touch `%`'s that are not plain characters, e.g. environment variables for expansion should stay intact: `-o "C:\%HOMEPATH%\Desktop\%%(title)s.%%(ext)s"`.
-
 #### Output template examples
 
 Note on Windows you may need to use double quotes instead of single.
@@ -658,11 +647,7 @@ $ youtube-dl -f 'best[filesize<50M]'
 
 # Download best format available via direct link over HTTP/HTTPS protocol
 $ youtube-dl -f '(bestvideo+bestaudio/best)[protocol^=http]'
-
-# Download the best video format and the best audio format without merging them
-$ youtube-dl -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s'
 ```
-Note that in the last example, an output template is recommended as bestvideo and bestaudio may have the same file name.
 
 
 # VIDEO SELECTION
@@ -743,7 +728,7 @@ Videos or video formats streamed via RTMP protocol can only be downloaded when [
 
 ### I have downloaded a video but how can I play it?
 
-Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org/) or [mplayer](http://www.mplayerhq.hu/).
+Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org) or [mplayer](http://www.mplayerhq.hu/).
 
 ### I extracted a video URL with `-g`, but it does not play on another machine / in my webbrowser.
 
@@ -825,32 +810,10 @@ Either prepend `http://www.youtube.com/watch?v=` or separate the ID from the opt
 
 ### How do I pass cookies to youtube-dl?
 
-Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`.
-
-In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [Export Cookies](https://addons.mozilla.org/en-US/firefox/addon/export-cookies/) (for Firefox).
-
-Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows, `LF` (`\n`) for Linux and `CR` (`\r`) for Mac OS. `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
+Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`. Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows, `LF` (`\n`) for Linux and `CR` (`\r`) for Mac OS. `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
 
 Passing cookies to youtube-dl is a good way to workaround login when a particular extractor does not implement it explicitly. Another use case is working around [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) some websites require you to solve in particular cases in order to get access (e.g. YouTube, CloudFlare).
 
-### How do I stream directly to media player?
-
-You will first need to tell youtube-dl to stream media to stdout with `-o -`, and also tell your media player to read from stdin (it must be capable of this for streaming) and then pipe former to latter. For example, streaming to [vlc](http://www.videolan.org/) can be achieved with:
-
-    youtube-dl -o - "http://www.youtube.com/watch?v=BaW_jenozKcj" | vlc -
-
-### How do I download only new videos from a playlist?
-
-Use download-archive feature. With this feature you should initially download the complete playlist with `--download-archive /path/to/download/archive/file.txt` that will record identifiers of all the videos in a special file. Each subsequent run with the same `--download-archive` will download only new videos and skip all videos that have been downloaded before. Note that only successful downloads are recorded in the file.
-
-For example, at first,
-
-    youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
-
-will download the complete `PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re` playlist and create a file `archive.txt`. Each subsequent run will only download new videos if any:
-
-    youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
-
 ### Can you add support for this anime video site, or site which shows current movies for free?
 
 As a matter of policy (as well as legality), youtube-dl does not include support for services that specialize in infringing copyright. As a rule of thumb, if you cannot easily find a video that the service is quite obviously allowed to distribute (i.e. that has been uploaded by the creator, the creator's distributor, or is published under a free license), the service is probably unfit for inclusion to youtube-dl.
@@ -879,12 +842,6 @@ It is *not* possible to detect whether a URL is supported or not. That's because
 
 If you want to find out whether a given URL is supported, simply call youtube-dl with it. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run youtube-dl on the console) or catching an `UnsupportedError` exception if you run it from a Python program.
 
-# Why do I need to go through that much red tape when filing bugs?
-
-Before we had the issue template, despite our extensive [bug reporting instructions](#bugs), about 80% of the issue reports we got were useless, for instance because people used ancient versions hundreds of releases old, because of simple syntactic errors (not in youtube-dl but in general shell usage), because the problem was alrady reported multiple times before, because people did not actually read an error message, even if it said "please install ffmpeg", because people did not mention the URL they were trying to download and many more simple, easy-to-avoid problems, many of whom were totally unrelated to youtube-dl.
-
-youtube-dl is an open-source project manned by too few volunteers, so we'd rather spend time fixing bugs where we are certain none of those simple problems apply, and where we can be reasonably confident to be able to reproduce the issue without asking the reporter repeatedly. As such, the output of `youtube-dl -v YOUR_URL_HERE` is really all that's required to file an issue. The issue template also guides you through some basic steps you can do, such as checking that your version of youtube-dl is current.
-
 # DEVELOPER INSTRUCTIONS
 
 Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
@@ -914,17 +871,9 @@ If you want to add support for a new site, first of all **make sure** this site
 After you have ensured this site is distributing it's content legally, you can follow this quick list (assuming your service is called `yourextractor`):
 
 1. [Fork this repository](https://github.com/rg3/youtube-dl/fork)
-2. Check out the source code with:
-
-        git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git
-
-3. Start a new git branch with
-
-        cd youtube-dl
-        git checkout -b yourextractor
-
+2. Check out the source code with `git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git`
+3. Start a new git branch with `cd youtube-dl; git checkout -b yourextractor`
 4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`:
-
     ```python
     # coding: utf-8
     from __future__ import unicode_literals
@@ -967,152 +916,20 @@ After you have ensured this site is distributing it's content legally, you can f
     ```
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
+7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L68-L226). Add tests and code for as many as you want.
+8. Keep in mind that the only mandatory fields in info dict for successful extraction process are `id`, `title` and either `url` or `formats`, i.e. these are the critical data the extraction does not make any sense without. This means that [any field](https://github.com/rg3/youtube-dl/blob/58525c94d547be1c8167d16c298bdd75506db328/youtube_dl/extractor/common.py#L138-L226) apart from aforementioned mandatory ones should be treated **as optional** and extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields. For example, if you have some intermediate dict `meta` that is a source of metadata and it has a key `summary` that you want to extract and put into resulting info dict as `description`, you should be ready that this key may be missing from the `meta` dict, i.e. you should extract it as `meta.get('summary')` and not `meta['summary']`. Similarly, you should pass `fatal=False` when extracting data from a webpage with `_search_regex/_html_search_regex`.
+9. Check the code with [flake8](https://pypi.python.org/pypi/flake8).
+10. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
 
         $ git add youtube_dl/extractor/extractors.py
         $ git add youtube_dl/extractor/yourextractor.py
         $ git commit -m '[yourextractor] Add new extractor'
         $ git push origin yourextractor
 
-10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
-## youtube-dl coding conventions
-
-This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
-
-Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hoster out of your control and this layout tend to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize code dependency on source's layout changes and even to make the code foresee potential future changes and be ready for that. This is important because it will allow extractor not to break on minor layout changes thus keeping old youtube-dl versions working. Even though this breakage issue is easily fixed by emitting a new version of youtube-dl with fix incorporated all the previous version become broken in all repositories and distros' packages that may not be so prompt in fetching the update from us. Needless to say some may never receive an update at all that is possible for non rolling release distros.
-
-### Mandatory and optional metafields
-
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by [information dictionary](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L75-L257) or simply *info dict*. Only the following meta fields in *info dict* are considered mandatory for successful extraction process by youtube-dl:
-
- - `id` (media identifier)
- - `title` (media title)
- - `url` (media download URL) or `formats`
-
-In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` to be mandatory. Thus aforementioned metafields are the critical data the extraction does not make any sense without and if any of them fail to be extracted then extractor is considered completely broken.
-
-[Any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L149-L257) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerate** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
-
-#### Example
-
-Say you have some source dictionary `meta` that you've fetched as JSON with HTTP request and it has a key `summary`:
-
-```python
-meta = self._download_json(url, video_id)
-```
-    
-Assume at this point `meta`'s layout is:
-
-```python
-{
-    ...
-    "summary": "some fancy summary text",
-    ...
-}
-```
-
-Assume you want to extract `summary` and put into resulting info dict as `description`. Since `description` is optional metafield you should be ready that this key may be missing from the `meta` dict, so that you should extract it like:
-
-```python
-description = meta.get('summary')  # correct
-```
-
-and not like:
-
-```python
-description = meta['summary']  # incorrect
-```
-
-The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some time later but with former approach extraction will just go ahead with `description` set to `None` that is perfectly fine (remember `None` is equivalent for absence of data). 
-
-Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance:
-
-```python
-description = self._search_regex(
-    r'<span[^>]+id="title"[^>]*>([^<]+)<',
-    webpage, 'description', fatal=False)
-```
-
-With `fatal` set to `False` if `_search_regex` fails to extract `description` it will emit a warning and continue extraction.
-
-You can also pass `default=<some fallback value>`, for example:
-
-```python
-description = self._search_regex(
-    r'<span[^>]+id="title"[^>]*>([^<]+)<',
-    webpage, 'description', default=None)
-```
-
-On failure this code will silently continue the extraction with `description` set to `None`. That is useful for metafields that are known to may or may not be present.
- 
-### Provide fallbacks
-
-When extracting metadata try to provide several scenarios for that. For example if `title` is present in several places/sources try extracting from at least some of them. This would make it more future-proof in case some of the sources became unavailable.
-
-#### Example
-
-Say `meta` from previous example has a `title` and you are about to extract it. Since `title` is mandatory meta field you should end up with something like:
-
-```python
-title = meta['title']
-```
-
-If `title` disappeares from `meta` in future due to some changes on hoster's side the extraction would fail since `title` is mandatory. That's expected.
-
-Assume that you have some another source you can extract `title` from, for example `og:title` HTML meta of a `webpage`. In this case you can provide a fallback scenario:
-
-```python
-title = meta.get('title') or self._og_search_title(webpage)
-```
-
-This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
-
-### Make regular expressions flexible
-
-When using regular expressions try to write them fuzzy and flexible.
- 
-#### Example
-
-Say you need to extract `title` from the following HTML code:
-
-```html
-<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">some fancy title</span>
-```
-
-The code for that task should look similar to:
-
-```python
-title = self._search_regex(
-    r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title')
-```
-
-Or even better:
-
-```python
-title = self._search_regex(
-    r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)',
-    webpage, 'title', group='title')
-```
-
-Note how you tolerate potential changes in `style` attribute's value or switch from using double quotes to single for `class` attribute: 
-
-The code definitely should not look like:
-
-```python
-title = self._search_regex(
-    r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>',
-    webpage, 'title', group='title')
-```
-
-### Use safe conversion functions
-
-Wrap all extracted numeric data into safe functions from `utils`: `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
-
 # EMBEDDING YOUTUBE-DL
 
 youtube-dl makes the best effort to be a good command-line program, and thus should be callable from any programming language. If you encounter any problems parsing its output, feel free to [create a report](https://github.com/rg3/youtube-dl/issues/new).
@@ -1128,7 +945,7 @@ with youtube_dl.YoutubeDL(ydl_opts) as ydl:
     ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
 ```
 
-Most likely, you'll want to use various options. For a list of options available, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L128-L278). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
+Most likely, you'll want to use various options. For a list of what can be done, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L121-L269). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
 
 Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file:
 
@@ -1219,7 +1036,7 @@ Make sure that someone has not already opened the issue you're trying to open. S
 
 ###  Why are existing options not enough?
 
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#synopsis). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
 
 ###  Is there enough context in your bug report?
 

devscripts/buildserver.py

@@ -13,7 +13,6 @@ import os.path
 
 sys.path.insert(0, os.path.dirname(os.path.dirname((os.path.abspath(__file__)))))
 from youtube_dl.compat import (
-    compat_input,
     compat_http_server,
     compat_str,
     compat_urlparse,
@@ -31,6 +30,11 @@ try:
 except ImportError:  # Python 2
     import SocketServer as compat_socketserver
 
+try:
+    compat_input = raw_input
+except NameError:  # Python 3
+    compat_input = input
+
 
 class BuildHTTPServer(compat_socketserver.ThreadingMixIn, compat_http_server.HTTPServer):
     allow_reuse_address = True

devscripts/create-github-release.py

@@ -1,111 +0,0 @@
-#!/usr/bin/env python
-from __future__ import unicode_literals
-
-import base64
-import json
-import mimetypes
-import netrc
-import optparse
-import os
-import sys
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from youtube_dl.compat import (
-    compat_basestring,
-    compat_input,
-    compat_getpass,
-    compat_print,
-    compat_urllib_request,
-)
-from youtube_dl.utils import (
-    make_HTTPS_handler,
-    sanitized_Request,
-)
-
-
-class GitHubReleaser(object):
-    _API_URL = 'https://api.github.com/repos/rg3/youtube-dl/releases'
-    _UPLOADS_URL = 'https://uploads.github.com/repos/rg3/youtube-dl/releases/%s/assets?name=%s'
-    _NETRC_MACHINE = 'github.com'
-
-    def __init__(self, debuglevel=0):
-        self._init_github_account()
-        https_handler = make_HTTPS_handler({}, debuglevel=debuglevel)
-        self._opener = compat_urllib_request.build_opener(https_handler)
-
-    def _init_github_account(self):
-        try:
-            info = netrc.netrc().authenticators(self._NETRC_MACHINE)
-            if info is not None:
-                self._username = info[0]
-                self._password = info[2]
-                compat_print('Using GitHub credentials found in .netrc...')
-                return
-            else:
-                compat_print('No GitHub credentials found in .netrc')
-        except (IOError, netrc.NetrcParseError):
-            compat_print('Unable to parse .netrc')
-        self._username = compat_input(
-            'Type your GitHub username or email address and press [Return]: ')
-        self._password = compat_getpass(
-            'Type your GitHub password and press [Return]: ')
-
-    def _call(self, req):
-        if isinstance(req, compat_basestring):
-            req = sanitized_Request(req)
-        # Authorizing manually since GitHub does not response with 401 with
-        # WWW-Authenticate header set (see
-        # https://developer.github.com/v3/#basic-authentication)
-        b64 = base64.b64encode(
-            ('%s:%s' % (self._username, self._password)).encode('utf-8')).decode('ascii')
-        req.add_header('Authorization', 'Basic %s' % b64)
-        response = self._opener.open(req).read().decode('utf-8')
-        return json.loads(response)
-
-    def list_releases(self):
-        return self._call(self._API_URL)
-
-    def create_release(self, tag_name, name=None, body='', draft=False, prerelease=False):
-        data = {
-            'tag_name': tag_name,
-            'target_commitish': 'master',
-            'name': name,
-            'body': body,
-            'draft': draft,
-            'prerelease': prerelease,
-        }
-        req = sanitized_Request(self._API_URL, json.dumps(data).encode('utf-8'))
-        return self._call(req)
-
-    def create_asset(self, release_id, asset):
-        asset_name = os.path.basename(asset)
-        url = self._UPLOADS_URL % (release_id, asset_name)
-        # Our files are small enough to be loaded directly into memory.
-        data = open(asset, 'rb').read()
-        req = sanitized_Request(url, data)
-        mime_type, _ = mimetypes.guess_type(asset_name)
-        req.add_header('Content-Type', mime_type or 'application/octet-stream')
-        return self._call(req)
-
-
-def main():
-    parser = optparse.OptionParser(usage='%prog VERSION BUILDPATH')
-    options, args = parser.parse_args()
-    if len(args) != 2:
-        parser.error('Expected a version and a build directory')
-
-    version, build_path = args
-
-    releaser = GitHubReleaser()
-
-    new_release = releaser.create_release(version, name='youtube-dl %s' % version)
-    release_id = new_release['id']
-
-    for asset in os.listdir(build_path):
-        compat_print('Uploading %s...' % asset)
-        releaser.create_asset(release_id, os.path.join(build_path, asset))
-
-
-if __name__ == '__main__':
-    main()

devscripts/gh-pages/generate-download.py

@@ -15,9 +15,13 @@ data = urllib.request.urlopen(URL).read()
 with open('download.html.in', 'r', encoding='utf-8') as tmplf:
     template = tmplf.read()
 
+md5sum = hashlib.md5(data).hexdigest()
+sha1sum = hashlib.sha1(data).hexdigest()
 sha256sum = hashlib.sha256(data).hexdigest()
 template = template.replace('@PROGRAM_VERSION@', version)
 template = template.replace('@PROGRAM_URL@', URL)
+template = template.replace('@PROGRAM_MD5SUM@', md5sum)
+template = template.replace('@PROGRAM_SHA1SUM@', sha1sum)
 template = template.replace('@PROGRAM_SHA256SUM@', sha256sum)
 template = template.replace('@EXE_URL@', versions_info['versions'][version]['exe'][0])
 template = template.replace('@EXE_SHA256SUM@', versions_info['versions'][version]['exe'][1])

devscripts/install_srelay.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+mkdir -p tmp && cd tmp
+wget -N http://downloads.sourceforge.net/project/socks-relay/socks-relay/srelay-0.4.8/srelay-0.4.8b6.tar.gz
+tar zxvf srelay-0.4.8b6.tar.gz
+cd srelay-0.4.8b6
+./configure
+make

devscripts/make_lazy_extractors.py

@@ -14,17 +14,15 @@ if os.path.exists(lazy_extractors_filename):
     os.remove(lazy_extractors_filename)
 
 from youtube_dl.extractor import _ALL_CLASSES
-from youtube_dl.extractor.common import InfoExtractor, SearchInfoExtractor
+from youtube_dl.extractor.common import InfoExtractor
 
 with open('devscripts/lazy_load_template.py', 'rt') as f:
     module_template = f.read()
 
-module_contents = [
-    module_template + '\n' + getsource(InfoExtractor.suitable) + '\n',
-    'class LazyLoadSearchExtractor(LazyLoadExtractor):\n    pass\n']
+module_contents = [module_template + '\n' + getsource(InfoExtractor.suitable)]
 
 ie_template = '''
-class {name}({bases}):
+class {name}(LazyLoadExtractor):
     _VALID_URL = {valid_url!r}
     _module = '{module}'
 '''
@@ -36,20 +34,10 @@ make_valid_template = '''
 '''
 
 
-def get_base_name(base):
-    if base is InfoExtractor:
-        return 'LazyLoadExtractor'
-    elif base is SearchInfoExtractor:
-        return 'LazyLoadSearchExtractor'
-    else:
-        return base.__name__
-
-
 def build_lazy_ie(ie, name):
     valid_url = getattr(ie, '_VALID_URL', None)
     s = ie_template.format(
         name=name,
-        bases=', '.join(map(get_base_name, ie.__bases__)),
         valid_url=valid_url,
         module=ie.__module__)
     if ie.suitable.__func__ is not InfoExtractor.suitable.__func__:
@@ -59,35 +47,12 @@ def build_lazy_ie(ie, name):
         s += make_valid_template.format(valid_url=ie._make_valid_url())
     return s
 
-# find the correct sorting and add the required base classes so that sublcasses
-# can be correctly created
-classes = _ALL_CLASSES[:-1]
-ordered_cls = []
-while classes:
-    for c in classes[:]:
-        bases = set(c.__bases__) - set((object, InfoExtractor, SearchInfoExtractor))
-        stop = False
-        for b in bases:
-            if b not in classes and b not in ordered_cls:
-                if b.__name__ == 'GenericIE':
-                    exit()
-                classes.insert(0, b)
-                stop = True
-        if stop:
-            break
-        if all(b in ordered_cls for b in bases):
-            ordered_cls.append(c)
-            classes.remove(c)
-            break
-ordered_cls.append(_ALL_CLASSES[-1])
-
 names = []
-for ie in ordered_cls:
-    name = ie.__name__
+for ie in list(sorted(_ALL_CLASSES[:-1], key=lambda cls: cls.ie_key())) + _ALL_CLASSES[-1:]:
+    name = ie.ie_key() + 'IE'
     src = build_lazy_ie(ie, name)
     module_contents.append(src)
-    if ie in _ALL_CLASSES:
-        names.append(name)
+    names.append(name)
 
 module_contents.append(
     '_ALL_CLASSES = [{0}]'.format(', '.join(names)))

devscripts/prepare_manpage.py

@@ -54,21 +54,17 @@ def filter_options(readme):
 
         if in_options:
             if line.lstrip().startswith('-'):
-                split = re.split(r'\s{2,}', line.lstrip())
-                # Description string may start with `-` as well. If there is
-                # only one piece then it's a description bit not an option.
-                if len(split) > 1:
-                    option, description = split
-                    split_option = option.split(' ')
+                option, description = re.split(r'\s{2,}', line.lstrip())
+                split_option = option.split(' ')
 
-                    if not split_option[-1].startswith('-'):  # metavar
-                        option = ' '.join(split_option[:-1] + ['*%s*' % split_option[-1]])
+                if not split_option[-1].startswith('-'):  # metavar
+                    option = ' '.join(split_option[:-1] + ['*%s*' % split_option[-1]])
 
-                    # Pandoc's definition_lists. See http://pandoc.org/README.html
-                    # for more information.
-                    ret += '\n%s\n:   %s\n' % (option, description)
-                    continue
-            ret += line.lstrip() + '\n'
+                # Pandoc's definition_lists. See http://pandoc.org/README.html
+                # for more information.
+                ret += '\n%s\n:   %s\n' % (option, description)
+            else:
+                ret += line.lstrip() + '\n'
         else:
             ret += line + '\n'
 

devscripts/release.sh

@@ -15,7 +15,6 @@
 set -e
 
 skip_tests=true
-gpg_sign_commits=""
 buildserver='localhost:8142'
 
 while true
@@ -25,10 +24,6 @@ case "$1" in
         skip_tests=false
         shift
     ;;
-    --gpg-sign-commits|-S)
-        gpg_sign_commits="-S"
-        shift
-    ;;
     --buildserver)
         buildserver="$2"
         shift 2
@@ -71,13 +66,10 @@ fi
 /bin/echo -e "\n### Changing version in version.py..."
 sed -i "s/__version__ = '.*'/__version__ = '$version'/" youtube_dl/version.py
 
-/bin/echo -e "\n### Changing version in ChangeLog..."
-sed -i "s/<unreleased>/$version/" ChangeLog
-
 /bin/echo -e "\n### Committing documentation, templates and youtube_dl/version.py..."
 make README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md supportedsites
-git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md docs/supportedsites.md youtube_dl/version.py ChangeLog
-git commit $gpg_sign_commits -m "release $version"
+git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md docs/supportedsites.md youtube_dl/version.py
+git commit -m "release $version"
 
 /bin/echo -e "\n### Now tagging, signing and pushing..."
 git tag -s -m "Release $version" "$version"
@@ -103,16 +95,15 @@ RELEASE_FILES="youtube-dl youtube-dl.exe youtube-dl-$version.tar.gz"
 (cd build/$version/ && sha256sum $RELEASE_FILES > SHA2-256SUMS)
 (cd build/$version/ && sha512sum $RELEASE_FILES > SHA2-512SUMS)
 
-/bin/echo -e "\n### Signing and uploading the new binaries to GitHub..."
+/bin/echo -e "\n### Signing and uploading the new binaries to yt-dl.org ..."
 for f in $RELEASE_FILES; do gpg --passphrase-repeat 5 --detach-sig "build/$version/$f"; done
-
-ROOT=$(pwd)
-python devscripts/create-github-release.py $version "$ROOT/build/$version"
-
+scp -r "build/$version" ytdl@yt-dl.org:html/tmp/
+ssh ytdl@yt-dl.org "mv html/tmp/$version html/downloads/"
 ssh ytdl@yt-dl.org "sh html/update_latest.sh $version"
 
 /bin/echo -e "\n### Now switching to gh-pages..."
 git clone --branch gh-pages --single-branch . build/gh-pages
+ROOT=$(pwd)
 (
     set -e
     ORIGIN_URL=$(git config --get remote.origin.url)
@@ -124,7 +115,7 @@ git clone --branch gh-pages --single-branch . build/gh-pages
     "$ROOT/devscripts/gh-pages/update-copyright.py"
     "$ROOT/devscripts/gh-pages/update-sites.py"
     git add *.html *.html.in update
-    git commit $gpg_sign_commits -m "release $version"
+    git commit -m "release $version"
     git push "$ROOT" gh-pages
     git push "$ORIGIN_URL" gh-pages
 )

devscripts/show-downloads-statistics.py

@@ -1,47 +0,0 @@
-#!/usr/bin/env python
-from __future__ import unicode_literals
-
-import itertools
-import json
-import os
-import re
-import sys
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from youtube_dl.compat import (
-    compat_print,
-    compat_urllib_request,
-)
-from youtube_dl.utils import format_bytes
-
-
-def format_size(bytes):
-    return '%s (%d bytes)' % (format_bytes(bytes), bytes)
-
-
-total_bytes = 0
-
-for page in itertools.count(1):
-    releases = json.loads(compat_urllib_request.urlopen(
-        'https://api.github.com/repos/rg3/youtube-dl/releases?page=%s' % page
-    ).read().decode('utf-8'))
-
-    if not releases:
-        break
-
-    for release in releases:
-        compat_print(release['name'])
-        for asset in release['assets']:
-            asset_name = asset['name']
-            total_bytes += asset['download_count'] * asset['size']
-            if all(not re.match(p, asset_name) for p in (
-                    r'^youtube-dl$',
-                    r'^youtube-dl-\d{4}\.\d{2}\.\d{2}(?:\.\d+)?\.tar\.gz$',
-                    r'^youtube-dl\.exe$')):
-                continue
-            compat_print(
-                ' %s size: %s downloads: %d'
-                % (asset_name, format_size(asset['size']), asset['download_count']))
-
-compat_print('total downloads traffic: %s' % format_size(total_bytes))

docs/supportedsites.md

@@ -13,12 +13,8 @@
  - **5min**
  - **8tracks**
  - **91porn**
- - **9c9media**
- - **9c9media:stack**
  - **9gag**
- - **9now.com.au**
  - **abc.net.au**
- - **abc.net.au:iview**
  - **Abc7News**
  - **abcnews**
  - **abcnews:video**
@@ -32,13 +28,11 @@
  - **AdobeTVVideo**
  - **AdultSwim**
  - **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network
- - **AfreecaTV**: afreecatv.com
  - **Aftonbladet**
  - **AirMozilla**
  - **AlJazeera**
  - **Allocine**
  - **AlphaPorno**
- - **AMCNetworks**
  - **AnimeOnDemand**
  - **anitube.se**
  - **AnySex**
@@ -50,7 +44,7 @@
  - **archive.org**: archive.org videos
  - **ARD**
  - **ARD:mediathek**
- - **Arkena**
+ - **ARD:mediathek**: Saarländischer Rundfunk
  - **arte.tv**
  - **arte.tv:+7**
  - **arte.tv:cinema**
@@ -69,10 +63,6 @@
  - **audiomack**
  - **audiomack:album**
  - **auroravid**: AuroraVid
- - **AWAAN**
- - **awaan:live**
- - **awaan:season**
- - **awaan:video**
  - **Azubu**
  - **AzubuLive**
  - **BaiduVideo**: 百度视频
@@ -83,8 +73,6 @@
  - **bbc**: BBC
  - **bbc.co.uk**: BBC iPlayer
  - **bbc.co.uk:article**: BBC articles
- - **bbc.co.uk:iplayer:playlist**
- - **bbc.co.uk:playlist**
  - **BeatportPro**
  - **Beeg**
  - **BehindKink**
@@ -115,13 +103,8 @@
  - **canalc2.tv**
  - **Canalplus**: canalplus.fr, piwiplus.fr and d8.tv
  - **Canvas**
- - **CarambaTV**
- - **CarambaTVPage**
- - **CartoonNetwork**
- - **cbc.ca**
- - **cbc.ca:player**
- - **cbc.ca:watch**
- - **cbc.ca:watch:video**
+ - **CBC**
+ - **CBCPlayer**
  - **CBS**
  - **CBSInteractive**
  - **CBSLocal**
@@ -131,7 +114,6 @@
  - **CDA**
  - **CeskaTelevize**
  - **channel9**: Channel 9
- - **CharlieRose**
  - **Chaturbate**
  - **Chilloutzone**
  - **chirbit**
@@ -141,7 +123,6 @@
  - **cliphunter**
  - **ClipRs**
  - **Clipsyndicate**
- - **CloserToTruth**
  - **cloudtime**: CloudTime
  - **Cloudy**
  - **Clubic**
@@ -154,8 +135,7 @@
  - **CollegeRama**
  - **ComCarCoff**
  - **ComedyCentral**
- - **ComedyCentralShortname**
- - **ComedyCentralTV**
+ - **ComedyCentralShows**: The Daily Show / The Colbert Report
  - **CondeNast**: Condé Nast media group: Allure, Architectural Digest, Ars Technica, Bon Appétit, Brides, Condé Nast, Condé Nast Traveler, Details, Epicurious, GQ, Glamour, Golf Digest, SELF, Teen Vogue, The New Yorker, Vanity Fair, Vogue, W Magazine, WIRED
  - **Coub**
  - **Cracked**
@@ -167,12 +147,8 @@
  - **CSNNE**
  - **CSpan**: C-SPAN
  - **CtsNews**: 華視新聞
- - **CTV**
- - **CTVNews**
  - **culturebox.francetvinfo.fr**
  - **CultureUnplugged**
- - **curiositystream**
- - **curiositystream:collection**
  - **CWTV**
  - **DailyMail**
  - **dailymotion**
@@ -184,6 +160,10 @@
  - **daum.net:playlist**
  - **daum.net:user**
  - **DBTV**
+ - **DCN**
+ - **dcn:live**
+ - **dcn:season**
+ - **dcn:video**
  - **DctpTv**
  - **DeezerPlaylist**
  - **defense.gouv.fr**
@@ -192,7 +172,6 @@
  - **DigitallySpeaking**
  - **Digiteka**
  - **Discovery**
- - **DiscoveryGo**
  - **Dotsub**
  - **DouyuTV**: 斗鱼
  - **DPlay**
@@ -225,31 +204,29 @@
  - **EsriVideo**
  - **Europa**
  - **EveryonesMixtape**
+ - **exfm**: ex.fm
  - **ExpoTV**
  - **ExtremeTube**
  - **EyedoTV**
  - **facebook**
- - **FacebookPluginsVideo**
  - **faz.net**
  - **fc2**
- - **fc2:embed**
  - **Fczenit**
  - **features.aol.com**
  - **fernsehkritik.tv**
  - **Firstpost**
  - **FiveTV**
  - **Flickr**
- - **Flipagram**
  - **Folketinget**: Folketinget (ft.dk; Danish parliament)
  - **FootyRoom**
  - **Formula1**
  - **FOX**
  - **Foxgay**
  - **FoxNews**: Fox News and Fox Business Video
- - **foxnews:insider**
  - **FoxSports**
  - **france2.fr:generation-quoi**
  - **FranceCulture**
+ - **FranceCultureEmission**
  - **FranceInter**
  - **francetv**: France 2, 3, 4, 5 and Ô
  - **francetvinfo.fr**
@@ -258,14 +235,14 @@
  - **FreeVideo**
  - **Funimation**
  - **FunnyOrDie**
- - **Fusion**
- - **FXNetworks**
  - **GameInformer**
+ - **Gamekings**
  - **GameOne**
  - **gameone:playlist**
  - **Gamersyde**
  - **GameSpot**
  - **GameStar**
+ - **Gametrailers**
  - **Gazeta**
  - **GDCVault**
  - **generic**: Generic downloader that works on some sites
@@ -275,9 +252,8 @@
  - **Glide**: Glide mobile video messages (glide.me)
  - **Globo**
  - **GloboArticle**
- - **Go**
  - **GodTube**
- - **GodTV**
+ - **GoldenMoustache**
  - **Golem**
  - **GoogleDrive**
  - **Goshgay**
@@ -290,10 +266,7 @@
  - **HellPorno**
  - **Helsinki**: helsinki.fi
  - **HentaiStigma**
- - **HGTV**
- - **hgtv.com:show**
  - **HistoricFilms**
- - **history:topic**: History.com Topic
  - **hitbox**
  - **hitbox:live**
  - **HornBunny**
@@ -301,8 +274,6 @@
  - **HotStar**
  - **Howcast**
  - **HowStuffWorks**
- - **HRTi**
- - **HRTiPlaylist**
  - **HuffPost**: Huffington Post
  - **Hypem**
  - **Iconosquare**
@@ -330,7 +301,6 @@
  - **jpopsuki.tv**
  - **JWPlatform**
  - **Kaltura**
- - **Kamcord**
  - **KanalPlay**: Kanal 5/9/11 Play
  - **Kankan**
  - **Karaoketv**
@@ -350,10 +320,8 @@
  - **kuwo:mv**: 酷我音乐 - MV
  - **kuwo:singer**: 酷我音乐 - 歌手
  - **kuwo:song**: 酷我音乐
- - **la7.it**
+ - **la7.tv**
  - **Laola1Tv**
- - **Lcp**
- - **LcpPlay**
  - **Le**: 乐视网
  - **Learnr**
  - **Lecture2Go**
@@ -385,7 +353,6 @@
  - **MatchTV**
  - **MDR**: MDR.DE and KiKA
  - **media.ccc.de**
- - **META**
  - **metacafe**
  - **Metacritic**
  - **Mgoon**
@@ -411,12 +378,11 @@
  - **MovieClips**
  - **MovieFap**
  - **Moviezine**
- - **MovingImage**
  - **MPORA**
- - **MSN**
- - **mtg**: MTG services
+ - **MSNBC**
  - **MTV**
  - **mtv.de**
+ - **mtviggy.com**
  - **mtvservices:embedded**
  - **MuenchenTV**: münchen.tv
  - **MusicPlayOn**
@@ -432,8 +398,7 @@
  - **MyVidster**
  - **n-tv.de**
  - **natgeo**
- - **natgeo:episodeguide**
- - **natgeo:video**
+ - **natgeo:channel**
  - **Naver**
  - **NBA**
  - **NBC**
@@ -457,18 +422,16 @@
  - **Newstube**
  - **NextMedia**: 蘋果日報
  - **NextMediaActionNews**: 蘋果日報 - 動新聞
+ - **nextmovie.com**
  - **nfb**: National Film Board of Canada
  - **nfl.com**
- - **NhkVod**
  - **nhl.com**
  - **nhl.com:news**: NHL news
  - **nhl.com:videocenter**
  - **nhl.com:videocenter:category**: NHL videocenter category
  - **nick.com**
- - **nick.de**
  - **niconico**: ニコニコ動画
  - **NiconicoPlaylist**
- - **Nintendo**
  - **njoy**: N-JOY
  - **njoy:embed**
  - **Noco**
@@ -496,12 +459,9 @@
  - **NYTimes**
  - **NYTimesArticle**
  - **ocw.mit.edu**
- - **OdaTV**
  - **Odnoklassniki**
  - **OktoberfestTV**
  - **on.aol.com**
- - **onet.tv**
- - **onet.tv:channel**
  - **OnionStudios**
  - **Ooyala**
  - **OoyalaExternal**
@@ -525,6 +485,7 @@
  - **Pinkbike**
  - **Pladform**
  - **play.fm**
+ - **played.to**
  - **PlaysTV**
  - **Playtvak**: Playtvak.cz, iDNES.cz and Lidovky.cz
  - **Playvid**
@@ -534,11 +495,8 @@
  - **plus.google**: Google Plus
  - **pluzz.francetv.fr**
  - **podomatic**
- - **Pokemon**
- - **PolskieRadio**
- - **PornCom**
  - **PornHd**
- - **PornHub**: PornHub and Thumbzilla
+ - **PornHub**
  - **PornHubPlaylist**
  - **PornHubUserVideos**
  - **Pornotube**
@@ -556,7 +514,6 @@
  - **qqmusic:singer**: QQ音乐 - 歌手
  - **qqmusic:toplist**: QQ音乐 - 排行榜
  - **R7**
- - **R7Article**
  - **radio.de**
  - **radiobremen**
  - **radiocanada**
@@ -576,11 +533,8 @@
  - **revision3:embed**
  - **RICE**
  - **RingTV**
- - **RockstarGames**
- - **RoosterTeeth**
  - **RottenTomatoes**
  - **Roxwel**
- - **Rozhlas**
  - **RTBF**
  - **rte**: Raidió Teilifís Éireann TV
  - **rte:radio**: Raidió Teilifís Éireann radio
@@ -591,9 +545,7 @@
  - **rtve.es:alacarta**: RTVE a la carta
  - **rtve.es:infantil**: RTVE infantil
  - **rtve.es:live**: RTVE.es live streams
- - **rtve.es:television**
  - **RTVNH**
- - **Rudo**
  - **RUHD**
  - **RulePorn**
  - **rutube**: Rutube videos
@@ -626,10 +578,8 @@
  - **Shared**: shared.sx and vivo.sx
  - **ShareSix**
  - **Sina**
- - **SixPlay**
- - **skynewsarabia:article**
  - **skynewsarabia:video**
- - **SkySports**
+ - **skynewsarabia:video**
  - **Slideshare**
  - **Slutload**
  - **smotri**: Smotri.com
@@ -638,7 +588,6 @@
  - **smotri:user**: Smotri.com user videos
  - **Snotr**
  - **Sohu**
- - **SonyLIV**
  - **soundcloud**
  - **soundcloud:playlist**
  - **soundcloud:search**: Soundcloud search
@@ -662,13 +611,12 @@
  - **SportBoxEmbed**
  - **SportDeutschland**
  - **Sportschau**
- - **sr:mediathek**: Saarländischer Rundfunk
  - **SRGSSR**
  - **SRGSSRPlay**: srf.ch, rts.ch, rsi.ch, rtr.ch and swissinfo.ch play sites
+ - **SSA**
  - **stanfordoc**: Stanford Open ClassRoom
  - **Steam**
  - **Stitcher**
- - **Streamable**
  - **streamcloud.eu**
  - **StreamCZ**
  - **StreetVoice**
@@ -680,8 +628,8 @@
  - **SztvHu**
  - **Tagesschau**
  - **tagesschau:player**
+ - **Tapely**
  - **Tass**
- - **TBS**
  - **TDSLifeway**
  - **teachertube**: teachertube.com videos
  - **teachertube:user:collection**: teachertube.com user and collection videos
@@ -697,7 +645,6 @@
  - **Telegraaf**
  - **TeleMB**
  - **TeleTask**
- - **Telewebion**
  - **TF1**
  - **TheIntercept**
  - **ThePlatform**
@@ -707,6 +654,8 @@
  - **TheStar**
  - **ThisAmericanLife**
  - **ThisAV**
+ - **THVideo**
+ - **THVideoPlaylist**
  - **tinypic**: tinypic.com videos
  - **tlc.de**
  - **TMZ**
@@ -714,13 +663,13 @@
  - **TNAFlix**
  - **TNAFlixNetworkEmbed**
  - **toggle**
- - **Tosh**: Tosh.0
  - **tou.tv**
  - **Toypics**: Toypics user profile
  - **ToypicsUser**: Toypics user profile
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
  - **trollvids**
+ - **TruTube**
  - **Tube8**
  - **TubiTv**
  - **tudou**
@@ -743,11 +692,10 @@
  - **tvigle**: Интернет-телевидение Tvigle.ru
  - **tvland.com**
  - **tvp**: Telewizja Polska
- - **tvp:embed**: Telewizja Polska
  - **tvp:series**
+ - **TVPlay**: TV3Play and related services
  - **Tweakers**
  - **twitch:chapter**
- - **twitch:clips**
  - **twitch:past_broadcasts**
  - **twitch:profile**
  - **twitch:stream**
@@ -760,12 +708,7 @@
  - **udemy:course**
  - **UDNEmbed**: 聯合影音
  - **Unistra**
- - **uol.com.br**
- - **uplynk**
- - **uplynk:preplay**
  - **Urort**: NRK P3 Urørt
- - **URPlay**
- - **USANetwork**
  - **USAToday**
  - **ustream**
  - **ustream:channel**
@@ -781,11 +724,8 @@
  - **VevoPlaylist**
  - **VGTV**: VGTV, BTTV, FTV, Aftenposten and Aftonbladet
  - **vh1.com**
- - **Viafree**
  - **Vice**
- - **Viceland**
  - **ViceShow**
- - **Vidbit**
  - **Viddler**
  - **video.google:search**: Google Video search
  - **video.mit.edu**
@@ -798,7 +738,6 @@
  - **VideoPremium**
  - **VideoTt**: video.tt - Your True Tube (Currently broken)
  - **videoweed**: VideoWeed
- - **Vidio**
  - **vidme**
  - **vidme:user**
  - **vidme:user:likes**
@@ -825,10 +764,8 @@
  - **vine:user**
  - **vk**: VK
  - **vk:uservideos**: VK - User's Videos
- - **vk:wallpost**
  - **vlive**
  - **Vodlocker**
- - **VODPlatform**
  - **VoiceRepublic**
  - **VoxMedia**
  - **Vporn**
@@ -836,6 +773,7 @@
  - **VRT**
  - **vube**: Vube.com
  - **VuClip**
+ - **vulture.com**
  - **Walla**
  - **washingtonpost**
  - **washingtonpost:article**
@@ -843,8 +781,10 @@
  - **WatchIndianPorn**: Watch Indian Porn
  - **WDR**
  - **wdr:mobile**
+ - **WDRMaus**: Sendung mit der Maus
  - **WebOfStories**
  - **WebOfStoriesPlaylist**
+ - **Weibo**
  - **WeiqiTV**: WQTV
  - **wholecloud**: WholeCloud
  - **Wimp**
@@ -852,11 +792,10 @@
  - **WNL**
  - **WorldStarHipHop**
  - **wrzuta.pl**
- - **wrzuta.pl:playlist**
  - **WSJ**: Wall Street Journal
  - **XBef**
  - **XboxClips**
- - **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To, XVIDSTAGE
+ - **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To
  - **XHamster**
  - **XHamsterEmbed**
  - **xiami:album**: 虾米音乐 - 专辑
@@ -881,7 +820,6 @@
  - **Ynet**
  - **YouJizz**
  - **youku**: 优酷
- - **youku:show**
  - **YouPorn**
  - **YourUpload**
  - **youtube**: YouTube.com
@@ -895,7 +833,6 @@
  - **youtube:search**: YouTube.com searches
  - **youtube:search:date**: YouTube.com searches, newest videos first
  - **youtube:search_url**: YouTube.com search URLs
- - **youtube:shared**
  - **youtube:show**: YouTube.com (multi-season) shows
  - **youtube:subscriptions**: YouTube.com subscriptions feed, "ytsubs" keyword (requires authentication)
  - **youtube:user**: YouTube.com user videos (URL or "ytuser" keyword)
@@ -903,4 +840,6 @@
  - **Zapiks**
  - **ZDF**
  - **ZDFChannel**
- - **zingmp3**: mp3.zing.vn
+ - **zingmp3:album**: mp3.zing.vn albums
+ - **zingmp3:song**: mp3.zing.vn songs
+ - **ZippCast**

setup.py

@@ -21,37 +21,25 @@ try:
     import py2exe
 except ImportError:
     if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
-        print('Cannot import py2exe', file=sys.stderr)
+        print("Cannot import py2exe", file=sys.stderr)
         exit(1)
 
 py2exe_options = {
-    'bundle_files': 1,
-    'compressed': 1,
-    'optimize': 2,
-    'dist_dir': '.',
-    'dll_excludes': ['w9xpopen.exe', 'crypt32.dll'],
+    "bundle_files": 1,
+    "compressed": 1,
+    "optimize": 2,
+    "dist_dir": '.',
+    "dll_excludes": ['w9xpopen.exe', 'crypt32.dll'],
 }
 
-# Get the version from youtube_dl/version.py without importing the package
-exec(compile(open('youtube_dl/version.py').read(),
-             'youtube_dl/version.py', 'exec'))
-
-DESCRIPTION = 'YouTube video downloader'
-LONG_DESCRIPTION = 'Command-line program to download videos from YouTube.com and other video sites'
-
 py2exe_console = [{
-    'script': './youtube_dl/__main__.py',
-    'dest_base': 'youtube-dl',
-    'version': __version__,
-    'description': DESCRIPTION,
-    'comments': LONG_DESCRIPTION,
-    'product_name': 'youtube-dl',
-    'product_version': __version__,
+    "script": "./youtube_dl/__main__.py",
+    "dest_base": "youtube-dl",
 }]
 
 py2exe_params = {
     'console': py2exe_console,
-    'options': {'py2exe': py2exe_options},
+    'options': {"py2exe": py2exe_options},
     'zipfile': None
 }
 
@@ -84,7 +72,7 @@ else:
         params['scripts'] = ['bin/youtube-dl']
 
 class build_lazy_extractors(Command):
-    description = 'Build the extractor lazy loading module'
+    description = "Build the extractor lazy loading module"
     user_options = []
 
     def initialize_options(self):
@@ -99,11 +87,16 @@ class build_lazy_extractors(Command):
             dry_run=self.dry_run,
         )
 
+# Get the version from youtube_dl/version.py without importing the package
+exec(compile(open('youtube_dl/version.py').read(),
+             'youtube_dl/version.py', 'exec'))
+
 setup(
     name='youtube_dl',
     version=__version__,
-    description=DESCRIPTION,
-    long_description=LONG_DESCRIPTION,
+    description='YouTube video downloader',
+    long_description='Small command-line program to download videos from'
+    ' YouTube.com and other video sites.',
     url='https://github.com/rg3/youtube-dl',
     author='Ricardo Garcia',
     author_email='ytdl@yt-dl.org',
@@ -119,17 +112,16 @@ setup(
     # test_requires = ['nosetest'],
 
     classifiers=[
-        'Topic :: Multimedia :: Video',
-        'Development Status :: 5 - Production/Stable',
-        'Environment :: Console',
-        'License :: Public Domain',
-        'Programming Language :: Python :: 2.6',
-        'Programming Language :: Python :: 2.7',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.2',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
+        "Topic :: Multimedia :: Video",
+        "Development Status :: 5 - Production/Stable",
+        "Environment :: Console",
+        "License :: Public Domain",
+        "Programming Language :: Python :: 2.6",
+        "Programming Language :: Python :: 2.7",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.2",
+        "Programming Language :: Python :: 3.3",
+        "Programming Language :: Python :: 3.4",
     ],
 
     cmdclass={'build_lazy_extractors': build_lazy_extractors},

test/test_InfoExtractor.py

@@ -11,7 +11,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 from test.helper import FakeYDL
 from youtube_dl.extractor.common import InfoExtractor
 from youtube_dl.extractor import YoutubeIE, get_info_extractor
-from youtube_dl.utils import encode_data_uri, strip_jsonp, ExtractorError, RegexNotFoundError
+from youtube_dl.utils import encode_data_uri, strip_jsonp, ExtractorError
 
 
 class TestIE(InfoExtractor):
@@ -48,9 +48,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertEqual(ie._og_search_property('foobar', html), 'Foo')
         self.assertEqual(ie._og_search_property('test1', html), 'foo > < bar')
         self.assertEqual(ie._og_search_property('test2', html), 'foo >//< bar')
-        self.assertEqual(ie._og_search_property(('test0', 'test1'), html), 'foo > < bar')
-        self.assertRaises(RegexNotFoundError, ie._og_search_property, 'test0', html, None, fatal=True)
-        self.assertRaises(RegexNotFoundError, ie._og_search_property, ('test0', 'test00'), html, None, fatal=True)
 
     def test_html_search_meta(self):
         ie = self.ie
@@ -69,11 +66,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertEqual(ie._html_search_meta('d', html), '4')
         self.assertEqual(ie._html_search_meta('e', html), '5')
         self.assertEqual(ie._html_search_meta('f', html), '6')
-        self.assertEqual(ie._html_search_meta(('a', 'b', 'c'), html), '1')
-        self.assertEqual(ie._html_search_meta(('c', 'b', 'a'), html), '3')
-        self.assertEqual(ie._html_search_meta(('z', 'x', 'c'), html), '3')
-        self.assertRaises(RegexNotFoundError, ie._html_search_meta, 'z', html, None, fatal=True)
-        self.assertRaises(RegexNotFoundError, ie._html_search_meta, ('z', 'x'), html, None, fatal=True)
 
     def test_download_json(self):
         uri = encode_data_uri(b'{"foo": "blah"}', 'application/json')

test/test_YoutubeDL.py

@@ -335,40 +335,6 @@ class TestFormatSelection(unittest.TestCase):
             downloaded = ydl.downloaded_info_dicts[0]
             self.assertEqual(downloaded['format_id'], f1['format_id'])
 
-    def test_audio_only_extractor_format_selection(self):
-        # For extractors with incomplete formats (all formats are audio-only or
-        # video-only) best and worst should fallback to corresponding best/worst
-        # video-only or audio-only formats (as per
-        # https://github.com/rg3/youtube-dl/pull/5556)
-        formats = [
-            {'format_id': 'low', 'ext': 'mp3', 'preference': 1, 'vcodec': 'none', 'url': TEST_URL},
-            {'format_id': 'high', 'ext': 'mp3', 'preference': 2, 'vcodec': 'none', 'url': TEST_URL},
-        ]
-        info_dict = _make_result(formats)
-
-        ydl = YDL({'format': 'best'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'high')
-
-        ydl = YDL({'format': 'worst'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'low')
-
-    def test_format_not_available(self):
-        formats = [
-            {'format_id': 'regular', 'ext': 'mp4', 'height': 360, 'url': TEST_URL},
-            {'format_id': 'video', 'ext': 'mp4', 'height': 720, 'acodec': 'none', 'url': TEST_URL},
-        ]
-        info_dict = _make_result(formats)
-
-        # This must fail since complete video-audio format does not match filter
-        # and extractor does not provide incomplete only formats (i.e. only
-        # video-only or audio-only).
-        ydl = YDL({'format': 'best[height>360]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
     def test_invalid_format_specs(self):
         def assert_syntax_error(format_spec):
             ydl = YDL({'format': format_spec})

test/test_all_urls.py

@@ -6,7 +6,6 @@ from __future__ import unicode_literals
 import os
 import sys
 import unittest
-import collections
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 
@@ -101,6 +100,8 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch(':ytsubs', ['youtube:subscriptions'])
         self.assertMatch(':ytsubscriptions', ['youtube:subscriptions'])
         self.assertMatch(':ythistory', ['youtube:history'])
+        self.assertMatch(':thedailyshow', ['ComedyCentralShows'])
+        self.assertMatch(':tds', ['ComedyCentralShows'])
 
     def test_vimeo_matching(self):
         self.assertMatch('https://vimeo.com/channels/tributes', ['vimeo:channel'])
@@ -129,15 +130,6 @@ class TestAllURLsMatching(unittest.TestCase):
             'https://screen.yahoo.com/smartwatches-latest-wearable-gadgets-163745379-cbs.html',
             ['Yahoo'])
 
-    def test_no_duplicated_ie_names(self):
-        name_accu = collections.defaultdict(list)
-        for ie in self.ies:
-            name_accu[ie.IE_NAME.lower()].append(type(ie).__name__)
-        for (ie_name, ie_list) in name_accu.items():
-            self.assertEqual(
-                len(ie_list), 1,
-                'Multiple extractors with the same IE_NAME "%s" (%s)' % (ie_name, ', '.join(ie_list)))
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_compat.py

@@ -87,8 +87,6 @@ class TestCompat(unittest.TestCase):
 
     def test_compat_shlex_split(self):
         self.assertEqual(compat_shlex_split('-option "one two"'), ['-option', 'one two'])
-        self.assertEqual(compat_shlex_split('-option "one\ntwo" \n -flag'), ['-option', 'one\ntwo', '-flag'])
-        self.assertEqual(compat_shlex_split('-val 中文'), ['-val', '中文'])
 
     def test_compat_etree_fromstring(self):
         xml = '''

test/test_http.py

@@ -138,27 +138,27 @@ class TestProxy(unittest.TestCase):
         self.proxy_thread.daemon = True
         self.proxy_thread.start()
 
-        self.geo_proxy = compat_http_server.HTTPServer(
-            ('localhost', 0), _build_proxy_handler('geo'))
-        self.geo_port = http_server_port(self.geo_proxy)
-        self.geo_proxy_thread = threading.Thread(target=self.geo_proxy.serve_forever)
-        self.geo_proxy_thread.daemon = True
-        self.geo_proxy_thread.start()
+        self.cn_proxy = compat_http_server.HTTPServer(
+            ('localhost', 0), _build_proxy_handler('cn'))
+        self.cn_port = http_server_port(self.cn_proxy)
+        self.cn_proxy_thread = threading.Thread(target=self.cn_proxy.serve_forever)
+        self.cn_proxy_thread.daemon = True
+        self.cn_proxy_thread.start()
 
     def test_proxy(self):
-        geo_proxy = 'localhost:{0}'.format(self.geo_port)
+        cn_proxy = 'localhost:{0}'.format(self.cn_port)
         ydl = YoutubeDL({
             'proxy': 'localhost:{0}'.format(self.port),
-            'geo_verification_proxy': geo_proxy,
+            'cn_verification_proxy': cn_proxy,
         })
         url = 'http://foo.com/bar'
         response = ydl.urlopen(url).read().decode('utf-8')
         self.assertEqual(response, 'normal: {0}'.format(url))
 
         req = compat_urllib_request.Request(url)
-        req.add_header('Ytdl-request-proxy', geo_proxy)
+        req.add_header('Ytdl-request-proxy', cn_proxy)
         response = ydl.urlopen(req).read().decode('utf-8')
-        self.assertEqual(response, 'geo: {0}'.format(url))
+        self.assertEqual(response, 'cn: {0}'.format(url))
 
     def test_proxy_with_idn(self):
         ydl = YoutubeDL({

test/test_utils.py

@@ -33,17 +33,14 @@ from youtube_dl.utils import (
     ExtractorError,
     find_xpath_attr,
     fix_xml_ampersands,
-    get_element_by_class,
     InAdvancePagedList,
     intlist_to_bytes,
     is_html,
     js_to_json,
     limit_length,
-    mimetype2ext,
     ohdave_rsa_encrypt,
     OnDemandPagedList,
     orderedSet,
-    parse_age_limit,
     parse_duration,
     parse_filesize,
     parse_count,
@@ -63,13 +60,11 @@ from youtube_dl.utils import (
     timeconvert,
     unescapeHTML,
     unified_strdate,
-    unified_timestamp,
     unsmuggle_url,
     uppercase_escape,
     lowercase_escape,
     url_basename,
     urlencode_postdata,
-    urshift,
     update_url_query,
     version_tuple,
     xpath_with_ns,
@@ -83,7 +78,6 @@ from youtube_dl.utils import (
     cli_option,
     cli_valueless_option,
     cli_bool_option,
-    parse_codecs,
 )
 from youtube_dl.compat import (
     compat_chr,
@@ -255,8 +249,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(unescapeHTML('/'), '/')
         self.assertEqual(unescapeHTML('é'), 'é')
         self.assertEqual(unescapeHTML('�'), '�')
-        # HTML5 entities
-        self.assertEqual(unescapeHTML('.''), '.\'')
 
     def test_date_from_str(self):
         self.assertEqual(date_from_str('yesterday'), date_from_str('now-1day'))
@@ -289,29 +281,8 @@ class TestUtil(unittest.TestCase):
             '20150202')
         self.assertEqual(unified_strdate('Feb 14th 2016 5:45PM'), '20160214')
         self.assertEqual(unified_strdate('25-09-2014'), '20140925')
-        self.assertEqual(unified_strdate('27.02.2016 17:30'), '20160227')
         self.assertEqual(unified_strdate('UNKNOWN DATE FORMAT'), None)
 
-    def test_unified_timestamps(self):
-        self.assertEqual(unified_timestamp('December 21, 2010'), 1292889600)
-        self.assertEqual(unified_timestamp('8/7/2009'), 1247011200)
-        self.assertEqual(unified_timestamp('Dec 14, 2012'), 1355443200)
-        self.assertEqual(unified_timestamp('2012/10/11 01:56:38 +0000'), 1349920598)
-        self.assertEqual(unified_timestamp('1968 12 10'), -33436800)
-        self.assertEqual(unified_timestamp('1968-12-10'), -33436800)
-        self.assertEqual(unified_timestamp('28/01/2014 21:00:00 +0100'), 1390939200)
-        self.assertEqual(
-            unified_timestamp('11/26/2014 11:30:00 AM PST', day_first=False),
-            1417001400)
-        self.assertEqual(
-            unified_timestamp('2/2/2015 6:47:40 PM', day_first=False),
-            1422902860)
-        self.assertEqual(unified_timestamp('Feb 14th 2016 5:45PM'), 1455471900)
-        self.assertEqual(unified_timestamp('25-09-2014'), 1411603200)
-        self.assertEqual(unified_timestamp('27.02.2016 17:30'), 1456594200)
-        self.assertEqual(unified_timestamp('UNKNOWN DATE FORMAT'), None)
-        self.assertEqual(unified_timestamp('May 16, 2016 11:15 PM'), 1463440500)
-
     def test_determine_ext(self):
         self.assertEqual(determine_ext('http://example.com/foo/bar.mp4/?download'), 'mp4')
         self.assertEqual(determine_ext('http://example.com/foo/bar/?download', None), None)
@@ -410,12 +381,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(res_url, url)
         self.assertEqual(res_data, None)
 
-        smug_url = smuggle_url(url, {'a': 'b'})
-        smug_smug_url = smuggle_url(smug_url, {'c': 'd'})
-        res_url, res_data = unsmuggle_url(smug_smug_url)
-        self.assertEqual(res_url, url)
-        self.assertEqual(res_data, {'a': 'b', 'c': 'd'})
-
     def test_shell_quote(self):
         args = ['ffmpeg', '-i', encodeFilename('ñ€ß\'.mp4')]
         self.assertEqual(shell_quote(args), """ffmpeg -i 'ñ€ß'"'"'.mp4'""")
@@ -434,20 +399,6 @@ class TestUtil(unittest.TestCase):
             url_basename('http://media.w3.org/2010/05/sintel/trailer.mp4'),
             'trailer.mp4')
 
-    def test_parse_age_limit(self):
-        self.assertEqual(parse_age_limit(None), None)
-        self.assertEqual(parse_age_limit(False), None)
-        self.assertEqual(parse_age_limit('invalid'), None)
-        self.assertEqual(parse_age_limit(0), 0)
-        self.assertEqual(parse_age_limit(18), 18)
-        self.assertEqual(parse_age_limit(21), 21)
-        self.assertEqual(parse_age_limit(22), None)
-        self.assertEqual(parse_age_limit('18'), 18)
-        self.assertEqual(parse_age_limit('18+'), 18)
-        self.assertEqual(parse_age_limit('PG-13'), 13)
-        self.assertEqual(parse_age_limit('TV-14'), 14)
-        self.assertEqual(parse_age_limit('TV-MA'), 17)
-
     def test_parse_duration(self):
         self.assertEqual(parse_duration(None), None)
         self.assertEqual(parse_duration(False), None)
@@ -626,37 +577,6 @@ class TestUtil(unittest.TestCase):
             limit_length('foo bar baz asd', 12).startswith('foo bar'))
         self.assertTrue('...' in limit_length('foo bar baz asd', 12))
 
-    def test_mimetype2ext(self):
-        self.assertEqual(mimetype2ext(None), None)
-        self.assertEqual(mimetype2ext('video/x-flv'), 'flv')
-        self.assertEqual(mimetype2ext('application/x-mpegURL'), 'm3u8')
-        self.assertEqual(mimetype2ext('text/vtt'), 'vtt')
-        self.assertEqual(mimetype2ext('text/vtt;charset=utf-8'), 'vtt')
-        self.assertEqual(mimetype2ext('text/html; charset=utf-8'), 'html')
-
-    def test_parse_codecs(self):
-        self.assertEqual(parse_codecs(''), {})
-        self.assertEqual(parse_codecs('avc1.77.30, mp4a.40.2'), {
-            'vcodec': 'avc1.77.30',
-            'acodec': 'mp4a.40.2',
-        })
-        self.assertEqual(parse_codecs('mp4a.40.2'), {
-            'vcodec': 'none',
-            'acodec': 'mp4a.40.2',
-        })
-        self.assertEqual(parse_codecs('mp4a.40.5,avc1.42001e'), {
-            'vcodec': 'avc1.42001e',
-            'acodec': 'mp4a.40.5',
-        })
-        self.assertEqual(parse_codecs('avc3.640028'), {
-            'vcodec': 'avc3.640028',
-            'acodec': 'none',
-        })
-        self.assertEqual(parse_codecs(', h264,,newcodec,aac'), {
-            'vcodec': 'h264',
-            'acodec': 'aac',
-        })
-
     def test_escape_rfc3986(self):
         reserved = "!*'();:@&=+$,/?#[]"
         unreserved = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~'
@@ -718,12 +638,6 @@ class TestUtil(unittest.TestCase):
             "1":{"src":"skipped", "type": "application/vnd.apple.mpegURL"}
         }''')
 
-        inp = '''{"foo":101}'''
-        self.assertEqual(js_to_json(inp), '''{"foo":101}''')
-
-        inp = '''{"duration": "00:01:07"}'''
-        self.assertEqual(js_to_json(inp), '''{"duration": "00:01:07"}''')
-
     def test_js_to_json_edgecases(self):
         on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
         self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
@@ -829,10 +743,7 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_filesize('2 MiB'), 2097152)
         self.assertEqual(parse_filesize('5 GB'), 5000000000)
         self.assertEqual(parse_filesize('1.2Tb'), 1200000000000)
-        self.assertEqual(parse_filesize('1.2tb'), 1200000000000)
         self.assertEqual(parse_filesize('1,24 KB'), 1240)
-        self.assertEqual(parse_filesize('1,24 kb'), 1240)
-        self.assertEqual(parse_filesize('8.5 megabytes'), 8500000)
 
     def test_parse_count(self):
         self.assertEqual(parse_count(None), None)
@@ -983,7 +894,6 @@ The first line
         self.assertEqual(cli_option({'proxy': '127.0.0.1:3128'}, '--proxy', 'proxy'), ['--proxy', '127.0.0.1:3128'])
         self.assertEqual(cli_option({'proxy': None}, '--proxy', 'proxy'), [])
         self.assertEqual(cli_option({}, '--proxy', 'proxy'), [])
-        self.assertEqual(cli_option({'retries': 10}, '--retries', 'retries'), ['--retries', '10'])
 
     def test_cli_valueless_option(self):
         self.assertEqual(cli_valueless_option(
@@ -1044,17 +954,5 @@ The first line
         self.assertRaises(ValueError, encode_base_n, 0, 70)
         self.assertRaises(ValueError, encode_base_n, 0, 60, custom_table)
 
-    def test_urshift(self):
-        self.assertEqual(urshift(3, 1), 1)
-        self.assertEqual(urshift(-3, 1), 2147483646)
-
-    def test_get_element_by_class(self):
-        html = '''
-            <span class="foo bar">nice</span>
-        '''
-
-        self.assertEqual(get_element_by_class('foo', html), 'nice')
-        self.assertEqual(get_element_by_class('no-such-class', html), None)
-
 if __name__ == '__main__':
     unittest.main()

test/test_verbose_output.py

@@ -1,70 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-from __future__ import unicode_literals
-
-import unittest
-
-import sys
-import os
-import subprocess
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-rootDir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-
-
-class TestVerboseOutput(unittest.TestCase):
-    def test_private_info_arg(self):
-        outp = subprocess.Popen(
-            [
-                sys.executable, 'youtube_dl/__main__.py', '-v',
-                '--username', 'johnsmith@gmail.com',
-                '--password', 'secret',
-            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        sout, serr = outp.communicate()
-        self.assertTrue(b'--username' in serr)
-        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue(b'--password' in serr)
-        self.assertTrue(b'secret' not in serr)
-
-    def test_private_info_shortarg(self):
-        outp = subprocess.Popen(
-            [
-                sys.executable, 'youtube_dl/__main__.py', '-v',
-                '-u', 'johnsmith@gmail.com',
-                '-p', 'secret',
-            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        sout, serr = outp.communicate()
-        self.assertTrue(b'-u' in serr)
-        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue(b'-p' in serr)
-        self.assertTrue(b'secret' not in serr)
-
-    def test_private_info_eq(self):
-        outp = subprocess.Popen(
-            [
-                sys.executable, 'youtube_dl/__main__.py', '-v',
-                '--username=johnsmith@gmail.com',
-                '--password=secret',
-            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        sout, serr = outp.communicate()
-        self.assertTrue(b'--username' in serr)
-        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue(b'--password' in serr)
-        self.assertTrue(b'secret' not in serr)
-
-    def test_private_info_shortarg_eq(self):
-        outp = subprocess.Popen(
-            [
-                sys.executable, 'youtube_dl/__main__.py', '-v',
-                '-u=johnsmith@gmail.com',
-                '-p=secret',
-            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        sout, serr = outp.communicate()
-        self.assertTrue(b'-u' in serr)
-        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue(b'-p' in serr)
-        self.assertTrue(b'secret' not in serr)
-
-if __name__ == '__main__':
-    unittest.main()

youtube_dl/YoutubeDL.py

@@ -5,7 +5,6 @@ from __future__ import absolute_import, unicode_literals
 
 import collections
 import contextlib
-import copy
 import datetime
 import errno
 import fileinput
@@ -197,8 +196,8 @@ class YoutubeDL(object):
     prefer_insecure:   Use HTTP instead of HTTPS to retrieve information.
                        At the moment, this is only supported by YouTube.
     proxy:             URL of the proxy server to use
-    geo_verification_proxy:  URL of the proxy to use for IP address verification
-                       on geo-restricted sites. (Experimental)
+    cn_verification_proxy:  URL of the proxy to use for IP address verification
+                       on Chinese sites. (Experimental)
     socket_timeout:    Time to wait for unresponsive hosts, in seconds
     bidi_workaround:   Work around buggy terminals without bidirectional text
                        support, using fridibi
@@ -249,16 +248,7 @@ class YoutubeDL(object):
     source_address:    (Experimental) Client-side IP address to bind to.
     call_home:         Boolean, true iff we are allowed to contact the
                        youtube-dl servers for debugging.
-    sleep_interval:    Number of seconds to sleep before each download when
-                       used alone or a lower bound of a range for randomized
-                       sleep before each download (minimum possible number
-                       of seconds to sleep) when used along with
-                       max_sleep_interval.
-    max_sleep_interval:Upper bound of a range for randomized sleep before each
-                       download (maximum possible number of seconds to sleep).
-                       Must only be used along with sleep_interval.
-                       Actual sleep time will be a random float from range
-                       [sleep_interval; max_sleep_interval].
+    sleep_interval:    Number of seconds to sleep before each download.
     listformats:       Print an overview of available video formats and exit.
     list_thumbnails:   Print a table of all thumbnails and exit.
     match_filter:      A function that gets called with the info_dict of
@@ -314,11 +304,6 @@ class YoutubeDL(object):
         self.params.update(params)
         self.cache = Cache(self)
 
-        if self.params.get('cn_verification_proxy') is not None:
-            self.report_warning('--cn-verification-proxy is deprecated. Use --geo-verification-proxy instead.')
-            if self.params.get('geo_verification_proxy') is None:
-                self.params['geo_verification_proxy'] = self.params['cn_verification_proxy']
-
         if params.get('bidi_workaround', False):
             try:
                 import pty
@@ -1061,9 +1046,9 @@ class YoutubeDL(object):
             if isinstance(selector, list):
                 fs = [_build_selector_function(s) for s in selector]
 
-                def selector_function(ctx):
+                def selector_function(formats):
                     for f in fs:
-                        for format in f(ctx):
+                        for format in f(formats):
                             yield format
                 return selector_function
             elif selector.type == GROUP:
@@ -1071,17 +1056,17 @@ class YoutubeDL(object):
             elif selector.type == PICKFIRST:
                 fs = [_build_selector_function(s) for s in selector.selector]
 
-                def selector_function(ctx):
+                def selector_function(formats):
                     for f in fs:
-                        picked_formats = list(f(ctx))
+                        picked_formats = list(f(formats))
                         if picked_formats:
                             return picked_formats
                     return []
             elif selector.type == SINGLE:
                 format_spec = selector.selector
 
-                def selector_function(ctx):
-                    formats = list(ctx['formats'])
+                def selector_function(formats):
+                    formats = list(formats)
                     if not formats:
                         return
                     if format_spec == 'all':
@@ -1094,10 +1079,9 @@ class YoutubeDL(object):
                             if f.get('vcodec') != 'none' and f.get('acodec') != 'none']
                         if audiovideo_formats:
                             yield audiovideo_formats[format_idx]
-                        # for extractors with incomplete formats (audio only (soundcloud)
-                        # or video only (imgur)) we will fallback to best/worst
-                        # {video,audio}-only format
-                        elif ctx['incomplete_formats']:
+                        # for audio only (soundcloud) or video only (imgur) urls, select the best/worst audio format
+                        elif (all(f.get('acodec') != 'none' for f in formats) or
+                              all(f.get('vcodec') != 'none' for f in formats)):
                             yield formats[format_idx]
                     elif format_spec == 'bestaudio':
                         audio_formats = [
@@ -1171,18 +1155,17 @@ class YoutubeDL(object):
                     }
                 video_selector, audio_selector = map(_build_selector_function, selector.selector)
 
-                def selector_function(ctx):
-                    for pair in itertools.product(
-                            video_selector(copy.deepcopy(ctx)), audio_selector(copy.deepcopy(ctx))):
+                def selector_function(formats):
+                    formats = list(formats)
+                    for pair in itertools.product(video_selector(formats), audio_selector(formats)):
                         yield _merge(pair)
 
             filters = [self._build_format_filter(f) for f in selector.filters]
 
-            def final_selector(ctx):
-                ctx_copy = copy.deepcopy(ctx)
+            def final_selector(formats):
                 for _filter in filters:
-                    ctx_copy['formats'] = list(filter(_filter, ctx_copy['formats']))
-                return selector_function(ctx_copy)
+                    formats = list(filter(_filter, formats))
+                return selector_function(formats)
             return final_selector
 
         stream = io.BytesIO(format_spec.encode('utf-8'))
@@ -1240,10 +1223,6 @@ class YoutubeDL(object):
         if 'title' not in info_dict:
             raise ExtractorError('Missing "title" field in extractor result')
 
-        if not isinstance(info_dict['id'], compat_str):
-            self.report_warning('"id" field is not a string - forcing string conversion')
-            info_dict['id'] = compat_str(info_dict['id'])
-
         if 'playlist' not in info_dict:
             # It isn't part of a playlist
             info_dict['playlist'] = None
@@ -1256,10 +1235,8 @@ class YoutubeDL(object):
                 info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
         if thumbnails:
             thumbnails.sort(key=lambda t: (
-                t.get('preference') if t.get('preference') is not None else -1,
-                t.get('width') if t.get('width') is not None else -1,
-                t.get('height') if t.get('height') is not None else -1,
-                t.get('id') if t.get('id') is not None else '', t.get('url')))
+                t.get('preference'), t.get('width'), t.get('height'),
+                t.get('id'), t.get('url')))
             for i, t in enumerate(thumbnails):
                 t['url'] = sanitize_url(t['url'])
                 if t.get('width') and t.get('height'):
@@ -1301,7 +1278,7 @@ class YoutubeDL(object):
                 for subtitle_format in subtitle:
                     if subtitle_format.get('url'):
                         subtitle_format['url'] = sanitize_url(subtitle_format['url'])
-                    if subtitle_format.get('ext') is None:
+                    if 'ext' not in subtitle_format:
                         subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
 
         if self.params.get('listsubtitles', False):
@@ -1356,7 +1333,7 @@ class YoutubeDL(object):
                     note=' ({0})'.format(format['format_note']) if format.get('format_note') is not None else '',
                 )
             # Automatically determine file extension if missing
-            if format.get('ext') is None:
+            if 'ext' not in format:
                 format['ext'] = determine_ext(format['url']).lower()
             # Automatically determine protocol if missing (useful for format
             # selection purposes)
@@ -1391,34 +1368,7 @@ class YoutubeDL(object):
             req_format_list.append('best')
             req_format = '/'.join(req_format_list)
         format_selector = self.build_format_selector(req_format)
-
-        # While in format selection we may need to have an access to the original
-        # format set in order to calculate some metrics or do some processing.
-        # For now we need to be able to guess whether original formats provided
-        # by extractor are incomplete or not (i.e. whether extractor provides only
-        # video-only or audio-only formats) for proper formats selection for
-        # extractors with such incomplete formats (see
-        # https://github.com/rg3/youtube-dl/pull/5556).
-        # Since formats may be filtered during format selection and may not match
-        # the original formats the results may be incorrect. Thus original formats
-        # or pre-calculated metrics should be passed to format selection routines
-        # as well.
-        # We will pass a context object containing all necessary additional data
-        # instead of just formats.
-        # This fixes incorrect format selection issue (see
-        # https://github.com/rg3/youtube-dl/issues/10083).
-        incomplete_formats = (
-            # All formats are video-only or
-            all(f.get('vcodec') != 'none' and f.get('acodec') == 'none' for f in formats) or
-            # all formats are audio-only
-            all(f.get('vcodec') == 'none' and f.get('acodec') != 'none' for f in formats))
-
-        ctx = {
-            'formats': formats,
-            'incomplete_formats': incomplete_formats,
-        }
-
-        formats_to_download = list(format_selector(ctx))
+        formats_to_download = list(format_selector(formats))
         if not formats_to_download:
             raise ExtractorError('requested format not available',
                                  expected=True)
@@ -1605,9 +1555,7 @@ class YoutubeDL(object):
                         self.to_screen('[info] Video subtitle %s.%s is already_present' % (sub_lang, sub_format))
                     else:
                         self.to_screen('[info] Writing video subtitles to: ' + sub_filename)
-                        # Use newline='' to prevent conversion of newline characters
-                        # See https://github.com/rg3/youtube-dl/issues/10268
-                        with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8', newline='') as subfile:
+                        with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8') as subfile:
                             subfile.write(sub_data)
                 except (OSError, IOError):
                     self.report_error('Cannot write subtitles file ' + sub_filename)

youtube_dl/__init__.py

@@ -18,6 +18,7 @@ from .options import (
 from .compat import (
     compat_expanduser,
     compat_getpass,
+    compat_print,
     compat_shlex_split,
     workaround_optparse_bug9161,
 )
@@ -75,7 +76,7 @@ def _real_main(argv=None):
 
     # Dump user agent
     if opts.dump_user_agent:
-        write_string(std_headers['User-Agent'] + '\n', out=sys.stdout)
+        compat_print(std_headers['User-Agent'])
         sys.exit(0)
 
     # Batch file verification
@@ -100,10 +101,10 @@ def _real_main(argv=None):
 
     if opts.list_extractors:
         for ie in list_extractors(opts.age_limit):
-            write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else '') + '\n', out=sys.stdout)
+            compat_print(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else ''))
             matchedUrls = [url for url in all_urls if ie.suitable(url)]
             for mu in matchedUrls:
-                write_string('  ' + mu + '\n', out=sys.stdout)
+                compat_print('  ' + mu)
         sys.exit(0)
     if opts.list_extractor_descriptions:
         for ie in list_extractors(opts.age_limit):
@@ -116,7 +117,7 @@ def _real_main(argv=None):
                 _SEARCHES = ('cute kittens', 'slithering pythons', 'falling cat', 'angry poodle', 'purple fish', 'running tortoise', 'sleeping bunny', 'burping cow')
                 _COUNTS = ('', '5', '10', 'all')
                 desc += ' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
-            write_string(desc + '\n', out=sys.stdout)
+            compat_print(desc)
         sys.exit(0)
 
     # Conflicting, missing and erroneous options
@@ -145,16 +146,6 @@ def _real_main(argv=None):
         if numeric_limit is None:
             parser.error('invalid max_filesize specified')
         opts.max_filesize = numeric_limit
-    if opts.sleep_interval is not None:
-        if opts.sleep_interval < 0:
-            parser.error('sleep interval must be positive or 0')
-    if opts.max_sleep_interval is not None:
-        if opts.max_sleep_interval < 0:
-            parser.error('max sleep interval must be positive or 0')
-        if opts.max_sleep_interval < opts.sleep_interval:
-            parser.error('max sleep interval must be greater than or equal to min sleep interval')
-    else:
-        opts.max_sleep_interval = opts.sleep_interval
 
     def parse_retries(retries):
         if retries in ('inf', 'infinite'):
@@ -318,7 +309,6 @@ def _real_main(argv=None):
         'nooverwrites': opts.nooverwrites,
         'retries': opts.retries,
         'fragment_retries': opts.fragment_retries,
-        'skip_unavailable_fragments': opts.skip_unavailable_fragments,
         'buffersize': opts.buffersize,
         'noresizebuffer': opts.noresizebuffer,
         'continuedl': opts.continue_dl,
@@ -381,7 +371,6 @@ def _real_main(argv=None):
         'source_address': opts.source_address,
         'call_home': opts.call_home,
         'sleep_interval': opts.sleep_interval,
-        'max_sleep_interval': opts.max_sleep_interval,
         'external_downloader': opts.external_downloader,
         'list_thumbnails': opts.list_thumbnails,
         'playlist_items': opts.playlist_items,
@@ -394,8 +383,6 @@ def _real_main(argv=None):
         'external_downloader_args': external_downloader_args,
         'postprocessor_args': postprocessor_args,
         'cn_verification_proxy': opts.cn_verification_proxy,
-        'geo_verification_proxy': opts.geo_verification_proxy,
-
     }
 
     with YoutubeDL(ydl_opts) as ydl:

youtube_dl/downloader/common.py

@@ -4,7 +4,6 @@ import os
 import re
 import sys
 import time
-import random
 
 from ..compat import compat_os_name
 from ..utils import (
@@ -343,11 +342,8 @@ class FileDownloader(object):
             })
             return True
 
-        min_sleep_interval = self.params.get('sleep_interval')
-        if min_sleep_interval:
-            max_sleep_interval = self.params.get('max_sleep_interval', min_sleep_interval)
-            print(min_sleep_interval, max_sleep_interval)
-            sleep_interval = random.uniform(min_sleep_interval, max_sleep_interval)
+        sleep_interval = self.params.get('sleep_interval')
+        if sleep_interval:
             self.to_screen('[download] Sleeping %s seconds...' % sleep_interval)
             time.sleep(sleep_interval)
 

youtube_dl/downloader/dash.py

@@ -38,10 +38,8 @@ class DashSegmentsFD(FragmentFD):
         segments_filenames = []
 
         fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
 
-        def process_segment(segment, tmp_filename, fatal):
-            target_url, segment_name = segment
+        def append_url_to_file(target_url, tmp_filename, segment_name):
             target_filename = '%s-%s' % (tmp_filename, segment_name)
             count = 0
             while count <= fragment_retries:
@@ -54,35 +52,26 @@ class DashSegmentsFD(FragmentFD):
                     down.close()
                     segments_filenames.append(target_sanitized)
                     break
-                except compat_urllib_error.HTTPError as err:
+                except (compat_urllib_error.HTTPError, ) as err:
                     # YouTube may often return 404 HTTP error for a fragment causing the
                     # whole download to fail. However if the same fragment is immediately
                     # retried with the same request data this usually succeeds (1-2 attemps
                     # is usually enough) thus allowing to download the whole file successfully.
-                    # To be future-proof we will retry all fragments that fail with any
-                    # HTTP error.
+                    # So, we will retry all fragments that fail with 404 HTTP error for now.
+                    if err.code != 404:
+                        raise
+                    # Retry fragment
                     count += 1
                     if count <= fragment_retries:
-                        self.report_retry_fragment(err, segment_name, count, fragment_retries)
+                        self.report_retry_fragment(segment_name, count, fragment_retries)
             if count > fragment_retries:
-                if not fatal:
-                    self.report_skip_fragment(segment_name)
-                    return True
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False
-            return True
 
-        segments_to_download = [(initialization_url, 'Init')] if initialization_url else []
-        segments_to_download.extend([
-            (segment_url, 'Seg%d' % i)
-            for i, segment_url in enumerate(segment_urls)])
-
-        for i, segment in enumerate(segments_to_download):
-            # In DASH, the first segment contains necessary headers to
-            # generate a valid MP4 file, so always abort for the first segment
-            fatal = i == 0 or not skip_unavailable_fragments
-            if not process_segment(segment, ctx['tmpfilename'], fatal):
-                return False
+        if initialization_url:
+            append_url_to_file(initialization_url, ctx['tmpfilename'], 'Init')
+        for i, segment_url in enumerate(segment_urls):
+            append_url_to_file(segment_url, ctx['tmpfilename'], 'Seg%d' % i)
 
         self._finish_frag_download(ctx)
 

youtube_dl/downloader/external.py

@@ -85,7 +85,7 @@ class ExternalFD(FileDownloader):
             cmd, stderr=subprocess.PIPE)
         _, stderr = p.communicate()
         if p.returncode != 0:
-            self.to_stderr(stderr.decode('utf-8', 'replace'))
+            self.to_stderr(stderr)
         return p.returncode
 
 
@@ -96,12 +96,6 @@ class CurlFD(ExternalFD):
         cmd = [self.exe, '--location', '-o', tmpfilename]
         for key, val in info_dict['http_headers'].items():
             cmd += ['--header', '%s: %s' % (key, val)]
-        cmd += self._bool_option('--continue-at', 'continuedl', '-', '0')
-        cmd += self._valueless_option('--silent', 'noprogress')
-        cmd += self._valueless_option('--verbose', 'verbose')
-        cmd += self._option('--limit-rate', 'ratelimit')
-        cmd += self._option('--retry', 'retries')
-        cmd += self._option('--max-filesize', 'max_filesize')
         cmd += self._option('--interface', 'source_address')
         cmd += self._option('--proxy', 'proxy')
         cmd += self._valueless_option('--insecure', 'nocheckcertificate')
@@ -109,16 +103,6 @@ class CurlFD(ExternalFD):
         cmd += ['--', info_dict['url']]
         return cmd
 
-    def _call_downloader(self, tmpfilename, info_dict):
-        cmd = [encodeArgument(a) for a in self._make_cmd(tmpfilename, info_dict)]
-
-        self._debug_cmd(cmd)
-
-        # curl writes the progress to stderr so don't capture it.
-        p = subprocess.Popen(cmd)
-        p.communicate()
-        return p.returncode
-
 
 class AxelFD(ExternalFD):
     AVAILABLE_OPT = '-V'
@@ -220,19 +204,12 @@ class FFmpegFD(ExternalFD):
         if proxy:
             if not re.match(r'^[\da-zA-Z]+://', proxy):
                 proxy = 'http://%s' % proxy
-
-            if proxy.startswith('socks'):
-                self.report_warning(
-                    '%s does not support SOCKS proxies. Downloading is likely to fail. '
-                    'Consider adding --hls-prefer-native to your command.' % self.get_basename())
-
             # Since December 2015 ffmpeg supports -http_proxy option (see
             # http://git.videolan.org/?p=ffmpeg.git;a=commit;h=b4eb1f29ebddd60c41a2eb39f5af701e38e0d3fd)
             # We could switch to the following code if we are able to detect version properly
             # args += ['-http_proxy', proxy]
             env = os.environ.copy()
             compat_setenv('HTTP_PROXY', proxy, env=env)
-            compat_setenv('http_proxy', proxy, env=env)
 
         protocol = info_dict.get('protocol')
 

youtube_dl/downloader/f4m.py

@@ -196,11 +196,6 @@ def build_fragments_list(boot_info):
     first_frag_number = fragment_run_entry_table[0]['first']
     fragments_counter = itertools.count(first_frag_number)
     for segment, fragments_count in segment_run_table['segment_run']:
-        # In some live HDS streams (for example Rai), `fragments_count` is
-        # abnormal and causing out-of-memory errors. It's OK to change the
-        # number of fragments for live streams as they are updated periodically
-        if fragments_count == 4294967295 and boot_info['live']:
-            fragments_count = 2
         for _ in range(fragments_count):
             res.append((segment, next(fragments_counter)))
 
@@ -334,11 +329,7 @@ class F4mFD(FragmentFD):
 
         base_url = compat_urlparse.urljoin(man_url, media.attrib['url'])
         bootstrap_node = doc.find(_add_ns('bootstrapInfo'))
-        # From Adobe F4M 3.0 spec:
-        # The <baseURL> element SHALL be the base URL for all relative
-        # (HTTP-based) URLs in the manifest. If <baseURL> is not present, said
-        # URLs should be relative to the location of the containing document.
-        boot_info, bootstrap_url = self._parse_bootstrap_node(bootstrap_node, man_url)
+        boot_info, bootstrap_url = self._parse_bootstrap_node(bootstrap_node, base_url)
         live = boot_info['live']
         metadata_node = media.find(_add_ns('metadata'))
         if metadata_node is not None:

youtube_dl/downloader/fragment.py

@@ -6,7 +6,6 @@ import time
 from .common import FileDownloader
 from .http import HttpFD
 from ..utils import (
-    error_to_compat_str,
     encodeFilename,
     sanitize_open,
 )
@@ -23,19 +22,13 @@ class FragmentFD(FileDownloader):
 
     Available options:
 
-    fragment_retries:   Number of times to retry a fragment for HTTP error (DASH
-                        and hlsnative only)
-    skip_unavailable_fragments:
-                        Skip unavailable fragments (DASH and hlsnative only)
+    fragment_retries:   Number of times to retry a fragment for HTTP error (DASH only)
     """
 
-    def report_retry_fragment(self, err, fragment_name, count, retries):
+    def report_retry_fragment(self, fragment_name, count, retries):
         self.to_screen(
-            '[download] Got server HTTP error: %s. Retrying fragment %s (attempt %d of %s)...'
-            % (error_to_compat_str(err), fragment_name, count, self.format_retries(retries)))
-
-    def report_skip_fragment(self, fragment_name):
-        self.to_screen('[download] Skipping fragment %s...' % fragment_name)
+            '[download] Got server HTTP error. Retrying fragment %s (attempt %d of %s)...'
+            % (fragment_name, count, self.format_retries(retries)))
 
     def _prepare_and_start_frag_download(self, ctx):
         self._prepare_frag_download(ctx)

youtube_dl/downloader/hls.py

@@ -2,26 +2,14 @@ from __future__ import unicode_literals
 
 import os.path
 import re
-import binascii
-try:
-    from Crypto.Cipher import AES
-    can_decrypt_frag = True
-except ImportError:
-    can_decrypt_frag = False
 
 from .fragment import FragmentFD
 from .external import FFmpegFD
 
-from ..compat import (
-    compat_urllib_error,
-    compat_urlparse,
-    compat_struct_pack,
-)
+from ..compat import compat_urlparse
 from ..utils import (
     encodeFilename,
     sanitize_open,
-    parse_m3u8_attributes,
-    update_url_query,
 )
 
 
@@ -33,27 +21,19 @@ class HlsFD(FragmentFD):
     @staticmethod
     def can_download(manifest):
         UNSUPPORTED_FEATURES = (
-            r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
+            r'#EXT-X-KEY:METHOD=(?!NONE)',  # encrypted streams [1]
             r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [2]
-
             # Live streams heuristic does not always work (e.g. geo restricted to Germany
             # http://hls-geo.daserste.de/i/videoportal/Film/c_620000/622873/format,716451,716457,716450,716458,716459,.mp4.csmil/index_4_av.m3u8?null=0)
             # r'#EXT-X-MEDIA-SEQUENCE:(?!0$)',  # live streams [3]
-
-            # This heuristic also is not correct since segments may not be appended as well.
-            # Twitch vods of finished streams have EXT-X-PLAYLIST-TYPE:EVENT despite
-            # no segments will definitely be appended to the end of the playlist.
-            # r'#EXT-X-PLAYLIST-TYPE:EVENT',  # media segments may be appended to the end of
-            #                                 # event media playlists [4]
-
+            r'#EXT-X-PLAYLIST-TYPE:EVENT',  # media segments may be appended to the end of
+                                            # event media playlists [4]
             # 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.4
             # 2. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
             # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
             # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
         )
-        check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
-        check_results.append(can_decrypt_frag or '#EXT-X-KEY:METHOD=AES-128' not in manifest)
-        return all(check_results)
+        return all(not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES)
 
     def real_download(self, filename, info_dict):
         man_url = info_dict['url']
@@ -71,94 +51,36 @@ class HlsFD(FragmentFD):
                 fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
-        total_frags = 0
+        fragment_urls = []
         for line in s.splitlines():
             line = line.strip()
             if line and not line.startswith('#'):
-                total_frags += 1
+                segment_url = (
+                    line
+                    if re.match(r'^https?://', line)
+                    else compat_urlparse.urljoin(man_url, line))
+                fragment_urls.append(segment_url)
+                # We only download the first fragment during the test
+                if self.params.get('test', False):
+                    break
 
         ctx = {
             'filename': filename,
-            'total_frags': total_frags,
+            'total_frags': len(fragment_urls),
         }
 
         self._prepare_and_start_frag_download(ctx)
 
-        fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
-        test = self.params.get('test', False)
-
-        extra_query = None
-        extra_param_to_segment_url = info_dict.get('extra_param_to_segment_url')
-        if extra_param_to_segment_url:
-            extra_query = compat_urlparse.parse_qs(extra_param_to_segment_url)
-        i = 0
-        media_sequence = 0
-        decrypt_info = {'METHOD': 'NONE'}
         frags_filenames = []
-        for line in s.splitlines():
-            line = line.strip()
-            if line:
-                if not line.startswith('#'):
-                    frag_url = (
-                        line
-                        if re.match(r'^https?://', line)
-                        else compat_urlparse.urljoin(man_url, line))
-                    frag_name = 'Frag%d' % i
-                    frag_filename = '%s-%s' % (ctx['tmpfilename'], frag_name)
-                    if extra_query:
-                        frag_url = update_url_query(frag_url, extra_query)
-                    count = 0
-                    while count <= fragment_retries:
-                        try:
-                            success = ctx['dl'].download(frag_filename, {'url': frag_url})
-                            if not success:
-                                return False
-                            down, frag_sanitized = sanitize_open(frag_filename, 'rb')
-                            frag_content = down.read()
-                            down.close()
-                            break
-                        except compat_urllib_error.HTTPError as err:
-                            # Unavailable (possibly temporary) fragments may be served.
-                            # First we try to retry then either skip or abort.
-                            # See https://github.com/rg3/youtube-dl/issues/10165,
-                            # https://github.com/rg3/youtube-dl/issues/10448).
-                            count += 1
-                            if count <= fragment_retries:
-                                self.report_retry_fragment(err, frag_name, count, fragment_retries)
-                    if count > fragment_retries:
-                        if skip_unavailable_fragments:
-                            i += 1
-                            media_sequence += 1
-                            self.report_skip_fragment(frag_name)
-                            continue
-                        self.report_error(
-                            'giving up after %s fragment retries' % fragment_retries)
-                        return False
-                    if decrypt_info['METHOD'] == 'AES-128':
-                        iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
-                        frag_content = AES.new(
-                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-                    ctx['dest_stream'].write(frag_content)
-                    frags_filenames.append(frag_sanitized)
-                    # We only download the first fragment during the test
-                    if test:
-                        break
-                    i += 1
-                    media_sequence += 1
-                elif line.startswith('#EXT-X-KEY'):
-                    decrypt_info = parse_m3u8_attributes(line[11:])
-                    if decrypt_info['METHOD'] == 'AES-128':
-                        if 'IV' in decrypt_info:
-                            decrypt_info['IV'] = binascii.unhexlify(decrypt_info['IV'][2:].zfill(32))
-                        if not re.match(r'^https?://', decrypt_info['URI']):
-                            decrypt_info['URI'] = compat_urlparse.urljoin(
-                                man_url, decrypt_info['URI'])
-                        if extra_query:
-                            decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
-                        decrypt_info['KEY'] = self.ydl.urlopen(decrypt_info['URI']).read()
-                elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
-                    media_sequence = int(line[22:])
+        for i, frag_url in enumerate(fragment_urls):
+            frag_filename = '%s-Frag%d' % (ctx['tmpfilename'], i)
+            success = ctx['dl'].download(frag_filename, {'url': frag_url})
+            if not success:
+                return False
+            down, frag_sanitized = sanitize_open(frag_filename, 'rb')
+            ctx['dest_stream'].write(down.read())
+            down.close()
+            frags_filenames.append(frag_sanitized)
 
         self._finish_frag_download(ctx)
 

youtube_dl/extractor/abc.py

@@ -7,7 +7,6 @@ from ..utils import (
     ExtractorError,
     js_to_json,
     int_or_none,
-    parse_iso8601,
 )
 
 
@@ -94,57 +93,3 @@ class ABCIE(InfoExtractor):
             'description': self._og_search_description(webpage),
             'thumbnail': self._og_search_thumbnail(webpage),
         }
-
-
-class ABCIViewIE(InfoExtractor):
-    IE_NAME = 'abc.net.au:iview'
-    _VALID_URL = r'https?://iview\.abc\.net\.au/programs/[^/]+/(?P<id>[^/?#]+)'
-
-    _TESTS = [{
-        'url': 'http://iview.abc.net.au/programs/gardening-australia/FA1505V024S00',
-        'md5': '979d10b2939101f0d27a06b79edad536',
-        'info_dict': {
-            'id': 'FA1505V024S00',
-            'ext': 'mp4',
-            'title': 'Series 27 Ep 24',
-            'description': 'md5:b28baeae7504d1148e1d2f0e3ed3c15d',
-            'upload_date': '20160820',
-            'uploader_id': 'abc1',
-            'timestamp': 1471719600,
-        },
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        video_params = self._parse_json(self._search_regex(
-            r'videoParams\s*=\s*({.+?});', webpage, 'video params'), video_id)
-        title = video_params['title']
-        stream = next(s for s in video_params['playlist'] if s.get('type') == 'program')
-
-        formats = self._extract_akamai_formats(stream['hds-unmetered'], video_id)
-        self._sort_formats(formats)
-
-        subtitles = {}
-        src_vtt = stream.get('captions', {}).get('src-vtt')
-        if src_vtt:
-            subtitles['en'] = [{
-                'url': src_vtt,
-                'ext': 'vtt',
-            }]
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': self._html_search_meta(['og:description', 'twitter:description'], webpage),
-            'thumbnail': self._html_search_meta(['og:image', 'twitter:image:src'], webpage),
-            'duration': int_or_none(video_params.get('eventDuration')),
-            'timestamp': parse_iso8601(video_params.get('pubDate'), ' '),
-            'series': video_params.get('seriesTitle'),
-            'series_id': video_params.get('seriesHouseNumber') or video_id[:7],
-            'episode_number': int_or_none(self._html_search_meta('episodeNumber', webpage)),
-            'episode': self._html_search_meta('episode_title', webpage),
-            'uploader_id': video_params.get('channel'),
-            'formats': formats,
-            'subtitles': subtitles,
-        }

youtube_dl/extractor/adobepass.py

@@ -1,137 +0,0 @@
-# -*- coding: utf-8 -*-
-from __future__ import unicode_literals
-
-import re
-import time
-import xml.etree.ElementTree as etree
-
-from .common import InfoExtractor
-from ..utils import (
-    unescapeHTML,
-    urlencode_postdata,
-    unified_timestamp,
-)
-
-
-class AdobePassIE(InfoExtractor):
-    _SERVICE_PROVIDER_TEMPLATE = 'https://sp.auth.adobe.com/adobe-services/%s'
-    _USER_AGENT = 'Mozilla/5.0 (X11; Linux i686; rv:47.0) Gecko/20100101 Firefox/47.0'
-
-    @staticmethod
-    def _get_mvpd_resource(provider_id, title, guid, rating):
-        channel = etree.Element('channel')
-        channel_title = etree.SubElement(channel, 'title')
-        channel_title.text = provider_id
-        item = etree.SubElement(channel, 'item')
-        resource_title = etree.SubElement(item, 'title')
-        resource_title.text = title
-        resource_guid = etree.SubElement(item, 'guid')
-        resource_guid.text = guid
-        resource_rating = etree.SubElement(item, 'media:rating')
-        resource_rating.attrib = {'scheme': 'urn:v-chip'}
-        resource_rating.text = rating
-        return '<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/">' + etree.tostring(channel).decode() + '</rss>'
-
-    def _extract_mvpd_auth(self, url, video_id, requestor_id, resource):
-        def xml_text(xml_str, tag):
-            return self._search_regex(
-                '<%s>(.+?)</%s>' % (tag, tag), xml_str, tag)
-
-        def is_expired(token, date_ele):
-            token_expires = unified_timestamp(re.sub(r'[_ ]GMT', '', xml_text(token, date_ele)))
-            return token_expires and token_expires <= int(time.time())
-
-        mvpd_headers = {
-            'ap_42': 'anonymous',
-            'ap_11': 'Linux i686',
-            'ap_z': self._USER_AGENT,
-            'User-Agent': self._USER_AGENT,
-        }
-
-        guid = xml_text(resource, 'guid')
-        requestor_info = self._downloader.cache.load('mvpd', requestor_id) or {}
-        authn_token = requestor_info.get('authn_token')
-        if authn_token and is_expired(authn_token, 'simpleTokenExpires'):
-            authn_token = None
-        if not authn_token:
-            # TODO add support for other TV Providers
-            mso_id = 'DTV'
-            username, password = self._get_netrc_login_info(mso_id)
-            if not username or not password:
-                return ''
-
-            def post_form(form_page, note, data={}):
-                post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_page, 'post url', group='url')
-                return self._download_webpage(
-                    post_url, video_id, note, data=urlencode_postdata(data or self._hidden_inputs(form_page)), headers={
-                        'Content-Type': 'application/x-www-form-urlencoded',
-                    })
-
-            provider_redirect_page = self._download_webpage(
-                self._SERVICE_PROVIDER_TEMPLATE % 'authenticate/saml', video_id,
-                'Downloading Provider Redirect Page', query={
-                    'noflash': 'true',
-                    'mso_id': mso_id,
-                    'requestor_id': requestor_id,
-                    'no_iframe': 'false',
-                    'domain_name': 'adobe.com',
-                    'redirect_url': url,
-                })
-            provider_login_page = post_form(
-                provider_redirect_page, 'Downloading Provider Login Page')
-            mvpd_confirm_page = post_form(provider_login_page, 'Logging in', {
-                'username': username,
-                'password': password,
-            })
-            post_form(mvpd_confirm_page, 'Confirming Login')
-
-            session = self._download_webpage(
-                self._SERVICE_PROVIDER_TEMPLATE % 'session', video_id,
-                'Retrieving Session', data=urlencode_postdata({
-                    '_method': 'GET',
-                    'requestor_id': requestor_id,
-                }), headers=mvpd_headers)
-            if '<pendingLogout' in session:
-                self._downloader.cache.store('mvpd', requestor_id, {})
-                return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
-            authn_token = unescapeHTML(xml_text(session, 'authnToken'))
-            requestor_info['authn_token'] = authn_token
-            self._downloader.cache.store('mvpd', requestor_id, requestor_info)
-
-        authz_token = requestor_info.get(guid)
-        if authz_token and is_expired(authz_token, 'simpleTokenTTL'):
-            authz_token = None
-        if not authz_token:
-            authorize = self._download_webpage(
-                self._SERVICE_PROVIDER_TEMPLATE % 'authorize', video_id,
-                'Retrieving Authorization Token', data=urlencode_postdata({
-                    'resource_id': resource,
-                    'requestor_id': requestor_id,
-                    'authentication_token': authn_token,
-                    'mso_id': xml_text(authn_token, 'simpleTokenMsoID'),
-                    'userMeta': '1',
-                }), headers=mvpd_headers)
-            if '<pendingLogout' in authorize:
-                self._downloader.cache.store('mvpd', requestor_id, {})
-                return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
-            authz_token = unescapeHTML(xml_text(authorize, 'authzToken'))
-            requestor_info[guid] = authz_token
-            self._downloader.cache.store('mvpd', requestor_id, requestor_info)
-
-        mvpd_headers.update({
-            'ap_19': xml_text(authn_token, 'simpleSamlNameID'),
-            'ap_23': xml_text(authn_token, 'simpleSamlSessionIndex'),
-        })
-
-        short_authorize = self._download_webpage(
-            self._SERVICE_PROVIDER_TEMPLATE % 'shortAuthorize',
-            video_id, 'Retrieving Media Token', data=urlencode_postdata({
-                'authz_token': authz_token,
-                'requestor_id': requestor_id,
-                'session_guid': xml_text(authn_token, 'simpleTokenAuthenticationGuid'),
-                'hashed_guid': 'false',
-            }), headers=mvpd_headers)
-        if '<pendingLogout' in short_authorize:
-            self._downloader.cache.store('mvpd', requestor_id, {})
-            return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
-        return short_authorize

youtube_dl/extractor/adobetv.py

@@ -156,10 +156,7 @@ class AdobeTVVideoIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-
-        video_data = self._parse_json(self._search_regex(
-            r'var\s+bridge\s*=\s*([^;]+);', webpage, 'bridged data'), video_id)
+        video_data = self._download_json(url + '?format=json', video_id)
 
         formats = [{
             'format_id': '%s-%s' % (determine_ext(source['src']), source.get('height')),

youtube_dl/extractor/adultswim.py

@@ -3,14 +3,16 @@ from __future__ import unicode_literals
 
 import re
 
-from .turner import TurnerBaseIE
+from .common import InfoExtractor
 from ..utils import (
+    determine_ext,
     ExtractorError,
-    int_or_none,
+    float_or_none,
+    xpath_text,
 )
 
 
-class AdultSwimIE(TurnerBaseIE):
+class AdultSwimIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?adultswim\.com/videos/(?P<is_playlist>playlists/)?(?P<show_path>[^/]+)/(?P<episode_path>[^/?#]+)/?'
 
     _TESTS = [{
@@ -81,21 +83,6 @@ class AdultSwimIE(TurnerBaseIE):
             # m3u8 download
             'skip_download': True,
         }
-    }, {
-        # heroMetadata.trailer
-        'url': 'http://www.adultswim.com/videos/decker/inside-decker-a-new-hero/',
-        'info_dict': {
-            'id': 'I0LQFQkaSUaFp8PnAWHhoQ',
-            'ext': 'mp4',
-            'title': 'Decker - Inside Decker: A New Hero',
-            'description': 'md5:c916df071d425d62d70c86d4399d3ee0',
-            'duration': 249.008,
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'expected_warnings': ['Unable to download f4m manifest'],
     }]
 
     @staticmethod
@@ -146,56 +133,79 @@ class AdultSwimIE(TurnerBaseIE):
             if video_info is None:
                 if bootstrapped_data.get('slugged_video', {}).get('slug') == episode_path:
                     video_info = bootstrapped_data['slugged_video']
-            if not video_info:
-                video_info = bootstrapped_data.get(
-                    'heroMetadata', {}).get('trailer', {}).get('video')
-            if not video_info:
-                video_info = bootstrapped_data.get('onlineOriginals', [None])[0]
-            if not video_info:
-                raise ExtractorError('Unable to find video info')
+                else:
+                    raise ExtractorError('Unable to find video info')
 
             show = bootstrapped_data['show']
             show_title = show['title']
             stream = video_info.get('stream')
-            if stream and stream.get('videoPlaybackID'):
-                segment_ids = [stream['videoPlaybackID']]
-            elif video_info.get('clips'):
-                segment_ids = [clip['videoPlaybackID'] for clip in video_info['clips']]
-            elif video_info.get('videoPlaybackID'):
-                segment_ids = [video_info['videoPlaybackID']]
-            else:
-                if video_info.get('auth') is True:
-                    raise ExtractorError(
-                        'This video is only available via cable service provider subscription that'
-                        ' is not currently supported. You may want to use --cookies.', expected=True)
-                else:
-                    raise ExtractorError('Unable to find stream or clips')
+            clips = [stream] if stream else video_info.get('clips')
+            if not clips:
+                raise ExtractorError(
+                    'This video is only available via cable service provider subscription that'
+                    ' is not currently supported. You may want to use --cookies.'
+                    if video_info.get('auth') is True else 'Unable to find stream or clips',
+                    expected=True)
+            segment_ids = [clip['videoPlaybackID'] for clip in clips]
 
         episode_id = video_info['id']
         episode_title = video_info['title']
-        episode_description = video_info.get('description')
-        episode_duration = int_or_none(video_info.get('duration'))
-        view_count = int_or_none(video_info.get('views'))
+        episode_description = video_info['description']
+        episode_duration = video_info.get('duration')
 
         entries = []
         for part_num, segment_id in enumerate(segment_ids):
-            segement_info = self._extract_cvp_info(
-                'http://www.adultswim.com/videos/api/v0/assets?id=%s&platform=desktop' % segment_id,
-                segment_id, {
-                    'secure': {
-                        'media_src': 'http://androidhls-secure.cdn.turner.com/adultswim/big',
-                        'tokenizer_src': 'http://www.adultswim.com/astv/mvpd/processors/services/token_ipadAdobe.do',
-                    },
-                })
+            segment_url = 'http://www.adultswim.com/videos/api/v0/assets?id=%s&platform=desktop' % segment_id
+
             segment_title = '%s - %s' % (show_title, episode_title)
             if len(segment_ids) > 1:
                 segment_title += ' Part %d' % (part_num + 1)
-            segement_info.update({
+
+            idoc = self._download_xml(
+                segment_url, segment_title,
+                'Downloading segment information', 'Unable to download segment information')
+
+            segment_duration = float_or_none(
+                xpath_text(idoc, './/trt', 'segment duration').strip())
+
+            formats = []
+            file_els = idoc.findall('.//files/file') or idoc.findall('./files/file')
+
+            unique_urls = []
+            unique_file_els = []
+            for file_el in file_els:
+                media_url = file_el.text
+                if not media_url or determine_ext(media_url) == 'f4m':
+                    continue
+                if file_el.text not in unique_urls:
+                    unique_urls.append(file_el.text)
+                    unique_file_els.append(file_el)
+
+            for file_el in unique_file_els:
+                bitrate = file_el.attrib.get('bitrate')
+                ftype = file_el.attrib.get('type')
+                media_url = file_el.text
+                if determine_ext(media_url) == 'm3u8':
+                    formats.extend(self._extract_m3u8_formats(
+                        media_url, segment_title, 'mp4', preference=0,
+                        m3u8_id='hls', fatal=False))
+                else:
+                    formats.append({
+                        'format_id': '%s_%s' % (bitrate, ftype),
+                        'url': file_el.text.strip(),
+                        # The bitrate may not be a number (for example: 'iphone')
+                        'tbr': int(bitrate) if bitrate.isdigit() else None,
+                    })
+
+            self._sort_formats(formats)
+
+            entries.append({
                 'id': segment_id,
                 'title': segment_title,
-                'description': episode_description,
+                'formats': formats,
+                'duration': segment_duration,
+                'description': episode_description
             })
-            entries.append(segement_info)
 
         return {
             '_type': 'playlist',
@@ -204,6 +214,5 @@ class AdultSwimIE(TurnerBaseIE):
             'entries': entries,
             'title': '%s - %s' % (show_title, episode_title),
             'description': episode_description,
-            'duration': episode_duration,
-            'view_count': view_count,
+            'duration': episode_duration
         }

youtube_dl/extractor/aenetworks.py

@@ -2,140 +2,23 @@ from __future__ import unicode_literals
 
 import re
 
-from .theplatform import ThePlatformIE
+from .common import InfoExtractor
 from ..utils import (
     smuggle_url,
     update_url_query,
     unescapeHTML,
-    extract_attributes,
-    get_element_by_attribute,
-)
-from ..compat import (
-    compat_urlparse,
 )
 
 
-class AENetworksBaseIE(ThePlatformIE):
-    _THEPLATFORM_KEY = 'crazyjava'
-    _THEPLATFORM_SECRET = 's3cr3t'
-
-
-class AENetworksIE(AENetworksBaseIE):
+class AENetworksIE(InfoExtractor):
     IE_NAME = 'aenetworks'
     IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network'
-    _VALID_URL = r'https?://(?:www\.)?(?P<domain>(?:history|aetv|mylifetime)\.com|fyi\.tv)/(?:shows/(?P<show_path>[^/]+(?:/[^/]+){0,2})|movies/(?P<movie_display_id>[^/]+)/full-movie)'
-    _TESTS = [{
-        'url': 'http://www.history.com/shows/mountain-men/season-1/episode-1',
-        'md5': '8ff93eb073449f151d6b90c0ae1ef0c7',
-        'info_dict': {
-            'id': '22253814',
-            'ext': 'mp4',
-            'title': 'Winter Is Coming',
-            'description': 'md5:641f424b7a19d8e24f26dea22cf59d74',
-            'timestamp': 1338306241,
-            'upload_date': '20120529',
-            'uploader': 'AENE-NEW',
-        },
-        'add_ie': ['ThePlatform'],
-    }, {
-        'url': 'http://www.history.com/shows/ancient-aliens/season-1',
-        'info_dict': {
-            'id': '71889446852',
-        },
-        'playlist_mincount': 5,
-    }, {
-        'url': 'http://www.mylifetime.com/shows/atlanta-plastic',
-        'info_dict': {
-            'id': 'SERIES4317',
-            'title': 'Atlanta Plastic',
-        },
-        'playlist_mincount': 2,
-    }, {
-        'url': 'http://www.aetv.com/shows/duck-dynasty/season-9/episode-1',
-        'only_matching': True
-    }, {
-        'url': 'http://www.fyi.tv/shows/tiny-house-nation/season-1/episode-8',
-        'only_matching': True
-    }, {
-        'url': 'http://www.mylifetime.com/shows/project-runway-junior/season-1/episode-6',
-        'only_matching': True
-    }, {
-        'url': 'http://www.mylifetime.com/movies/center-stage-on-pointe/full-movie',
-        'only_matching': True
-    }]
-    _DOMAIN_TO_REQUESTOR_ID = {
-        'history.com': 'HISTORY',
-        'aetv.com': 'AETV',
-        'mylifetime.com': 'LIFETIME',
-        'fyi.tv': 'FYI',
-    }
+    _VALID_URL = r'https?://(?:www\.)?(?:(?:history|aetv|mylifetime)\.com|fyi\.tv)/(?P<type>[^/]+)/(?:[^/]+/)+(?P<id>[^/]+?)(?:$|[?#])'
 
-    def _real_extract(self, url):
-        domain, show_path, movie_display_id = re.match(self._VALID_URL, url).groups()
-        display_id = show_path or movie_display_id
-        webpage = self._download_webpage(url, display_id)
-        if show_path:
-            url_parts = show_path.split('/')
-            url_parts_len = len(url_parts)
-            if url_parts_len == 1:
-                entries = []
-                for season_url_path in re.findall(r'(?s)<li[^>]+data-href="(/shows/%s/season-\d+)"' % url_parts[0], webpage):
-                    entries.append(self.url_result(
-                        compat_urlparse.urljoin(url, season_url_path), 'AENetworks'))
-                return self.playlist_result(
-                    entries, self._html_search_meta('aetn:SeriesId', webpage),
-                    self._html_search_meta('aetn:SeriesTitle', webpage))
-            elif url_parts_len == 2:
-                entries = []
-                for episode_item in re.findall(r'(?s)<div[^>]+class="[^"]*episode-item[^"]*"[^>]*>', webpage):
-                    episode_attributes = extract_attributes(episode_item)
-                    episode_url = compat_urlparse.urljoin(
-                        url, episode_attributes['data-canonical'])
-                    entries.append(self.url_result(
-                        episode_url, 'AENetworks',
-                        episode_attributes['data-videoid']))
-                return self.playlist_result(
-                    entries, self._html_search_meta('aetn:SeasonId', webpage))
-
-        query = {
-            'mbr': 'true',
-            'assetTypes': 'medium_video_s3'
-        }
-        video_id = self._html_search_meta('aetn:VideoID', webpage)
-        media_url = self._search_regex(
-            r"media_url\s*=\s*'([^']+)'", webpage, 'video url')
-        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
-            r'https?://link.theplatform.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
-        info = self._parse_theplatform_metadata(theplatform_metadata)
-        if theplatform_metadata.get('AETN$isBehindWall'):
-            requestor_id = self._DOMAIN_TO_REQUESTOR_ID[domain]
-            resource = self._get_mvpd_resource(
-                requestor_id, theplatform_metadata['title'],
-                theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
-                theplatform_metadata['ratings'][0]['rating'])
-            query['auth'] = self._extract_mvpd_auth(
-                url, video_id, requestor_id, resource)
-        info.update(self._search_json_ld(webpage, video_id, fatal=False))
-        media_url = update_url_query(media_url, query)
-        media_url = self._sign_url(media_url, self._THEPLATFORM_KEY, self._THEPLATFORM_SECRET)
-        formats, subtitles = self._extract_theplatform_smil(media_url, video_id)
-        self._sort_formats(formats)
-        info.update({
-            'id': video_id,
-            'formats': formats,
-            'subtitles': subtitles,
-        })
-        return info
-
-
-class HistoryTopicIE(AENetworksBaseIE):
-    IE_NAME = 'history:topic'
-    IE_DESC = 'History.com Topic'
-    _VALID_URL = r'https?://(?:www\.)?history\.com/topics/(?:[^/]+/)?(?P<topic_id>[^/]+)(?:/[^/]+(?:/(?P<video_display_id>[^/?#]+))?)?'
     _TESTS = [{
         'url': 'http://www.history.com/topics/valentines-day/history-of-valentines-day/videos/bet-you-didnt-know-valentines-day?m=528e394da93ae&s=undefined&f=1&free=false',
         'info_dict': {
-            'id': '40700995724',
+            'id': 'g12m5Gyt3fdR',
             'ext': 'mp4',
             'title': "Bet You Didn't Know: Valentine's Day",
             'description': 'md5:7b57ea4829b391995b405fa60bd7b5f7',
@@ -148,61 +31,57 @@ class HistoryTopicIE(AENetworksBaseIE):
             'skip_download': True,
         },
         'add_ie': ['ThePlatform'],
+        'expected_warnings': ['JSON-LD'],
     }, {
-        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/videos',
-        'info_dict':
-        {
-            'id': 'world-war-i-history',
-            'title': 'World War I History',
+        'url': 'http://www.history.com/shows/mountain-men/season-1/episode-1',
+        'md5': '8ff93eb073449f151d6b90c0ae1ef0c7',
+        'info_dict': {
+            'id': 'eg47EERs_JsZ',
+            'ext': 'mp4',
+            'title': 'Winter Is Coming',
+            'description': 'md5:641f424b7a19d8e24f26dea22cf59d74',
+            'timestamp': 1338306241,
+            'upload_date': '20120529',
+            'uploader': 'AENE-NEW',
         },
-        'playlist_mincount': 24,
+        'add_ie': ['ThePlatform'],
     }, {
-        'url': 'http://www.history.com/topics/world-war-i-history/videos',
-        'only_matching': True,
+        'url': 'http://www.aetv.com/shows/duck-dynasty/video/inlawful-entry',
+        'only_matching': True
     }, {
-        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history',
-        'only_matching': True,
+        'url': 'http://www.fyi.tv/shows/tiny-house-nation/videos/207-sq-ft-minnesota-prairie-cottage',
+        'only_matching': True
     }, {
-        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/speeches',
-        'only_matching': True,
+        'url': 'http://www.mylifetime.com/shows/project-runway-junior/video/season-1/episode-6/superstar-clients',
+        'only_matching': True
     }]
 
-    def theplatform_url_result(self, theplatform_url, video_id, query):
-        return {
+    def _real_extract(self, url):
+        page_type, video_id = re.match(self._VALID_URL, url).groups()
+
+        webpage = self._download_webpage(url, video_id)
+
+        video_url_re = [
+            r'data-href="[^"]*/%s"[^>]+data-release-url="([^"]+)"' % video_id,
+            r"media_url\s*=\s*'([^']+)'"
+        ]
+        video_url = unescapeHTML(self._search_regex(video_url_re, webpage, 'video url'))
+        query = {'mbr': 'true'}
+        if page_type == 'shows':
+            query['assetTypes'] = 'medium_video_s3'
+        if 'switch=hds' in video_url:
+            query['switch'] = 'hls'
+
+        info = self._search_json_ld(webpage, video_id, fatal=False)
+        info.update({
             '_type': 'url_transparent',
-            'id': video_id,
             'url': smuggle_url(
-                update_url_query(theplatform_url, query),
+                update_url_query(video_url, query),
                 {
                     'sig': {
-                        'key': self._THEPLATFORM_KEY,
-                        'secret': self._THEPLATFORM_SECRET,
-                    },
+                        'key': 'crazyjava',
+                        'secret': 's3cr3t'},
                     'force_smil_url': True
                 }),
-            'ie_key': 'ThePlatform',
-        }
-
-    def _real_extract(self, url):
-        topic_id, video_display_id = re.match(self._VALID_URL, url).groups()
-        if video_display_id:
-            webpage = self._download_webpage(url, video_display_id)
-            release_url, video_id = re.search(r"_videoPlayer.play\('([^']+)'\s*,\s*'[^']+'\s*,\s*'(\d+)'\)", webpage).groups()
-            release_url = unescapeHTML(release_url)
-
-            return self.theplatform_url_result(
-                release_url, video_id, {
-                    'mbr': 'true',
-                    'switch': 'hls'
-                })
-        else:
-            webpage = self._download_webpage(url, topic_id)
-            entries = []
-            for episode_item in re.findall(r'<a.+?data-release-url="[^"]+"[^>]*>', webpage):
-                video_attributes = extract_attributes(episode_item)
-                entries.append(self.theplatform_url_result(
-                    video_attributes['data-release-url'], video_attributes['data-id'], {
-                        'mbr': 'true',
-                        'switch': 'hls'
-                    }))
-            return self.playlist_result(entries, topic_id, get_element_by_attribute('class', 'show-title', webpage))
+        })
+        return info

youtube_dl/extractor/afreecatv.py

@@ -1,133 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..compat import (
-    compat_urllib_parse_urlparse,
-    compat_urlparse,
-)
-from ..utils import (
-    ExtractorError,
-    int_or_none,
-    xpath_element,
-    xpath_text,
-)
-
-
-class AfreecaTVIE(InfoExtractor):
-    IE_DESC = 'afreecatv.com'
-    _VALID_URL = r'''(?x)^
-        https?://(?:(live|afbbs|www)\.)?afreeca(?:tv)?\.com(?::\d+)?
-        (?:
-            /app/(?:index|read_ucc_bbs)\.cgi|
-            /player/[Pp]layer\.(?:swf|html))
-        \?.*?\bnTitleNo=(?P<id>\d+)'''
-    _TESTS = [{
-        'url': 'http://live.afreecatv.com:8079/app/index.cgi?szType=read_ucc_bbs&szBjId=dailyapril&nStationNo=16711924&nBbsNo=18605867&nTitleNo=36164052&szSkin=',
-        'md5': 'f72c89fe7ecc14c1b5ce506c4996046e',
-        'info_dict': {
-            'id': '36164052',
-            'ext': 'mp4',
-            'title': '데일리 에이프릴 요정들의 시상식!',
-            'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
-            'uploader': 'dailyapril',
-            'uploader_id': 'dailyapril',
-            'upload_date': '20160503',
-        }
-    }, {
-        'url': 'http://afbbs.afreecatv.com:8080/app/read_ucc_bbs.cgi?nStationNo=16711924&nTitleNo=36153164&szBjId=dailyapril&nBbsNo=18605867',
-        'info_dict': {
-            'id': '36153164',
-            'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
-            'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
-            'uploader': 'dailyapril',
-            'uploader_id': 'dailyapril',
-        },
-        'playlist_count': 2,
-        'playlist': [{
-            'md5': 'd8b7c174568da61d774ef0203159bf97',
-            'info_dict': {
-                'id': '36153164_1',
-                'ext': 'mp4',
-                'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
-                'upload_date': '20160502',
-            },
-        }, {
-            'md5': '58f2ce7f6044e34439ab2d50612ab02b',
-            'info_dict': {
-                'id': '36153164_2',
-                'ext': 'mp4',
-                'title': "BJ유트루와 함께하는 '팅커벨 메이크업!'",
-                'upload_date': '20160502',
-            },
-        }],
-    }, {
-        'url': 'http://www.afreecatv.com/player/Player.swf?szType=szBjId=djleegoon&nStationNo=11273158&nBbsNo=13161095&nTitleNo=36327652',
-        'only_matching': True,
-    }]
-
-    @staticmethod
-    def parse_video_key(key):
-        video_key = {}
-        m = re.match(r'^(?P<upload_date>\d{8})_\w+_(?P<part>\d+)$', key)
-        if m:
-            video_key['upload_date'] = m.group('upload_date')
-            video_key['part'] = m.group('part')
-        return video_key
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        parsed_url = compat_urllib_parse_urlparse(url)
-        info_url = compat_urlparse.urlunparse(parsed_url._replace(
-            netloc='afbbs.afreecatv.com:8080',
-            path='/api/video/get_video_info.php'))
-        video_xml = self._download_xml(info_url, video_id)
-
-        if xpath_element(video_xml, './track/video/file') is None:
-            raise ExtractorError('Specified AfreecaTV video does not exist',
-                                 expected=True)
-
-        title = xpath_text(video_xml, './track/title', 'title')
-        uploader = xpath_text(video_xml, './track/nickname', 'uploader')
-        uploader_id = xpath_text(video_xml, './track/bj_id', 'uploader id')
-        duration = int_or_none(xpath_text(video_xml, './track/duration',
-                                          'duration'))
-        thumbnail = xpath_text(video_xml, './track/titleImage', 'thumbnail')
-
-        entries = []
-        for i, video_file in enumerate(video_xml.findall('./track/video/file')):
-            video_key = self.parse_video_key(video_file.get('key', ''))
-            if not video_key:
-                continue
-            entries.append({
-                'id': '%s_%s' % (video_id, video_key.get('part', i + 1)),
-                'title': title,
-                'upload_date': video_key.get('upload_date'),
-                'duration': int_or_none(video_file.get('duration')),
-                'url': video_file.text,
-            })
-
-        info = {
-            'id': video_id,
-            'title': title,
-            'uploader': uploader,
-            'uploader_id': uploader_id,
-            'duration': duration,
-            'thumbnail': thumbnail,
-        }
-
-        if len(entries) > 1:
-            info['_type'] = 'multi_video'
-            info['entries'] = entries
-        elif len(entries) == 1:
-            info['url'] = entries[0]['url']
-            info['upload_date'] = entries[0].get('upload_date')
-        else:
-            raise ExtractorError(
-                'No files found for the specified AfreecaTV video, either'
-                ' the URL is incorrect or the video has been made private.',
-                expected=True)
-
-        return info

youtube_dl/extractor/aftonbladet.py

@@ -24,10 +24,10 @@ class AftonbladetIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
 
         # find internal video meta data
-        meta_url = 'http://aftonbladet-play-metadata.cdn.drvideo.aptoma.no/video/%s.json'
+        meta_url = 'http://aftonbladet-play.drlib.aptoma.no/video/%s.json'
         player_config = self._parse_json(self._html_search_regex(
             r'data-player-config="([^"]+)"', webpage, 'player config'), video_id)
-        internal_meta_id = player_config['aptomaVideoId']
+        internal_meta_id = player_config['videoId']
         internal_meta_url = meta_url % internal_meta_id
         internal_meta_json = self._download_json(
             internal_meta_url, video_id, 'Downloading video meta data')

youtube_dl/extractor/amcnetworks.py

@@ -1,91 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .theplatform import ThePlatformIE
-from ..utils import (
-    update_url_query,
-    parse_age_limit,
-    int_or_none,
-)
-
-
-class AMCNetworksIE(ThePlatformIE):
-    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|wetv)\.com/(?:movies/|shows/[^/]+/(?:full-episodes/)?season-\d+/episode-\d+(?:-(?:[^/]+/)?|/))(?P<id>[^/?#]+)'
-    _TESTS = [{
-        'url': 'http://www.ifc.com/shows/maron/season-04/episode-01/step-1',
-        'md5': '',
-        'info_dict': {
-            'id': 's3MX01Nl4vPH',
-            'ext': 'mp4',
-            'title': 'Maron - Season 4 - Step 1',
-            'description': 'In denial about his current situation, Marc is reluctantly convinced by his friends to enter rehab. Starring Marc Maron and Constance Zimmer.',
-            'age_limit': 17,
-            'upload_date': '20160505',
-            'timestamp': 1462468831,
-            'uploader': 'AMCN',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://www.bbcamerica.com/shows/the-hunt/full-episodes/season-1/episode-01-the-hardest-challenge',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.amc.com/shows/preacher/full-episodes/season-01/episode-00/pilot',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.wetv.com/shows/million-dollar-matchmaker/season-01/episode-06-the-dumped-dj-and-shallow-hal',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.ifc.com/movies/chaos',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        query = {
-            'mbr': 'true',
-            'manifest': 'm3u',
-        }
-        media_url = self._search_regex(r'window\.platformLinkURL\s*=\s*[\'"]([^\'"]+)', webpage, 'media url')
-        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
-            r'https?://link.theplatform.com/s/([^?]+)', media_url, 'theplatform_path'), display_id)
-        info = self._parse_theplatform_metadata(theplatform_metadata)
-        video_id = theplatform_metadata['pid']
-        title = theplatform_metadata['title']
-        rating = theplatform_metadata['ratings'][0]['rating']
-        auth_required = self._search_regex(r'window\.authRequired\s*=\s*(true|false);', webpage, 'auth required')
-        if auth_required == 'true':
-            requestor_id = self._search_regex(r'window\.requestor_id\s*=\s*[\'"]([^\'"]+)', webpage, 'requestor id')
-            resource = self._get_mvpd_resource(requestor_id, title, video_id, rating)
-            query['auth'] = self._extract_mvpd_auth(url, video_id, requestor_id, resource)
-        media_url = update_url_query(media_url, query)
-        formats, subtitles = self._extract_theplatform_smil(media_url, video_id)
-        self._sort_formats(formats)
-        info.update({
-            'id': video_id,
-            'subtitles': subtitles,
-            'formats': formats,
-            'age_limit': parse_age_limit(parse_age_limit(rating)),
-        })
-        ns_keys = theplatform_metadata.get('$xmlns', {}).keys()
-        if ns_keys:
-            ns = list(ns_keys)[0]
-            series = theplatform_metadata.get(ns + '$show')
-            season_number = int_or_none(theplatform_metadata.get(ns + '$season'))
-            episode = theplatform_metadata.get(ns + '$episodeTitle')
-            episode_number = int_or_none(theplatform_metadata.get(ns + '$episode'))
-            if season_number:
-                title = 'Season %d - %s' % (season_number, title)
-            if series:
-                title = '%s - %s' % (series, title)
-            info.update({
-                'title': title,
-                'series': series,
-                'season_number': season_number,
-                'episode': episode,
-                'episode_number': episode_number,
-            })
-        return info

youtube_dl/extractor/amp.py

@@ -5,8 +5,6 @@ from .common import InfoExtractor
 from ..utils import (
     int_or_none,
     parse_iso8601,
-    mimetype2ext,
-    determine_ext,
 )
 
 
@@ -52,25 +50,21 @@ class AMPIE(InfoExtractor):
         if isinstance(media_content, dict):
             media_content = [media_content]
         for media_data in media_content:
-            media = media_data.get('@attributes', {})
-            media_url = media.get('url')
-            if not media_url:
-                continue
-            ext = mimetype2ext(media.get('type')) or determine_ext(media_url)
-            if ext == 'f4m':
+            media = media_data['@attributes']
+            media_type = media['type']
+            if media_type in ('video/f4m', 'application/f4m+xml'):
                 formats.extend(self._extract_f4m_formats(
-                    media_url + '?hdcore=3.4.0&plugin=aasp-3.4.0.132.124',
+                    media['url'] + '?hdcore=3.4.0&plugin=aasp-3.4.0.132.124',
                     video_id, f4m_id='hds', fatal=False))
-            elif ext == 'm3u8':
+            elif media_type == 'application/x-mpegURL':
                 formats.extend(self._extract_m3u8_formats(
-                    media_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+                    media['url'], video_id, 'mp4', m3u8_id='hls', fatal=False))
             else:
                 formats.append({
                     'format_id': media_data.get('media-category', {}).get('@attributes', {}).get('label'),
                     'url': media['url'],
                     'tbr': int_or_none(media.get('bitrate')),
                     'filesize': int_or_none(media.get('fileSize')),
-                    'ext': ext,
                 })
 
         self._sort_formats(formats)

youtube_dl/extractor/animeondemand.py

@@ -22,7 +22,6 @@ class AnimeOnDemandIE(InfoExtractor):
     _APPLY_HTML5_URL = 'https://www.anime-on-demand.de/html5apply'
     _NETRC_MACHINE = 'animeondemand'
     _TESTS = [{
-        # jap, OmU
         'url': 'https://www.anime-on-demand.de/anime/161',
         'info_dict': {
             'id': '161',
@@ -31,21 +30,17 @@ class AnimeOnDemandIE(InfoExtractor):
         },
         'playlist_mincount': 4,
     }, {
-        # Film wording is used instead of Episode, ger/jap, Dub/OmU
+        # Film wording is used instead of Episode
         'url': 'https://www.anime-on-demand.de/anime/39',
         'only_matching': True,
     }, {
-        # Episodes without titles, jap, OmU
+        # Episodes without titles
         'url': 'https://www.anime-on-demand.de/anime/162',
         'only_matching': True,
     }, {
         # ger/jap, Dub/OmU, account required
         'url': 'https://www.anime-on-demand.de/anime/169',
         'only_matching': True,
-    }, {
-        # Full length film, non-series, ger/jap, Dub/OmU, account required
-        'url': 'https://www.anime-on-demand.de/anime/185',
-        'only_matching': True,
     }]
 
     def _login(self):
@@ -115,12 +110,35 @@ class AnimeOnDemandIE(InfoExtractor):
 
         entries = []
 
-        def extract_info(html, video_id, num=None):
-            title, description = [None] * 2
+        for num, episode_html in enumerate(re.findall(
+                r'(?s)<h3[^>]+class="episodebox-title".+?>Episodeninhalt<', webpage), 1):
+            episodebox_title = self._search_regex(
+                (r'class="episodebox-title"[^>]+title=(["\'])(?P<title>.+?)\1',
+                 r'class="episodebox-title"[^>]+>(?P<title>.+?)<'),
+                episode_html, 'episodebox title', default=None, group='title')
+            if not episodebox_title:
+                continue
+
+            episode_number = int(self._search_regex(
+                r'(?:Episode|Film)\s*(\d+)',
+                episodebox_title, 'episode number', default=num))
+            episode_title = self._search_regex(
+                r'(?:Episode|Film)\s*\d+\s*-\s*(.+)',
+                episodebox_title, 'episode title', default=None)
+
+            video_id = 'episode-%d' % episode_number
+
+            common_info = {
+                'id': video_id,
+                'series': anime_title,
+                'episode': episode_title,
+                'episode_number': episode_number,
+            }
+
             formats = []
 
             for input_ in re.findall(
-                    r'<input[^>]+class=["\'].*?streamstarter_html5[^>]+>', html):
+                    r'<input[^>]+class=["\'].*?streamstarter_html5[^>]+>', episode_html):
                 attributes = extract_attributes(input_)
                 playlist_urls = []
                 for playlist_key in ('data-playlist', 'data-otherplaylist'):
@@ -143,7 +161,7 @@ class AnimeOnDemandIE(InfoExtractor):
                         format_id_list.append(lang)
                     if kind:
                         format_id_list.append(kind)
-                    if not format_id_list and num is not None:
+                    if not format_id_list:
                         format_id_list.append(compat_str(num))
                     format_id = '-'.join(format_id_list)
                     format_note = ', '.join(filter(None, (kind, lang_note)))
@@ -197,74 +215,28 @@ class AnimeOnDemandIE(InfoExtractor):
                             })
                         formats.extend(file_formats)
 
-            return {
-                'title': title,
-                'description': description,
-                'formats': formats,
-            }
-
-        def extract_entries(html, video_id, common_info, num=None):
-            info = extract_info(html, video_id, num)
-
-            if info['formats']:
-                self._sort_formats(info['formats'])
+            if formats:
+                self._sort_formats(formats)
                 f = common_info.copy()
-                f.update(info)
+                f.update({
+                    'title': title,
+                    'description': description,
+                    'formats': formats,
+                })
                 entries.append(f)
 
-            # Extract teaser/trailer only when full episode is not available
-            if not info['formats']:
+            # Extract teaser only when full episode is not available
+            if not formats:
                 m = re.search(
-                    r'data-dialog-header=(["\'])(?P<title>.+?)\1[^>]+href=(["\'])(?P<href>.+?)\3[^>]*>(?P<kind>Teaser|Trailer)<',
-                    html)
+                    r'data-dialog-header=(["\'])(?P<title>.+?)\1[^>]+href=(["\'])(?P<href>.+?)\3[^>]*>Teaser<',
+                    episode_html)
                 if m:
                     f = common_info.copy()
                     f.update({
-                        'id': '%s-%s' % (f['id'], m.group('kind').lower()),
+                        'id': '%s-teaser' % f['id'],
                         'title': m.group('title'),
                         'url': compat_urlparse.urljoin(url, m.group('href')),
                     })
                     entries.append(f)
 
-        def extract_episodes(html):
-            for num, episode_html in enumerate(re.findall(
-                    r'(?s)<h3[^>]+class="episodebox-title".+?>Episodeninhalt<', html), 1):
-                episodebox_title = self._search_regex(
-                    (r'class="episodebox-title"[^>]+title=(["\'])(?P<title>.+?)\1',
-                     r'class="episodebox-title"[^>]+>(?P<title>.+?)<'),
-                    episode_html, 'episodebox title', default=None, group='title')
-                if not episodebox_title:
-                    continue
-
-                episode_number = int(self._search_regex(
-                    r'(?:Episode|Film)\s*(\d+)',
-                    episodebox_title, 'episode number', default=num))
-                episode_title = self._search_regex(
-                    r'(?:Episode|Film)\s*\d+\s*-\s*(.+)',
-                    episodebox_title, 'episode title', default=None)
-
-                video_id = 'episode-%d' % episode_number
-
-                common_info = {
-                    'id': video_id,
-                    'series': anime_title,
-                    'episode': episode_title,
-                    'episode_number': episode_number,
-                }
-
-                extract_entries(episode_html, video_id, common_info)
-
-        def extract_film(html, video_id):
-            common_info = {
-                'id': anime_id,
-                'title': anime_title,
-                'description': anime_description,
-            }
-            extract_entries(html, video_id, common_info)
-
-        extract_episodes(webpage)
-
-        if not entries:
-            extract_film(webpage, anime_id)
-
         return self.playlist_result(entries, anime_id, anime_title, anime_description)

youtube_dl/extractor/aol.py

@@ -123,10 +123,6 @@ class AolFeaturesIE(InfoExtractor):
             'title': 'What To Watch - February 17, 2016',
         },
         'add_ie': ['FiveMin'],
-        'params': {
-            # encrypted m3u8 download
-            'skip_download': True,
-        },
     }]
 
     def _real_extract(self, url):

youtube_dl/extractor/aparat.py

@@ -1,6 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import re
+
 from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
@@ -13,7 +15,7 @@ class AparatIE(InfoExtractor):
 
     _TEST = {
         'url': 'http://www.aparat.com/v/wP8On',
-        'md5': '131aca2e14fe7c4dcb3c4877ba300c89',
+        'md5': '6714e0af7e0d875c5a39c4dc4ab46ad1',
         'info_dict': {
             'id': 'wP8On',
             'ext': 'mp4',
@@ -29,13 +31,13 @@ class AparatIE(InfoExtractor):
         # Note: There is an easier-to-parse configuration at
         # http://www.aparat.com/video/video/config/videohash/%video_id
         # but the URL in there does not work
-        embed_url = 'http://www.aparat.com/video/video/embed/vt/frame/showvideo/yes/videohash/' + video_id
+        embed_url = ('http://www.aparat.com/video/video/embed/videohash/' +
+                     video_id + '/vt/frame')
         webpage = self._download_webpage(embed_url, video_id)
 
-        file_list = self._parse_json(self._search_regex(
-            r'fileList\s*=\s*JSON\.parse\(\'([^\']+)\'\)', webpage, 'file list'), video_id)
-        for i, item in enumerate(file_list[0]):
-            video_url = item['file']
+        video_urls = [video_url.replace('\\/', '/') for video_url in re.findall(
+            r'(?:fileList\[[0-9]+\]\s*=|"file"\s*:)\s*"([^"]+)"', webpage)]
+        for i, video_url in enumerate(video_urls):
             req = HEADRequest(video_url)
             res = self._request_webpage(
                 req, video_id, note='Testing video URL %d' % i, errnote=False)

youtube_dl/extractor/appletrailers.py

@@ -7,8 +7,6 @@ from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
     int_or_none,
-    parse_duration,
-    unified_strdate,
 )
 
 
@@ -18,8 +16,7 @@ class AppleTrailersIE(InfoExtractor):
     _TESTS = [{
         'url': 'http://trailers.apple.com/trailers/wb/manofsteel/',
         'info_dict': {
-            'id': '5111',
-            'title': 'Man of Steel',
+            'id': 'manofsteel',
         },
         'playlist': [
             {
@@ -73,15 +70,6 @@ class AppleTrailersIE(InfoExtractor):
             'id': 'blackthorn',
         },
         'playlist_mincount': 2,
-        'expected_warnings': ['Unable to download JSON metadata'],
-    }, {
-        # json data only available from http://trailers.apple.com/trailers/feeds/data/15881.json
-        'url': 'http://trailers.apple.com/trailers/fox/kungfupanda3/',
-        'info_dict': {
-            'id': '15881',
-            'title': 'Kung Fu Panda 3',
-        },
-        'playlist_mincount': 4,
     }, {
         'url': 'http://trailers.apple.com/ca/metropole/autrui/',
         'only_matching': True,
@@ -97,45 +85,6 @@ class AppleTrailersIE(InfoExtractor):
         movie = mobj.group('movie')
         uploader_id = mobj.group('company')
 
-        webpage = self._download_webpage(url, movie)
-        film_id = self._search_regex(r"FilmId\s*=\s*'(\d+)'", webpage, 'film id')
-        film_data = self._download_json(
-            'http://trailers.apple.com/trailers/feeds/data/%s.json' % film_id,
-            film_id, fatal=False)
-
-        if film_data:
-            entries = []
-            for clip in film_data.get('clips', []):
-                clip_title = clip['title']
-
-                formats = []
-                for version, version_data in clip.get('versions', {}).items():
-                    for size, size_data in version_data.get('sizes', {}).items():
-                        src = size_data.get('src')
-                        if not src:
-                            continue
-                        formats.append({
-                            'format_id': '%s-%s' % (version, size),
-                            'url': re.sub(r'_(\d+p.mov)', r'_h\1', src),
-                            'width': int_or_none(size_data.get('width')),
-                            'height': int_or_none(size_data.get('height')),
-                            'language': version[:2],
-                        })
-                self._sort_formats(formats)
-
-                entries.append({
-                    'id': movie + '-' + re.sub(r'[^a-zA-Z0-9]', '', clip_title).lower(),
-                    'formats': formats,
-                    'title': clip_title,
-                    'thumbnail': clip.get('screen') or clip.get('thumb'),
-                    'duration': parse_duration(clip.get('runtime') or clip.get('faded')),
-                    'upload_date': unified_strdate(clip.get('posted')),
-                    'uploader_id': uploader_id,
-                })
-
-            page_data = film_data.get('page', {})
-            return self.playlist_result(entries, film_id, page_data.get('movie_title'))
-
         playlist_url = compat_urlparse.urljoin(url, 'includes/playlists/itunes.inc')
 
         def fix_html(s):

youtube_dl/extractor/archiveorg.py

@@ -1,65 +1,67 @@
 from __future__ import unicode_literals
 
-from .jwplatform import JWPlatformBaseIE
-from ..utils import (
-    unified_strdate,
-    clean_html,
-)
+from .common import InfoExtractor
+from ..utils import unified_strdate
 
 
-class ArchiveOrgIE(JWPlatformBaseIE):
+class ArchiveOrgIE(InfoExtractor):
     IE_NAME = 'archive.org'
     IE_DESC = 'archive.org videos'
-    _VALID_URL = r'https?://(?:www\.)?archive\.org/(?:details|embed)/(?P<id>[^/?#]+)(?:[?].*)?$'
+    _VALID_URL = r'https?://(?:www\.)?archive\.org/details/(?P<id>[^?/]+)(?:[?].*)?$'
     _TESTS = [{
         'url': 'http://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
         'md5': '8af1d4cf447933ed3c7f4871162602db',
         'info_dict': {
             'id': 'XD300-23_68HighlightsAResearchCntAugHumanIntellect',
-            'ext': 'ogg',
+            'ext': 'ogv',
             'title': '1968 Demo - FJCC Conference Presentation Reel #1',
-            'description': 'md5:da45c349df039f1cc8075268eb1b5c25',
+            'description': 'md5:1780b464abaca9991d8968c877bb53ed',
             'upload_date': '19681210',
             'uploader': 'SRI International'
         }
     }, {
         'url': 'https://archive.org/details/Cops1922',
-        'md5': 'bc73c8ab3838b5a8fc6c6651fa7b58ba',
+        'md5': '18f2a19e6d89af8425671da1cf3d4e04',
         'info_dict': {
             'id': 'Cops1922',
-            'ext': 'mp4',
+            'ext': 'ogv',
             'title': 'Buster Keaton\'s "Cops" (1922)',
-            'description': 'md5:b4544662605877edd99df22f9620d858',
+            'description': 'md5:70f72ee70882f713d4578725461ffcc3',
         }
-    }, {
-        'url': 'http://archive.org/embed/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(
-            'http://archive.org/embed/' + video_id, video_id)
-        jwplayer_playlist = self._parse_json(self._search_regex(
-            r"(?s)Play\('[^']+'\s*,\s*(\[.+\])\s*,\s*{.*?}\);",
-            webpage, 'jwplayer playlist'), video_id)
-        info = self._parse_jwplayer_data(
-            {'playlist': jwplayer_playlist}, video_id, base_url=url)
 
-        def get_optional(metadata, field):
-            return metadata.get(field, [None])[0]
+        json_url = url + ('&' if '?' in url else '?') + 'output=json'
+        data = self._download_json(json_url, video_id)
 
-        metadata = self._download_json(
-            'http://archive.org/details/' + video_id, video_id, query={
-                'output': 'json',
-            })['metadata']
-        info.update({
-            'title': get_optional(metadata, 'title') or info.get('title'),
-            'description': clean_html(get_optional(metadata, 'description')),
-        })
-        if info.get('_type') != 'playlist':
-            info.update({
-                'uploader': get_optional(metadata, 'creator'),
-                'upload_date': unified_strdate(get_optional(metadata, 'date')),
-            })
-        return info
+        def get_optional(data_dict, field):
+            return data_dict['metadata'].get(field, [None])[0]
+
+        title = get_optional(data, 'title')
+        description = get_optional(data, 'description')
+        uploader = get_optional(data, 'creator')
+        upload_date = unified_strdate(get_optional(data, 'date'))
+
+        formats = [
+            {
+                'format': fdata['format'],
+                'url': 'http://' + data['server'] + data['dir'] + fn,
+                'file_size': int(fdata['size']),
+            }
+            for fn, fdata in data['files'].items()
+            if 'Video' in fdata['format']]
+
+        self._sort_formats(formats)
+
+        return {
+            '_type': 'video',
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'description': description,
+            'uploader': uploader,
+            'upload_date': upload_date,
+            'thumbnail': data.get('misc', {}).get('image'),
+        }

youtube_dl/extractor/ard.py

@@ -8,19 +8,19 @@ from .generic import GenericIE
 from ..utils import (
     determine_ext,
     ExtractorError,
+    get_element_by_attribute,
     qualities,
     int_or_none,
     parse_duration,
     unified_strdate,
     xpath_text,
-    update_url_query,
 )
 from ..compat import compat_etree_fromstring
 
 
 class ARDMediathekIE(InfoExtractor):
     IE_NAME = 'ARD:mediathek'
-    _VALID_URL = r'^https?://(?:(?:www\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
+    _VALID_URL = r'^https?://(?:(?:www\.)?ardmediathek\.de|mediathek\.daserste\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
 
     _TESTS = [{
         'url': 'http://www.ardmediathek.de/tv/Dokumentation-und-Reportage/Ich-liebe-das-Leben-trotzdem/rbb-Fernsehen/Video?documentId=29582122&bcastId=3822114',
@@ -35,7 +35,6 @@ class ARDMediathekIE(InfoExtractor):
             # m3u8 download
             'skip_download': True,
         },
-        'skip': 'HTTP Error 404: Not Found',
     }, {
         'url': 'http://www.ardmediathek.de/tv/Tatort/Tatort-Scheinwelten-H%C3%B6rfassung-Video/Das-Erste/Video?documentId=29522730&bcastId=602916',
         'md5': 'f4d98b10759ac06c0072bbcd1f0b9e3e',
@@ -46,7 +45,6 @@ class ARDMediathekIE(InfoExtractor):
             'description': 'md5:196392e79876d0ac94c94e8cdb2875f1',
             'duration': 5252,
         },
-        'skip': 'HTTP Error 404: Not Found',
     }, {
         # audio
         'url': 'http://www.ardmediathek.de/tv/WDR-H%C3%B6rspiel-Speicher/Tod-eines-Fu%C3%9Fballers/WDR-3/Audio-Podcast?documentId=28488308&bcastId=23074086',
@@ -58,22 +56,9 @@ class ARDMediathekIE(InfoExtractor):
             'description': 'md5:f6e39f3461f0e1f54bfa48c8875c86ef',
             'duration': 3240,
         },
-        'skip': 'HTTP Error 404: Not Found',
     }, {
         'url': 'http://mediathek.daserste.de/sendungen_a-z/328454_anne-will/22429276_vertrauen-ist-gut-spionieren-ist-besser-geht',
         'only_matching': True,
-    }, {
-        # audio
-        'url': 'http://mediathek.rbb-online.de/radio/Hörspiel/Vor-dem-Fest/kulturradio/Audio?documentId=30796318&topRessort=radio&bcastId=9839158',
-        'md5': '4e8f00631aac0395fee17368ac0e9867',
-        'info_dict': {
-            'id': '30796318',
-            'ext': 'mp3',
-            'title': 'Vor dem Fest',
-            'description': 'md5:c0c1c8048514deaed2a73b3a60eecacb',
-            'duration': 3287,
-        },
-        'skip': 'Video is no longer available',
     }]
 
     def _extract_media_info(self, media_info_url, webpage, video_id):
@@ -129,14 +114,11 @@ class ARDMediathekIE(InfoExtractor):
                         continue
                     if ext == 'f4m':
                         formats.extend(self._extract_f4m_formats(
-                            update_url_query(stream_url, {
-                                'hdcore': '3.1.1',
-                                'plugin': 'aasp-3.1.1.69.124'
-                            }),
-                            video_id, f4m_id='hds', fatal=False))
+                            stream_url + '?hdcore=3.1.1&plugin=aasp-3.1.1.69.124',
+                            video_id, preference=-1, f4m_id='hds', fatal=False))
                     elif ext == 'm3u8':
                         formats.extend(self._extract_m3u8_formats(
-                            stream_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+                            stream_url, video_id, 'mp4', preference=1, m3u8_id='hls', fatal=False))
                     else:
                         if server and server.startswith('rtmp'):
                             f = {
@@ -250,8 +232,7 @@ class ARDIE(InfoExtractor):
             'title': 'Die Story im Ersten: Mission unter falscher Flagge',
             'upload_date': '20140804',
             'thumbnail': 're:^https?://.*\.jpg$',
-        },
-        'skip': 'HTTP Error 404: Not Found',
+        }
     }
 
     def _real_extract(self, url):
@@ -293,3 +274,41 @@ class ARDIE(InfoExtractor):
             'upload_date': upload_date,
             'thumbnail': thumbnail,
         }
+
+
+class SportschauIE(ARDMediathekIE):
+    IE_NAME = 'Sportschau'
+    _VALID_URL = r'(?P<baseurl>https?://(?:www\.)?sportschau\.de/(?:[^/]+/)+video(?P<id>[^/#?]+))\.html'
+    _TESTS = [{
+        'url': 'http://www.sportschau.de/tourdefrance/videoseppeltkokainhatnichtsmitklassischemdopingzutun100.html',
+        'info_dict': {
+            'id': 'seppeltkokainhatnichtsmitklassischemdopingzutun100',
+            'ext': 'mp4',
+            'title': 'Seppelt: "Kokain hat nichts mit klassischem Doping zu tun"',
+            'thumbnail': 're:^https?://.*\.jpg$',
+            'description': 'Der ARD-Doping Experte Hajo Seppelt gibt seine Einschätzung zum ersten Dopingfall der diesjährigen Tour de France um den Italiener Luca Paolini ab.',
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        },
+    }]
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
+        base_url = mobj.group('baseurl')
+
+        webpage = self._download_webpage(url, video_id)
+        title = get_element_by_attribute('class', 'headline', webpage)
+        description = self._html_search_meta('description', webpage, 'description')
+
+        info = self._extract_media_info(
+            base_url + '-mc_defaultQuality-h.json', webpage, video_id)
+
+        info.update({
+            'title': title,
+            'description': description,
+        })
+
+        return info

youtube_dl/extractor/arkena.py

@@ -1,115 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import (
-    determine_ext,
-    float_or_none,
-    int_or_none,
-    mimetype2ext,
-    parse_iso8601,
-    strip_jsonp,
-)
-
-
-class ArkenaIE(InfoExtractor):
-    _VALID_URL = r'https?://play\.arkena\.com/(?:config|embed)/avp/v\d/player/media/(?P<id>[^/]+)/[^/]+/(?P<account_id>\d+)'
-    _TESTS = [{
-        'url': 'https://play.arkena.com/embed/avp/v2/player/media/b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe/1/129411',
-        'md5': 'b96f2f71b359a8ecd05ce4e1daa72365',
-        'info_dict': {
-            'id': 'b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe',
-            'ext': 'mp4',
-            'title': 'Big Buck Bunny',
-            'description': 'Royalty free test video',
-            'timestamp': 1432816365,
-            'upload_date': '20150528',
-            'is_live': False,
-        },
-    }, {
-        'url': 'https://play.arkena.com/config/avp/v2/player/media/b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe/1/129411/?callbackMethod=jQuery1111023664739129262213_1469227693893',
-        'only_matching': True,
-    }, {
-        'url': 'http://play.arkena.com/config/avp/v1/player/media/327336/darkmatter/131064/?callbackMethod=jQuery1111002221189684892677_1469227595972',
-        'only_matching': True,
-    }, {
-        'url': 'http://play.arkena.com/embed/avp/v1/player/media/327336/darkmatter/131064/',
-        'only_matching': True,
-    }]
-
-    @staticmethod
-    def _extract_url(webpage):
-        # See https://support.arkena.com/display/PLAY/Ways+to+embed+your+video
-        mobj = re.search(
-            r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//play\.arkena\.com/embed/avp/.+?)\1',
-            webpage)
-        if mobj:
-            return mobj.group('url')
-
-    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
-        account_id = mobj.group('account_id')
-
-        playlist = self._download_json(
-            'https://play.arkena.com/config/avp/v2/player/media/%s/0/%s/?callbackMethod=_'
-            % (video_id, account_id),
-            video_id, transform_source=strip_jsonp)['Playlist'][0]
-
-        media_info = playlist['MediaInfo']
-        title = media_info['Title']
-        media_files = playlist['MediaFiles']
-
-        is_live = False
-        formats = []
-        for kind_case, kind_formats in media_files.items():
-            kind = kind_case.lower()
-            for f in kind_formats:
-                f_url = f.get('Url')
-                if not f_url:
-                    continue
-                is_live = f.get('Live') == 'true'
-                exts = (mimetype2ext(f.get('Type')), determine_ext(f_url, None))
-                if kind == 'm3u8' or 'm3u8' in exts:
-                    formats.extend(self._extract_m3u8_formats(
-                        f_url, video_id, 'mp4',
-                        entry_protocol='m3u8' if is_live else 'm3u8_native',
-                        m3u8_id=kind, fatal=False, live=is_live))
-                elif kind == 'flash' or 'f4m' in exts:
-                    formats.extend(self._extract_f4m_formats(
-                        f_url, video_id, f4m_id=kind, fatal=False))
-                elif kind == 'dash' or 'mpd' in exts:
-                    formats.extend(self._extract_mpd_formats(
-                        f_url, video_id, mpd_id=kind, fatal=False))
-                elif kind == 'silverlight':
-                    # TODO: process when ism is supported (see
-                    # https://github.com/rg3/youtube-dl/issues/8118)
-                    continue
-                else:
-                    tbr = float_or_none(f.get('Bitrate'), 1000)
-                    formats.append({
-                        'url': f_url,
-                        'format_id': '%s-%d' % (kind, tbr) if tbr else kind,
-                        'tbr': tbr,
-                    })
-        self._sort_formats(formats)
-
-        description = media_info.get('Description')
-        video_id = media_info.get('VideoId') or video_id
-        timestamp = parse_iso8601(media_info.get('PublishDate'))
-        thumbnails = [{
-            'url': thumbnail['Url'],
-            'width': int_or_none(thumbnail.get('Size')),
-        } for thumbnail in (media_info.get('Poster') or []) if thumbnail.get('Url')]
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': description,
-            'timestamp': timestamp,
-            'is_live': is_live,
-            'thumbnails': thumbnails,
-            'formats': formats,
-        }

youtube_dl/extractor/arte.py

@@ -180,14 +180,11 @@ class ArteTVBaseIE(InfoExtractor):
 
 class ArteTVPlus7IE(ArteTVBaseIE):
     IE_NAME = 'arte.tv:+7'
-    _VALID_URL = r'https?://(?:(?:www|sites)\.)?arte\.tv/[^/]+/(?P<lang>fr|de|en|es)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?arte\.tv/guide/(?P<lang>fr|de|en|es)/(?:(?:sendungen|emissions|embed)/)?(?P<id>[^/]+)/(?P<name>[^/?#&]+)'
 
     _TESTS = [{
         'url': 'http://www.arte.tv/guide/de/sendungen/XEN/xenius/?vid=055918-015_PLUS7-D',
         'only_matching': True,
-    }, {
-        'url': 'http://sites.arte.tv/karambolage/de/video/karambolage-22',
-        'only_matching': True,
     }]
 
     @classmethod
@@ -243,10 +240,10 @@ class ArteTVPlus7IE(ArteTVBaseIE):
             return self._extract_from_json_url(json_url, video_id, lang, title=title)
         # Different kind of embed URL (e.g.
         # http://www.arte.tv/magazine/trepalium/fr/episode-0406-replay-trepalium)
-        entries = [
-            self.url_result(url)
-            for _, url in re.findall(r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1', webpage)]
-        return self.playlist_result(entries)
+        embed_url = self._search_regex(
+            r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1',
+            webpage, 'embed url', group='url')
+        return self.url_result(embed_url)
 
 
 # It also uses the arte_vp_url url from the webpage to extract the information
@@ -255,17 +252,22 @@ class ArteTVCreativeIE(ArteTVPlus7IE):
     _VALID_URL = r'https?://creative\.arte\.tv/(?P<lang>fr|de|en|es)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
 
     _TESTS = [{
-        'url': 'http://creative.arte.tv/fr/episode/osmosis-episode-1',
+        'url': 'http://creative.arte.tv/de/magazin/agentur-amateur-corporate-design',
         'info_dict': {
-            'id': '057405-001-A',
+            'id': '72176',
             'ext': 'mp4',
-            'title': 'OSMOSIS - N\'AYEZ PLUS PEUR D\'AIMER (1)',
-            'upload_date': '20150716',
+            'title': 'Folge 2 - Corporate Design',
+            'upload_date': '20131004',
         },
     }, {
         'url': 'http://creative.arte.tv/fr/Monty-Python-Reunion',
-        'playlist_count': 11,
-        'add_ie': ['Youtube'],
+        'info_dict': {
+            'id': '160676',
+            'ext': 'mp4',
+            'title': 'Monty Python live (mostly)',
+            'description': 'Événement ! Quarante-cinq ans après leurs premiers succès, les légendaires Monty Python remontent sur scène.\n',
+            'upload_date': '20140805',
+        }
     }, {
         'url': 'http://creative.arte.tv/de/episode/agentur-amateur-4-der-erste-kunde',
         'only_matching': True,
@@ -347,13 +349,14 @@ class ArteTVCinemaIE(ArteTVPlus7IE):
     _VALID_URL = r'https?://cinema\.arte\.tv/(?P<lang>fr|de|en|es)/(?P<id>.+)'
 
     _TESTS = [{
-        'url': 'http://cinema.arte.tv/fr/article/les-ailes-du-desir-de-julia-reck',
-        'md5': 'a5b9dd5575a11d93daf0e3f404f45438',
+        'url': 'http://cinema.arte.tv/de/node/38291',
+        'md5': '6b275511a5107c60bacbeeda368c3aa1',
         'info_dict': {
-            'id': '062494-000-A',
+            'id': '055876-000_PWA12025-D',
             'ext': 'mp4',
-            'title': 'Film lauréat du concours web - "Les ailes du désir" de Julia Reck',
-            'upload_date': '20150807',
+            'title': 'Tod auf dem Nil',
+            'upload_date': '20160122',
+            'description': 'md5:7f749bbb77d800ef2be11d54529b96bc',
         },
     }]
 
@@ -419,7 +422,6 @@ class ArteTVPlaylistIE(ArteTVBaseIE):
         'info_dict': {
             'id': 'PL-013263',
             'title': 'Areva & Uramin',
-            'description': 'md5:a1dc0312ce357c262259139cfd48c9bf',
         },
         'playlist_mincount': 6,
     }, {

youtube_dl/extractor/audiomack.py

@@ -6,7 +6,6 @@ import time
 
 from .common import InfoExtractor
 from .soundcloud import SoundcloudIE
-from ..compat import compat_str
 from ..utils import (
     ExtractorError,
     url_basename,
@@ -137,7 +136,7 @@ class AudiomackAlbumIE(InfoExtractor):
                         result[resultkey] = api_response[apikey]
                 song_id = url_basename(api_response['url']).rpartition('.')[0]
                 result['entries'].append({
-                    'id': compat_str(api_response.get('id', song_id)),
+                    'id': api_response.get('id', song_id),
                     'uploader': api_response.get('artist'),
                     'title': api_response.get('title', song_id),
                     'url': api_response['url'],

youtube_dl/extractor/azubu.py

@@ -46,7 +46,6 @@ class AzubuIE(InfoExtractor):
                 'uploader_id': 272749,
                 'view_count': int,
             },
-            'skip': 'Channel offline',
         },
     ]
 
@@ -57,26 +56,22 @@ class AzubuIE(InfoExtractor):
             'http://www.azubu.tv/api/video/%s' % video_id, video_id)['data']
 
         title = data['title'].strip()
-        description = data.get('description')
-        thumbnail = data.get('thumbnail')
-        view_count = data.get('view_count')
-        user = data.get('user', {})
-        uploader = user.get('username')
-        uploader_id = user.get('id')
+        description = data['description']
+        thumbnail = data['thumbnail']
+        view_count = data['view_count']
+        uploader = data['user']['username']
+        uploader_id = data['user']['id']
 
         stream_params = json.loads(data['stream_params'])
 
-        timestamp = float_or_none(stream_params.get('creationDate'), 1000)
-        duration = float_or_none(stream_params.get('length'), 1000)
+        timestamp = float_or_none(stream_params['creationDate'], 1000)
+        duration = float_or_none(stream_params['length'], 1000)
 
         renditions = stream_params.get('renditions') or []
         video = stream_params.get('FLVFullLength') or stream_params.get('videoFullLength')
         if video:
             renditions.append(video)
 
-        if not renditions and not user.get('channel', {}).get('is_live', True):
-            raise ExtractorError('%s said: channel is offline.' % self.IE_NAME, expected=True)
-
         formats = [{
             'url': fmt['url'],
             'width': fmt['frameWidth'],

youtube_dl/extractor/bandcamp.py

@@ -162,15 +162,6 @@ class BandcampAlbumIE(InfoExtractor):
             'uploader_id': 'dotscale',
         },
         'playlist_mincount': 7,
-    }, {
-        # with escaped quote in title
-        'url': 'https://jstrecords.bandcamp.com/album/entropy-ep',
-        'info_dict': {
-            'title': '"Entropy" EP',
-            'uploader_id': 'jstrecords',
-            'id': 'entropy-ep',
-        },
-        'playlist_mincount': 3,
     }]
 
     def _real_extract(self, url):
@@ -185,11 +176,8 @@ class BandcampAlbumIE(InfoExtractor):
         entries = [
             self.url_result(compat_urlparse.urljoin(url, t_path), ie=BandcampIE.ie_key())
             for t_path in tracks_paths]
-        title = self._html_search_regex(
-            r'album_title\s*:\s*"((?:\\.|[^"\\])+?)"',
-            webpage, 'title', fatal=False)
-        if title:
-            title = title.replace(r'\"', '"')
+        title = self._search_regex(
+            r'album_title\s*:\s*"(.*?)"', webpage, 'title', fatal=False)
         return {
             '_type': 'playlist',
             'uploader_id': uploader_id,

youtube_dl/extractor/bbc.py

@@ -2,23 +2,19 @@
 from __future__ import unicode_literals
 
 import re
-import itertools
 
 from .common import InfoExtractor
 from ..utils import (
-    dict_get,
     ExtractorError,
     float_or_none,
     int_or_none,
     parse_duration,
     parse_iso8601,
-    try_get,
     unescapeHTML,
 )
 from ..compat import (
     compat_etree_fromstring,
     compat_HTTPError,
-    compat_urlparse,
 )
 
 
@@ -35,7 +31,7 @@ class BBCCoUkIE(InfoExtractor):
                             music/clips[/#]|
                             radio/player/
                         )
-                        (?P<id>%s)(?!/(?:episodes|broadcasts|clips))
+                        (?P<id>%s)
                     ''' % _ID_REGEX
 
     _MEDIASELECTOR_URLS = [
@@ -196,7 +192,6 @@ class BBCCoUkIE(InfoExtractor):
                 # rtmp download
                 'skip_download': True,
             },
-            'skip': 'Now it\'s really geo-restricted',
         }, {
             # compact player (https://github.com/rg3/youtube-dl/issues/8147)
             'url': 'http://www.bbc.co.uk/programmes/p028bfkf/player',
@@ -233,6 +228,51 @@ class BBCCoUkIE(InfoExtractor):
         asx = self._download_xml(connection.get('href'), programme_id, 'Downloading ASX playlist')
         return [ref.get('href') for ref in asx.findall('./Entry/ref')]
 
+    def _extract_connection(self, connection, programme_id):
+        formats = []
+        kind = connection.get('kind')
+        protocol = connection.get('protocol')
+        supplier = connection.get('supplier')
+        if protocol == 'http':
+            href = connection.get('href')
+            transfer_format = connection.get('transferFormat')
+            # ASX playlist
+            if supplier == 'asx':
+                for i, ref in enumerate(self._extract_asx_playlist(connection, programme_id)):
+                    formats.append({
+                        'url': ref,
+                        'format_id': 'ref%s_%s' % (i, supplier),
+                    })
+            # Skip DASH until supported
+            elif transfer_format == 'dash':
+                pass
+            elif transfer_format == 'hls':
+                formats.extend(self._extract_m3u8_formats(
+                    href, programme_id, ext='mp4', entry_protocol='m3u8_native',
+                    m3u8_id=supplier, fatal=False))
+            # Direct link
+            else:
+                formats.append({
+                    'url': href,
+                    'format_id': supplier or kind or protocol,
+                })
+        elif protocol == 'rtmp':
+            application = connection.get('application', 'ondemand')
+            auth_string = connection.get('authString')
+            identifier = connection.get('identifier')
+            server = connection.get('server')
+            formats.append({
+                'url': '%s://%s/%s?%s' % (protocol, server, application, auth_string),
+                'play_path': identifier,
+                'app': '%s?%s' % (application, auth_string),
+                'page_url': 'http://www.bbc.co.uk',
+                'player_url': 'http://www.bbc.co.uk/emp/releases/iplayer/revisions/617463_618125_4/617463_618125_4_emp.swf',
+                'rtmp_live': False,
+                'ext': 'flv',
+                'format_id': supplier,
+            })
+        return formats
+
     def _extract_items(self, playlist):
         return playlist.findall('./{%s}item' % self._EMP_PLAYLIST_NS)
 
@@ -253,6 +293,46 @@ class BBCCoUkIE(InfoExtractor):
     def _extract_connections(self, media):
         return self._findall_ns(media, './{%s}connection')
 
+    def _extract_video(self, media, programme_id):
+        formats = []
+        vbr = int_or_none(media.get('bitrate'))
+        vcodec = media.get('encoding')
+        service = media.get('service')
+        width = int_or_none(media.get('width'))
+        height = int_or_none(media.get('height'))
+        file_size = int_or_none(media.get('media_file_size'))
+        for connection in self._extract_connections(media):
+            conn_formats = self._extract_connection(connection, programme_id)
+            for format in conn_formats:
+                format.update({
+                    'width': width,
+                    'height': height,
+                    'vbr': vbr,
+                    'vcodec': vcodec,
+                    'filesize': file_size,
+                })
+                if service:
+                    format['format_id'] = '%s_%s' % (service, format['format_id'])
+            formats.extend(conn_formats)
+        return formats
+
+    def _extract_audio(self, media, programme_id):
+        formats = []
+        abr = int_or_none(media.get('bitrate'))
+        acodec = media.get('encoding')
+        service = media.get('service')
+        for connection in self._extract_connections(media):
+            conn_formats = self._extract_connection(connection, programme_id)
+            for format in conn_formats:
+                format.update({
+                    'format_id': '%s_%s' % (service, format['format_id']),
+                    'abr': abr,
+                    'acodec': acodec,
+                    'vcodec': 'none',
+                })
+            formats.extend(conn_formats)
+        return formats
+
     def _get_subtitles(self, media, programme_id):
         subtitles = {}
         for connection in self._extract_connections(media):
@@ -298,87 +378,13 @@ class BBCCoUkIE(InfoExtractor):
     def _process_media_selector(self, media_selection, programme_id):
         formats = []
         subtitles = None
-        urls = []
 
         for media in self._extract_medias(media_selection):
             kind = media.get('kind')
-            if kind in ('video', 'audio'):
-                bitrate = int_or_none(media.get('bitrate'))
-                encoding = media.get('encoding')
-                service = media.get('service')
-                width = int_or_none(media.get('width'))
-                height = int_or_none(media.get('height'))
-                file_size = int_or_none(media.get('media_file_size'))
-                for connection in self._extract_connections(media):
-                    href = connection.get('href')
-                    if href in urls:
-                        continue
-                    if href:
-                        urls.append(href)
-                    conn_kind = connection.get('kind')
-                    protocol = connection.get('protocol')
-                    supplier = connection.get('supplier')
-                    transfer_format = connection.get('transferFormat')
-                    format_id = supplier or conn_kind or protocol
-                    if service:
-                        format_id = '%s_%s' % (service, format_id)
-                    # ASX playlist
-                    if supplier == 'asx':
-                        for i, ref in enumerate(self._extract_asx_playlist(connection, programme_id)):
-                            formats.append({
-                                'url': ref,
-                                'format_id': 'ref%s_%s' % (i, format_id),
-                            })
-                    elif transfer_format == 'dash':
-                        formats.extend(self._extract_mpd_formats(
-                            href, programme_id, mpd_id=format_id, fatal=False))
-                    elif transfer_format == 'hls':
-                        formats.extend(self._extract_m3u8_formats(
-                            href, programme_id, ext='mp4', entry_protocol='m3u8_native',
-                            m3u8_id=format_id, fatal=False))
-                    elif transfer_format == 'hds':
-                        formats.extend(self._extract_f4m_formats(
-                            href, programme_id, f4m_id=format_id, fatal=False))
-                    else:
-                        if not service and not supplier and bitrate:
-                            format_id += '-%d' % bitrate
-                        fmt = {
-                            'format_id': format_id,
-                            'filesize': file_size,
-                        }
-                        if kind == 'video':
-                            fmt.update({
-                                'width': width,
-                                'height': height,
-                                'vbr': bitrate,
-                                'vcodec': encoding,
-                            })
-                        else:
-                            fmt.update({
-                                'abr': bitrate,
-                                'acodec': encoding,
-                                'vcodec': 'none',
-                            })
-                        if protocol == 'http':
-                            # Direct link
-                            fmt.update({
-                                'url': href,
-                            })
-                        elif protocol == 'rtmp':
-                            application = connection.get('application', 'ondemand')
-                            auth_string = connection.get('authString')
-                            identifier = connection.get('identifier')
-                            server = connection.get('server')
-                            fmt.update({
-                                'url': '%s://%s/%s?%s' % (protocol, server, application, auth_string),
-                                'play_path': identifier,
-                                'app': '%s?%s' % (application, auth_string),
-                                'page_url': 'http://www.bbc.co.uk',
-                                'player_url': 'http://www.bbc.co.uk/emp/releases/iplayer/revisions/617463_618125_4/617463_618125_4_emp.swf',
-                                'rtmp_live': False,
-                                'ext': 'flv',
-                            })
-                        formats.append(fmt)
+            if kind == 'audio':
+                formats.extend(self._extract_audio(media, programme_id))
+            elif kind == 'video':
+                formats.extend(self._extract_video(media, programme_id))
             elif kind == 'captions':
                 subtitles = self.extract_subtitles(media, programme_id)
         return formats, subtitles
@@ -583,7 +589,6 @@ class BBCIE(BBCCoUkIE):
             'id': '150615_telabyad_kentin_cogu',
             'ext': 'mp4',
             'title': "YPG: Tel Abyad'ın tamamı kontrolümüzde",
-            'description': 'md5:33a4805a855c9baf7115fcbde57e7025',
             'timestamp': 1434397334,
             'upload_date': '20150615',
         },
@@ -597,7 +602,6 @@ class BBCIE(BBCCoUkIE):
             'id': '150619_video_honduras_militares_hospitales_corrupcion_aw',
             'ext': 'mp4',
             'title': 'Honduras militariza sus hospitales por nuevo escándalo de corrupción',
-            'description': 'md5:1525f17448c4ee262b64b8f0c9ce66c8',
             'timestamp': 1434713142,
             'upload_date': '20150619',
         },
@@ -647,23 +651,6 @@ class BBCIE(BBCCoUkIE):
             # rtmp download
             'skip_download': True,
         }
-    }, {
-        # single video embedded with Morph
-        'url': 'http://www.bbc.co.uk/sport/live/olympics/36895975',
-        'info_dict': {
-            'id': 'p041vhd0',
-            'ext': 'mp4',
-            'title': "Nigeria v Japan - Men's First Round",
-            'description': 'Live coverage of the first round from Group B at the Amazonia Arena.',
-            'duration': 7980,
-            'uploader': 'BBC Sport',
-            'uploader_id': 'bbc_sport',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'skip': 'Georestricted to UK',
     }, {
         # single video with playlist.sxml URL in playlist param
         'url': 'http://www.bbc.com/sport/0/football/33653409',
@@ -711,9 +698,7 @@ class BBCIE(BBCCoUkIE):
 
     @classmethod
     def suitable(cls, url):
-        EXCLUDE_IE = (BBCCoUkIE, BBCCoUkArticleIE, BBCCoUkIPlayerPlaylistIE, BBCCoUkPlaylistIE)
-        return (False if any(ie.suitable(url) for ie in EXCLUDE_IE)
-                else super(BBCIE, cls).suitable(url))
+        return False if BBCCoUkIE.suitable(url) or BBCCoUkArticleIE.suitable(url) else super(BBCIE, cls).suitable(url)
 
     def _extract_from_media_meta(self, media_meta, video_id):
         # Direct links to media in media metadata (e.g.
@@ -761,7 +746,7 @@ class BBCIE(BBCCoUkIE):
 
         webpage = self._download_webpage(url, playlist_id)
 
-        json_ld_info = self._search_json_ld(webpage, playlist_id, default={})
+        json_ld_info = self._search_json_ld(webpage, playlist_id, default=None)
         timestamp = json_ld_info.get('timestamp')
 
         playlist_title = json_ld_info.get('title')
@@ -830,29 +815,8 @@ class BBCIE(BBCCoUkIE):
                         # http://www.bbc.com/turkce/multimedya/2015/10/151010_vid_ankara_patlama_ani)
                         playlist = data_playable.get('otherSettings', {}).get('playlist', {})
                         if playlist:
-                            entry = None
-                            for key in ('streaming', 'progressiveDownload'):
-                                playlist_url = playlist.get('%sUrl' % key)
-                                if not playlist_url:
-                                    continue
-                                try:
-                                    info = self._extract_from_playlist_sxml(
-                                        playlist_url, playlist_id, timestamp)
-                                    if not entry:
-                                        entry = info
-                                    else:
-                                        entry['title'] = info['title']
-                                        entry['formats'].extend(info['formats'])
-                                except Exception as e:
-                                    # Some playlist URL may fail with 500, at the same time
-                                    # the other one may work fine (e.g.
-                                    # http://www.bbc.com/turkce/haberler/2015/06/150615_telabyad_kentin_cogu)
-                                    if isinstance(e.cause, compat_HTTPError) and e.cause.code == 500:
-                                        continue
-                                    raise
-                            if entry:
-                                self._sort_formats(entry['formats'])
-                                entries.append(entry)
+                            entries.append(self._extract_from_playlist_sxml(
+                                playlist.get('progressiveDownloadUrl'), playlist_id, timestamp))
 
         if entries:
             return self.playlist_result(entries, playlist_id, playlist_title, playlist_description)
@@ -885,50 +849,6 @@ class BBCIE(BBCCoUkIE):
                 'subtitles': subtitles,
             }
 
-        # Morph based embed (e.g. http://www.bbc.co.uk/sport/live/olympics/36895975)
-        # There are several setPayload calls may be present but the video
-        # seems to be always related to the first one
-        morph_payload = self._parse_json(
-            self._search_regex(
-                r'Morph\.setPayload\([^,]+,\s*({.+?})\);',
-                webpage, 'morph payload', default='{}'),
-            playlist_id, fatal=False)
-        if morph_payload:
-            components = try_get(morph_payload, lambda x: x['body']['components'], list) or []
-            for component in components:
-                if not isinstance(component, dict):
-                    continue
-                lead_media = try_get(component, lambda x: x['props']['leadMedia'], dict)
-                if not lead_media:
-                    continue
-                identifiers = lead_media.get('identifiers')
-                if not identifiers or not isinstance(identifiers, dict):
-                    continue
-                programme_id = identifiers.get('vpid') or identifiers.get('playablePid')
-                if not programme_id:
-                    continue
-                title = lead_media.get('title') or self._og_search_title(webpage)
-                formats, subtitles = self._download_media_selector(programme_id)
-                self._sort_formats(formats)
-                description = lead_media.get('summary')
-                uploader = lead_media.get('masterBrand')
-                uploader_id = lead_media.get('mid')
-                duration = None
-                duration_d = lead_media.get('duration')
-                if isinstance(duration_d, dict):
-                    duration = parse_duration(dict_get(
-                        duration_d, ('rawDuration', 'formattedDuration', 'spokenDuration')))
-                return {
-                    'id': programme_id,
-                    'title': title,
-                    'description': description,
-                    'duration': duration,
-                    'uploader': uploader,
-                    'uploader_id': uploader_id,
-                    'formats': formats,
-                    'subtitles': subtitles,
-                }
-
         def extract_all(pattern):
             return list(filter(None, map(
                 lambda s: self._parse_json(s, playlist_id, fatal=False),
@@ -946,7 +866,7 @@ class BBCIE(BBCCoUkIE):
             r'setPlaylist\("(%s)"\)' % EMBED_URL, webpage))
         if entries:
             return self.playlist_result(
-                [self.url_result(entry_, 'BBCCoUk') for entry_ in entries],
+                [self.url_result(entry, 'BBCCoUk') for entry in entries],
                 playlist_id, playlist_title, playlist_description)
 
         # Multiple video article (e.g. http://www.bbc.com/news/world-europe-32668511)
@@ -1055,116 +975,3 @@ class BBCCoUkArticleIE(InfoExtractor):
             r'<div[^>]+typeof="Clip"[^>]+resource="([^"]+)"', webpage)]
 
         return self.playlist_result(entries, playlist_id, title, description)
-
-
-class BBCCoUkPlaylistBaseIE(InfoExtractor):
-    def _entries(self, webpage, url, playlist_id):
-        single_page = 'page' in compat_urlparse.parse_qs(
-            compat_urlparse.urlparse(url).query)
-        for page_num in itertools.count(2):
-            for video_id in re.findall(
-                    self._VIDEO_ID_TEMPLATE % BBCCoUkIE._ID_REGEX, webpage):
-                yield self.url_result(
-                    self._URL_TEMPLATE % video_id, BBCCoUkIE.ie_key())
-            if single_page:
-                return
-            next_page = self._search_regex(
-                r'<li[^>]+class=(["\'])pagination_+next\1[^>]*><a[^>]+href=(["\'])(?P<url>(?:(?!\2).)+)\2',
-                webpage, 'next page url', default=None, group='url')
-            if not next_page:
-                break
-            webpage = self._download_webpage(
-                compat_urlparse.urljoin(url, next_page), playlist_id,
-                'Downloading page %d' % page_num, page_num)
-
-    def _real_extract(self, url):
-        playlist_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, playlist_id)
-
-        title, description = self._extract_title_and_description(webpage)
-
-        return self.playlist_result(
-            self._entries(webpage, url, playlist_id),
-            playlist_id, title, description)
-
-
-class BBCCoUkIPlayerPlaylistIE(BBCCoUkPlaylistBaseIE):
-    IE_NAME = 'bbc.co.uk:iplayer:playlist'
-    _VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/iplayer/(?:episodes|group)/(?P<id>%s)' % BBCCoUkIE._ID_REGEX
-    _URL_TEMPLATE = 'http://www.bbc.co.uk/iplayer/episode/%s'
-    _VIDEO_ID_TEMPLATE = r'data-ip-id=["\'](%s)'
-    _TESTS = [{
-        'url': 'http://www.bbc.co.uk/iplayer/episodes/b05rcz9v',
-        'info_dict': {
-            'id': 'b05rcz9v',
-            'title': 'The Disappearance',
-            'description': 'French thriller serial about a missing teenager.',
-        },
-        'playlist_mincount': 6,
-        'skip': 'This programme is not currently available on BBC iPlayer',
-    }, {
-        # Available for over a year unlike 30 days for most other programmes
-        'url': 'http://www.bbc.co.uk/iplayer/group/p02tcc32',
-        'info_dict': {
-            'id': 'p02tcc32',
-            'title': 'Bohemian Icons',
-            'description': 'md5:683e901041b2fe9ba596f2ab04c4dbe7',
-        },
-        'playlist_mincount': 10,
-    }]
-
-    def _extract_title_and_description(self, webpage):
-        title = self._search_regex(r'<h1>([^<]+)</h1>', webpage, 'title', fatal=False)
-        description = self._search_regex(
-            r'<p[^>]+class=(["\'])subtitle\1[^>]*>(?P<value>[^<]+)</p>',
-            webpage, 'description', fatal=False, group='value')
-        return title, description
-
-
-class BBCCoUkPlaylistIE(BBCCoUkPlaylistBaseIE):
-    IE_NAME = 'bbc.co.uk:playlist'
-    _VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/programmes/(?P<id>%s)/(?:episodes|broadcasts|clips)' % BBCCoUkIE._ID_REGEX
-    _URL_TEMPLATE = 'http://www.bbc.co.uk/programmes/%s'
-    _VIDEO_ID_TEMPLATE = r'data-pid=["\'](%s)'
-    _TESTS = [{
-        'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/clips',
-        'info_dict': {
-            'id': 'b05rcz9v',
-            'title': 'The Disappearance - Clips - BBC Four',
-            'description': 'French thriller serial about a missing teenager.',
-        },
-        'playlist_mincount': 7,
-    }, {
-        # multipage playlist, explicit page
-        'url': 'http://www.bbc.co.uk/programmes/b00mfl7n/clips?page=1',
-        'info_dict': {
-            'id': 'b00mfl7n',
-            'title': 'Frozen Planet - Clips - BBC One',
-            'description': 'md5:65dcbf591ae628dafe32aa6c4a4a0d8c',
-        },
-        'playlist_mincount': 24,
-    }, {
-        # multipage playlist, all pages
-        'url': 'http://www.bbc.co.uk/programmes/b00mfl7n/clips',
-        'info_dict': {
-            'id': 'b00mfl7n',
-            'title': 'Frozen Planet - Clips - BBC One',
-            'description': 'md5:65dcbf591ae628dafe32aa6c4a4a0d8c',
-        },
-        'playlist_mincount': 142,
-    }, {
-        'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/broadcasts/2016/06',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/clips',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.bbc.co.uk/programmes/b055jkys/episodes/player',
-        'only_matching': True,
-    }]
-
-    def _extract_title_and_description(self, webpage):
-        title = self._og_search_title(webpage, fatal=False)
-        description = self._og_search_description(webpage)
-        return title, description

youtube_dl/extractor/bet.py

@@ -1,26 +1,31 @@
 from __future__ import unicode_literals
 
-from .mtv import MTVServicesInfoExtractor
-from ..utils import unified_strdate
+from .common import InfoExtractor
+from ..compat import compat_urllib_parse_unquote
+from ..utils import (
+    xpath_text,
+    xpath_with_ns,
+    int_or_none,
+    parse_iso8601,
+)
 
 
-class BetIE(MTVServicesInfoExtractor):
+class BetIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?bet\.com/(?:[^/]+/)+(?P<id>.+?)\.html'
     _TESTS = [
         {
             'url': 'http://www.bet.com/news/politics/2014/12/08/in-bet-exclusive-obama-talks-race-and-racism.html',
             'info_dict': {
-                'id': '07e96bd3-8850-3051-b856-271b457f0ab8',
+                'id': 'news/national/2014/a-conversation-with-president-obama',
                 'display_id': 'in-bet-exclusive-obama-talks-race-and-racism',
                 'ext': 'flv',
                 'title': 'A Conversation With President Obama',
-                'description': 'President Obama urges persistence in confronting racism and bias.',
+                'description': 'md5:699d0652a350cf3e491cd15cc745b5da',
                 'duration': 1534,
+                'timestamp': 1418075340,
                 'upload_date': '20141208',
+                'uploader': 'admin',
                 'thumbnail': 're:(?i)^https?://.*\.jpg$',
-                'subtitles': {
-                    'en': 'mincount:2',
-                }
             },
             'params': {
                 # rtmp download
@@ -30,17 +35,16 @@ class BetIE(MTVServicesInfoExtractor):
         {
             'url': 'http://www.bet.com/video/news/national/2014/justice-for-ferguson-a-community-reacts.html',
             'info_dict': {
-                'id': '9f516bf1-7543-39c4-8076-dd441b459ba9',
+                'id': 'news/national/2014/justice-for-ferguson-a-community-reacts',
                 'display_id': 'justice-for-ferguson-a-community-reacts',
                 'ext': 'flv',
                 'title': 'Justice for Ferguson: A Community Reacts',
                 'description': 'A BET News special.',
                 'duration': 1696,
+                'timestamp': 1416942360,
                 'upload_date': '20141125',
+                'uploader': 'admin',
                 'thumbnail': 're:(?i)^https?://.*\.jpg$',
-                'subtitles': {
-                    'en': 'mincount:2',
-                }
             },
             'params': {
                 # rtmp download
@@ -49,32 +53,57 @@ class BetIE(MTVServicesInfoExtractor):
         }
     ]
 
-    _FEED_URL = "http://feeds.mtvnservices.com/od/feed/bet-mrss-player"
-
-    def _get_feed_query(self, uri):
-        return {
-            'uuid': uri,
-        }
-
-    def _extract_mgid(self, webpage):
-        return self._search_regex(r'data-uri="([^"]+)', webpage, 'mgid')
-
     def _real_extract(self, url):
         display_id = self._match_id(url)
-
         webpage = self._download_webpage(url, display_id)
-        mgid = self._extract_mgid(webpage)
-        videos_info = self._get_videos_info(mgid)
 
-        info_dict = videos_info['entries'][0]
+        media_url = compat_urllib_parse_unquote(self._search_regex(
+            [r'mediaURL\s*:\s*"([^"]+)"', r"var\s+mrssMediaUrl\s*=\s*'([^']+)'"],
+            webpage, 'media URL'))
 
-        upload_date = unified_strdate(self._html_search_meta('date', webpage))
-        description = self._html_search_meta('description', webpage)
+        video_id = self._search_regex(
+            r'/video/(.*)/_jcr_content/', media_url, 'video id')
 
-        info_dict.update({
+        mrss = self._download_xml(media_url, display_id)
+
+        item = mrss.find('./channel/item')
+
+        NS_MAP = {
+            'dc': 'http://purl.org/dc/elements/1.1/',
+            'media': 'http://search.yahoo.com/mrss/',
+            'ka': 'http://kickapps.com/karss',
+        }
+
+        title = xpath_text(item, './title', 'title')
+        description = xpath_text(
+            item, './description', 'description', fatal=False)
+
+        timestamp = parse_iso8601(xpath_text(
+            item, xpath_with_ns('./dc:date', NS_MAP),
+            'upload date', fatal=False))
+        uploader = xpath_text(
+            item, xpath_with_ns('./dc:creator', NS_MAP),
+            'uploader', fatal=False)
+
+        media_content = item.find(
+            xpath_with_ns('./media:content', NS_MAP))
+        duration = int_or_none(media_content.get('duration'))
+        smil_url = media_content.get('url')
+
+        thumbnail = media_content.find(
+            xpath_with_ns('./media:thumbnail', NS_MAP)).get('url')
+
+        formats = self._extract_smil_formats(smil_url, display_id)
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
             'display_id': display_id,
+            'title': title,
             'description': description,
-            'upload_date': upload_date,
-        })
-
-        return info_dict
+            'thumbnail': thumbnail,
+            'timestamp': timestamp,
+            'uploader': uploader,
+            'duration': duration,
+            'formats': formats,
+        }

youtube_dl/extractor/bigflix.py

@@ -11,13 +11,22 @@ from ..compat import compat_urllib_parse_unquote
 class BigflixIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?bigflix\.com/.+/(?P<id>[0-9]+)'
     _TESTS = [{
+        'url': 'http://www.bigflix.com/Hindi-movies/Action-movies/Singham-Returns/16537',
+        'md5': 'ec76aa9b1129e2e5b301a474e54fab74',
+        'info_dict': {
+            'id': '16537',
+            'ext': 'mp4',
+            'title': 'Singham Returns',
+            'description': 'md5:3d2ba5815f14911d5cc6a501ae0cf65d',
+        }
+    }, {
         # 2 formats
         'url': 'http://www.bigflix.com/Tamil-movies/Drama-movies/Madarasapatinam/16070',
         'info_dict': {
             'id': '16070',
             'ext': 'mp4',
             'title': 'Madarasapatinam',
-            'description': 'md5:9f0470b26a4ba8e824c823b5d95c2f6b',
+            'description': 'md5:63b9b8ed79189c6f0418c26d9a3452ca',
             'formats': 'mincount:2',
         },
         'params': {

youtube_dl/extractor/bilibili.py

@@ -1,15 +1,22 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import hashlib
+import calendar
+import datetime
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_parse_qs
+from ..compat import (
+    compat_etree_fromstring,
+    compat_str,
+    compat_parse_qs,
+    compat_xml_parse_error,
+)
 from ..utils import (
+    ExtractorError,
     int_or_none,
     float_or_none,
-    unified_timestamp,
+    xpath_text,
 )
 
 
@@ -18,13 +25,13 @@ class BiliBiliIE(InfoExtractor):
 
     _TESTS = [{
         'url': 'http://www.bilibili.tv/video/av1074402/',
-        'md5': '9fa226fe2b8a9a4d5a69b4c6a183417e',
+        'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
         'info_dict': {
-            'id': '1074402',
-            'ext': 'mp4',
+            'id': '1554319',
+            'ext': 'flv',
             'title': '【金坷垃】金泡沫',
             'description': 'md5:ce18c2a2d2193f0df2917d270f2e5923',
-            'duration': 308.315,
+            'duration': 308.067,
             'timestamp': 1398012660,
             'upload_date': '20140420',
             'thumbnail': 're:^https?://.+\.jpg',
@@ -35,105 +42,147 @@ class BiliBiliIE(InfoExtractor):
         'url': 'http://www.bilibili.com/video/av1041170/',
         'info_dict': {
             'id': '1041170',
-            'ext': 'mp4',
             'title': '【BD1080P】刀语【诸神&异域】',
             'description': '这是个神奇的故事~每个人不留弹幕不给走哦~切利哦！~',
-            'duration': 3382.259,
-            'timestamp': 1396530060,
-            'upload_date': '20140403',
-            'thumbnail': 're:^https?://.+\.jpg',
-            'uploader': '枫叶逝去',
-            'uploader_id': '520116',
         },
+        'playlist_count': 9,
     }, {
         'url': 'http://www.bilibili.com/video/av4808130/',
         'info_dict': {
             'id': '4808130',
-            'ext': 'mp4',
             'title': '【长篇】哆啦A梦443【钉铛】',
             'description': '(2016.05.27)来组合客人的脸吧&寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&illust_id=56912929',
-            'duration': 1493.995,
-            'timestamp': 1464564180,
-            'upload_date': '20160529',
-            'thumbnail': 're:^https?://.+\.jpg',
-            'uploader': '喜欢拉面',
-            'uploader_id': '151066',
         },
-    }, {
-        # Missing upload time
-        'url': 'http://www.bilibili.com/video/av1867637/',
-        'info_dict': {
-            'id': '1867637',
-            'ext': 'mp4',
-            'title': '【HDTV】【喜剧】岳父岳母真难当 （2014）【法国票房冠军】',
-            'description': '一个信奉天主教的法国旧式传统资产阶级家庭中有四个女儿。三个女儿却分别找了阿拉伯、犹太、中国丈夫，老夫老妻唯独期盼剩下未嫁的小女儿能找一个信奉天主教的法国白人，结果没想到小女儿找了一位非裔黑人……【这次应该不会跳帧了】',
-            'duration': 5760.0,
-            'uploader': '黑夜为猫',
-            'uploader_id': '610729',
-            'thumbnail': 're:^https?://.+\.jpg',
-        },
-        'params': {
-            # Just to test metadata extraction
-            'skip_download': True,
-        },
-        'expected_warnings': ['upload time'],
+        'playlist': [{
+            'md5': '55cdadedf3254caaa0d5d27cf20a8f9c',
+            'info_dict': {
+                'id': '4808130_part1',
+                'ext': 'flv',
+                'title': '【长篇】哆啦A梦443【钉铛】',
+                'description': '(2016.05.27)来组合客人的脸吧&寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&illust_id=56912929',
+                'timestamp': 1464564180,
+                'upload_date': '20160529',
+                'uploader': '喜欢拉面',
+                'uploader_id': '151066',
+            },
+        }, {
+            'md5': '926f9f67d0c482091872fbd8eca7ea3d',
+            'info_dict': {
+                'id': '4808130_part2',
+                'ext': 'flv',
+                'title': '【长篇】哆啦A梦443【钉铛】',
+                'description': '(2016.05.27)来组合客人的脸吧&寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&illust_id=56912929',
+                'timestamp': 1464564180,
+                'upload_date': '20160529',
+                'uploader': '喜欢拉面',
+                'uploader_id': '151066',
+            },
+        }, {
+            'md5': '4b7b225b968402d7c32348c646f1fd83',
+            'info_dict': {
+                'id': '4808130_part3',
+                'ext': 'flv',
+                'title': '【长篇】哆啦A梦443【钉铛】',
+                'description': '(2016.05.27)来组合客人的脸吧&寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&illust_id=56912929',
+                'timestamp': 1464564180,
+                'upload_date': '20160529',
+                'uploader': '喜欢拉面',
+                'uploader_id': '151066',
+            },
+        }, {
+            'md5': '7b795e214166501e9141139eea236e91',
+            'info_dict': {
+                'id': '4808130_part4',
+                'ext': 'flv',
+                'title': '【长篇】哆啦A梦443【钉铛】',
+                'description': '(2016.05.27)来组合客人的脸吧&寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&illust_id=56912929',
+                'timestamp': 1464564180,
+                'upload_date': '20160529',
+                'uploader': '喜欢拉面',
+                'uploader_id': '151066',
+            },
+        }],
     }]
 
-    _APP_KEY = '6f90a59ac58a4123'
-    _BILIBILI_KEY = '0bfd84cc3940035173f35e6777508326'
+    # BiliBili blocks keys from time to time. The current key is extracted from
+    # the Android client
+    # TODO: find the sign algorithm used in the flash player
+    _APP_KEY = '86385cdc024c0f6c'
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
 
         webpage = self._download_webpage(url, video_id)
 
-        cid = compat_parse_qs(self._search_regex(
+        params = compat_parse_qs(self._search_regex(
             [r'EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
              r'<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
-            webpage, 'player parameters'))['cid'][0]
+            webpage, 'player parameters'))
+        cid = params['cid'][0]
 
-        payload = 'appkey=%s&cid=%s&otype=json&quality=2&type=mp4' % (self._APP_KEY, cid)
-        sign = hashlib.md5((payload + self._BILIBILI_KEY).encode('utf-8')).hexdigest()
+        info_xml_str = self._download_webpage(
+            'http://interface.bilibili.com/v_cdn_play',
+            cid, query={'appkey': self._APP_KEY, 'cid': cid},
+            note='Downloading video info page')
 
-        video_info = self._download_json(
-            'http://interface.bilibili.com/playurl?%s&sign=%s' % (payload, sign),
-            video_id, note='Downloading video info page')
+        err_msg = None
+        durls = None
+        info_xml = None
+        try:
+            info_xml = compat_etree_fromstring(info_xml_str.encode('utf-8'))
+        except compat_xml_parse_error:
+            info_json = self._parse_json(info_xml_str, video_id, fatal=False)
+            err_msg = (info_json or {}).get('error_text')
+        else:
+            err_msg = xpath_text(info_xml, './message')
+
+        if info_xml is not None:
+            durls = info_xml.findall('./durl')
+        if not durls:
+            if err_msg:
+                raise ExtractorError('%s said: %s' % (self.IE_NAME, err_msg), expected=True)
+            else:
+                raise ExtractorError('No videos found!')
 
         entries = []
 
-        for idx, durl in enumerate(video_info['durl']):
+        for durl in durls:
+            size = xpath_text(durl, ['./filesize', './size'])
             formats = [{
-                'url': durl['url'],
-                'filesize': int_or_none(durl['size']),
+                'url': durl.find('./url').text,
+                'filesize': int_or_none(size),
             }]
-            for backup_url in durl['backup_url']:
+            for backup_url in durl.findall('./backup_url/url'):
                 formats.append({
-                    'url': backup_url,
+                    'url': backup_url.text,
                     # backup URLs have lower priorities
-                    'preference': -2 if 'hd.mp4' in backup_url else -3,
+                    'preference': -2 if 'hd.mp4' in backup_url.text else -3,
                 })
 
             self._sort_formats(formats)
 
             entries.append({
-                'id': '%s_part%s' % (video_id, idx),
-                'duration': float_or_none(durl.get('length'), 1000),
+                'id': '%s_part%s' % (cid, xpath_text(durl, './order')),
+                'duration': int_or_none(xpath_text(durl, './length'), 1000),
                 'formats': formats,
             })
 
         title = self._html_search_regex('<h1[^>]+title="([^"]+)">', webpage, 'title')
         description = self._html_search_meta('description', webpage)
-        timestamp = unified_timestamp(self._html_search_regex(
-            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False))
+        datetime_str = self._html_search_regex(
+            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False)
+        if datetime_str:
+            timestamp = calendar.timegm(datetime.datetime.strptime(datetime_str, '%Y-%m-%dT%H:%M').timetuple())
 
         # TODO 'view_count' requires deobfuscating Javascript
         info = {
-            'id': video_id,
+            'id': compat_str(cid),
             'title': title,
             'description': description,
             'timestamp': timestamp,
             'thumbnail': self._html_search_meta('thumbnailUrl', webpage),
-            'duration': float_or_none(video_info.get('timelength'), scale=1000),
+            'duration': float_or_none(xpath_text(info_xml, './timelength'), scale=1000),
         }
 
         uploader_mobj = re.search(

youtube_dl/extractor/biobiochiletv.py

@@ -2,15 +2,11 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..utils import (
-    ExtractorError,
-    remove_end,
-)
-from .rudo import RudoIE
+from ..utils import remove_end
 
 
 class BioBioChileTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:tv|www)\.biobiochile\.cl/(?:notas|noticias)/(?:[^/]+/)+(?P<id>[^/]+)\.shtml'
+    _VALID_URL = r'https?://tv\.biobiochile\.cl/notas/(?:[^/]+/)+(?P<id>[^/]+)\.shtml'
 
     _TESTS = [{
         'url': 'http://tv.biobiochile.cl/notas/2015/10/21/sobre-camaras-y-camarillas-parlamentarias.shtml',
@@ -22,7 +18,6 @@ class BioBioChileTVIE(InfoExtractor):
             'thumbnail': 're:^https?://.*\.jpg$',
             'uploader': 'Fernando Atria',
         },
-        'skip': 'URL expired and redirected to http://www.biobiochile.cl/portada/bbtv/index.html',
     }, {
         # different uploader layout
         'url': 'http://tv.biobiochile.cl/notas/2016/03/18/natalia-valdebenito-repasa-a-diputado-hasbun-paso-a-la-categoria-de-hablar-brutalidades.shtml',
@@ -37,16 +32,6 @@ class BioBioChileTVIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
-        'skip': 'URL expired and redirected to http://www.biobiochile.cl/portada/bbtv/index.html',
-    }, {
-        'url': 'http://www.biobiochile.cl/noticias/bbtv/comentarios-bio-bio/2016/07/08/edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos.shtml',
-        'info_dict': {
-            'id': 'edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos',
-            'ext': 'mp4',
-            'uploader': '(none)',
-            'upload_date': '20160708',
-            'title': 'Edecanes del Congreso: Figuras decorativas que le cuestan muy caro a los chilenos',
-        },
     }, {
         'url': 'http://tv.biobiochile.cl/notas/2015/10/22/ninos-transexuales-de-quien-es-la-decision.shtml',
         'only_matching': True,
@@ -60,22 +45,42 @@ class BioBioChileTVIE(InfoExtractor):
 
         webpage = self._download_webpage(url, video_id)
 
-        rudo_url = RudoIE._extract_url(webpage)
-        if not rudo_url:
-            raise ExtractorError('No videos found')
-
         title = remove_end(self._og_search_title(webpage), ' - BioBioChile TV')
 
+        file_url = self._search_regex(
+            r'loadFWPlayerVideo\([^,]+,\s*(["\'])(?P<url>.+?)\1',
+            webpage, 'file url', group='url')
+
+        base_url = self._search_regex(
+            r'file\s*:\s*(["\'])(?P<url>.+?)\1\s*\+\s*fileURL', webpage,
+            'base url', default='http://unlimited2-cl.digitalproserver.com/bbtv/',
+            group='url')
+
+        formats = self._extract_m3u8_formats(
+            '%s%s/playlist.m3u8' % (base_url, file_url), video_id, 'mp4',
+            entry_protocol='m3u8_native', m3u8_id='hls', fatal=False)
+        f = {
+            'url': '%s%s' % (base_url, file_url),
+            'format_id': 'http',
+            'protocol': 'http',
+            'preference': 1,
+        }
+        if formats:
+            f_copy = formats[-1].copy()
+            f_copy.update(f)
+            f = f_copy
+        formats.append(f)
+        self._sort_formats(formats)
+
         thumbnail = self._og_search_thumbnail(webpage)
         uploader = self._html_search_regex(
-            r'<a[^>]+href=["\']https?://(?:busca|www)\.biobiochile\.cl/(?:lista/)?(?:author|autor)[^>]+>(.+?)</a>',
+            r'<a[^>]+href=["\']https?://busca\.biobiochile\.cl/author[^>]+>(.+?)</a>',
             webpage, 'uploader', fatal=False)
 
         return {
-            '_type': 'url_transparent',
-            'url': rudo_url,
             'id': video_id,
             'title': title,
             'thumbnail': thumbnail,
             'uploader': uploader,
+            'formats': formats,
         }

youtube_dl/extractor/biqle.py

@@ -24,8 +24,7 @@ class BIQLEIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Ребенок в шоке от автоматической мойки',
             'uploader': 'Dmitry Kotov',
-        },
-        'skip': ' This video was marked as adult.  Embedding adult videos on external sites is prohibited.',
+        }
     }]
 
     def _real_extract(self, url):

youtube_dl/extractor/bloomberg.py

@@ -1,4 +1,3 @@
-# coding: utf-8
 from __future__ import unicode_literals
 
 import re
@@ -21,18 +20,6 @@ class BloombergIE(InfoExtractor):
         'params': {
             'format': 'best[format_id^=hds]',
         },
-    }, {
-        # video ID in BPlayer(...)
-        'url': 'http://www.bloomberg.com/features/2016-hello-world-new-zealand/',
-        'info_dict': {
-            'id': '938c7e72-3f25-4ddb-8b85-a9be731baa74',
-            'ext': 'flv',
-            'title': 'Meet the Real-Life Tech Wizards of Middle Earth',
-            'description': 'Hello World, Episode 1: New Zealand’s freaky AI babies, robot exoskeletons, and a virtual you.',
-        },
-        'params': {
-            'format': 'best[format_id^=hds]',
-        },
     }, {
         'url': 'http://www.bloomberg.com/news/articles/2015-11-12/five-strange-things-that-have-been-happening-in-financial-markets',
         'only_matching': True,
@@ -46,11 +33,7 @@ class BloombergIE(InfoExtractor):
         webpage = self._download_webpage(url, name)
         video_id = self._search_regex(
             r'["\']bmmrId["\']\s*:\s*(["\'])(?P<url>.+?)\1',
-            webpage, 'id', group='url', default=None)
-        if not video_id:
-            bplayer_data = self._parse_json(self._search_regex(
-                r'BPlayer\(null,\s*({[^;]+})\);', webpage, 'id'), name)
-            video_id = bplayer_data['id']
+            webpage, 'id', group='url')
         title = re.sub(': Video$', '', self._og_search_title(webpage))
 
         embed_info = self._download_json(

youtube_dl/extractor/br.py

@@ -29,8 +29,7 @@ class BRIE(InfoExtractor):
                 'duration': 180,
                 'uploader': 'Reinhard Weber',
                 'upload_date': '20150422',
-            },
-            'skip': '404 not found',
+            }
         },
         {
             'url': 'http://www.br.de/nachrichten/oberbayern/inhalt/muenchner-polizeipraesident-schreiber-gestorben-100.html',
@@ -41,8 +40,7 @@ class BRIE(InfoExtractor):
                 'title': 'Manfred Schreiber ist tot',
                 'description': 'md5:b454d867f2a9fc524ebe88c3f5092d97',
                 'duration': 26,
-            },
-            'skip': '404 not found',
+            }
         },
         {
             'url': 'https://www.br-klassik.de/audio/peeping-tom-premierenkritik-dance-festival-muenchen-100.html',
@@ -53,8 +51,7 @@ class BRIE(InfoExtractor):
                 'title': 'Kurzweilig und sehr bewegend',
                 'description': 'md5:0351996e3283d64adeb38ede91fac54e',
                 'duration': 296,
-            },
-            'skip': '404 not found',
+            }
         },
         {
             'url': 'http://www.br.de/radio/bayern1/service/team/videos/team-video-erdelt100.html',

youtube_dl/extractor/bravotv.py

@@ -1,74 +1,31 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-from .adobepass import AdobePassIE
-from ..utils import (
-    smuggle_url,
-    update_url_query,
-    int_or_none,
-)
+from .common import InfoExtractor
+from ..utils import smuggle_url
 
 
-class BravoTVIE(AdobePassIE):
-    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
-    _TESTS = [{
+class BravoTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+videos/(?P<id>[^/?]+)'
+    _TEST = {
         'url': 'http://www.bravotv.com/last-chance-kitchen/season-5/videos/lck-ep-12-fishy-finale',
-        'md5': '9086d0b7ef0ea2aabc4781d75f4e5863',
+        'md5': 'd60cdf68904e854fac669bd26cccf801',
         'info_dict': {
-            'id': 'zHyk1_HU_mPy',
+            'id': 'LitrBdX64qLn',
             'ext': 'mp4',
-            'title': 'LCK Ep 12: Fishy Finale',
-            'description': 'S13/E12: Two eliminated chefs have just 12 minutes to cook up a delicious fish dish.',
+            'title': 'Last Chance Kitchen Returns',
+            'description': 'S13: Last Chance Kitchen Returns for Top Chef Season 13',
+            'timestamp': 1448926740,
+            'upload_date': '20151130',
             'uploader': 'NBCU-BRAV',
-            'upload_date': '20160302',
-            'timestamp': 1456945320,
         }
-    }, {
-        'url': 'http://www.bravotv.com/below-deck/season-3/ep-14-reunion-part-1',
-        'only_matching': True,
-    }]
+    }
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        settings = self._parse_json(self._search_regex(
-            r'jQuery\.extend\(Drupal\.settings\s*,\s*({.+?})\);', webpage, 'drupal settings'),
-            display_id)
-        info = {}
-        query = {
-            'mbr': 'true',
-        }
-        account_pid, release_pid = [None] * 2
-        tve = settings.get('sharedTVE')
-        if tve:
-            query['manifest'] = 'm3u'
-            account_pid = 'HNK2IC'
-            release_pid = tve['release_pid']
-            if tve.get('entitlement') == 'auth':
-                adobe_pass = settings.get('adobePass', {})
-                resource = self._get_mvpd_resource(
-                    adobe_pass.get('adobePassResourceId', 'bravo'),
-                    tve['title'], release_pid, tve.get('rating'))
-                query['auth'] = self._extract_mvpd_auth(
-                    url, release_pid, adobe_pass.get('adobePassRequestorId', 'bravo'), resource)
-        else:
-            shared_playlist = settings['shared_playlist']
-            account_pid = shared_playlist['account_pid']
-            metadata = shared_playlist['video_metadata'][shared_playlist['default_clip']]
-            release_pid = metadata['release_pid']
-            info.update({
-                'title': metadata['title'],
-                'description': metadata.get('description'),
-                'season_number': int_or_none(metadata.get('season_num')),
-                'episode_number': int_or_none(metadata.get('episode_num')),
-            })
-            query['switch'] = 'progressive'
-        info.update({
-            '_type': 'url_transparent',
-            'id': release_pid,
-            'url': smuggle_url(update_url_query(
-                'http://link.theplatform.com/s/%s/%s' % (account_pid, release_pid),
-                query), {'force_smil_url': True}),
-            'ie_key': 'ThePlatform',
-        })
-        return info
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        account_pid = self._search_regex(r'"account_pid"\s*:\s*"([^"]+)"', webpage, 'account pid')
+        release_pid = self._search_regex(r'"release_pid"\s*:\s*"([^"]+)"', webpage, 'release pid')
+        return self.url_result(smuggle_url(
+            'http://link.theplatform.com/s/%s/%s?mbr=true&switch=progressive' % (account_pid, release_pid),
+            {'force_smil_url': True}), 'ThePlatform', release_pid)

youtube_dl/extractor/brightcove.py

@@ -26,8 +26,6 @@ from ..utils import (
     unescapeHTML,
     unsmuggle_url,
     update_url_query,
-    clean_html,
-    mimetype2ext,
 )
 
 
@@ -92,7 +90,6 @@ class BrightcoveLegacyIE(InfoExtractor):
                 'description': 'md5:363109c02998fee92ec02211bd8000df',
                 'uploader': 'National Ballet of Canada',
             },
-            'skip': 'Video gone',
         },
         {
             # test flv videos served by akamaihd.net
@@ -111,7 +108,7 @@ class BrightcoveLegacyIE(InfoExtractor):
             },
         },
         {
-            # playlist with 'videoList'
+            # playlist test
             # from http://support.brightcove.com/en/video-cloud/docs/playlist-support-single-video-players
             'url': 'http://c.brightcove.com/services/viewer/htmlFederated?playerID=3550052898001&playerKey=AQ%7E%7E%2CAAABmA9XpXk%7E%2C-Kp7jNgisre1fG5OdqpAFUTcs0lP_ZoL',
             'info_dict': {
@@ -120,15 +117,6 @@ class BrightcoveLegacyIE(InfoExtractor):
             },
             'playlist_mincount': 7,
         },
-        {
-            # playlist with 'playlistTab' (https://github.com/rg3/youtube-dl/issues/9965)
-            'url': 'http://c.brightcove.com/services/json/experience/runtime/?command=get_programming_for_experience&playerKey=AQ%7E%7E,AAABXlLMdok%7E,NJ4EoMlZ4rZdx9eU1rkMVd8EaYPBBUlg',
-            'info_dict': {
-                'id': '1522758701001',
-                'title': 'Lesson 08',
-            },
-            'playlist_mincount': 10,
-        },
     ]
     FLV_VCODECS = {
         1: 'SORENSON',
@@ -310,19 +298,13 @@ class BrightcoveLegacyIE(InfoExtractor):
             info_url, player_key, 'Downloading playlist information')
 
         json_data = json.loads(playlist_info)
-        if 'videoList' in json_data:
-            playlist_info = json_data['videoList']
-            playlist_dto = playlist_info['mediaCollectionDTO']
-        elif 'playlistTabs' in json_data:
-            playlist_info = json_data['playlistTabs']
-            playlist_dto = playlist_info['lineupListDTO']['playlistDTOs'][0]
-        else:
+        if 'videoList' not in json_data:
             raise ExtractorError('Empty playlist')
-
-        videos = [self._extract_video_info(video_info) for video_info in playlist_dto['videoDTOs']]
+        playlist_info = json_data['videoList']
+        videos = [self._extract_video_info(video_info) for video_info in playlist_info['mediaCollectionDTO']['videoDTOs']]
 
         return self.playlist_result(videos, playlist_id='%s' % playlist_info['id'],
-                                    playlist_title=playlist_dto['displayName'])
+                                    playlist_title=playlist_info['mediaCollectionDTO']['displayName'])
 
     def _extract_video_info(self, video_info):
         video_id = compat_str(video_info['id'])
@@ -546,16 +528,14 @@ class BrightcoveNewIE(InfoExtractor):
         formats = []
         for source in json_data.get('sources', []):
             container = source.get('container')
-            ext = mimetype2ext(source.get('type'))
+            source_type = source.get('type')
             src = source.get('src')
-            if ext == 'ism':
-                continue
-            elif ext == 'm3u8' or container == 'M2TS':
+            if source_type == 'application/x-mpegURL' or container == 'M2TS':
                 if not src:
                     continue
                 formats.extend(self._extract_m3u8_formats(
                     src, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
-            elif ext == 'mpd':
+            elif source_type == 'application/dash+xml':
                 if not src:
                     continue
                 formats.extend(self._extract_mpd_formats(src, video_id, 'dash', fatal=False))
@@ -571,7 +551,7 @@ class BrightcoveNewIE(InfoExtractor):
                     'tbr': tbr,
                     'filesize': int_or_none(source.get('size')),
                     'container': container,
-                    'ext': ext or container.lower(),
+                    'ext': container.lower(),
                 }
                 if width == 0 and height == 0:
                     f.update({
@@ -605,13 +585,6 @@ class BrightcoveNewIE(InfoExtractor):
                         'format_id': build_format_id('rtmp'),
                     })
                 formats.append(f)
-
-        errors = json_data.get('errors')
-        if not formats and errors:
-            error = errors[0]
-            raise ExtractorError(
-                error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
-
         self._sort_formats(formats)
 
         subtitles = {}
@@ -624,7 +597,7 @@ class BrightcoveNewIE(InfoExtractor):
         return {
             'id': video_id,
             'title': title,
-            'description': clean_html(json_data.get('description')),
+            'description': json_data.get('description'),
             'thumbnail': json_data.get('thumbnail') or json_data.get('poster'),
             'duration': float_or_none(json_data.get('duration'), 1000),
             'timestamp': parse_iso8601(json_data.get('published_at')),

youtube_dl/extractor/buzzfeed.py

@@ -5,7 +5,6 @@ import json
 import re
 
 from .common import InfoExtractor
-from .facebook import FacebookIE
 
 
 class BuzzFeedIE(InfoExtractor):
@@ -21,11 +20,11 @@ class BuzzFeedIE(InfoExtractor):
             'info_dict': {
                 'id': 'aVCR29aE_OQ',
                 'ext': 'mp4',
-                'title': 'Angry Ram destroys a punching bag..',
-                'description': 'md5:c59533190ef23fd4458a5e8c8c872345',
                 'upload_date': '20141024',
                 'uploader_id': 'Buddhanz1',
-                'uploader': 'Angry Ram',
+                'description': 'He likes to stay in shape with his heavy bag, he wont stop until its on the ground\n\nFollow Angry Ram on Facebook for regular updates -\nhttps://www.facebook.com/pages/Angry-Ram/1436897249899558?ref=hl',
+                'uploader': 'Buddhanz',
+                'title': 'Angry Ram destroys a punching bag',
             }
         }]
     }, {
@@ -42,30 +41,13 @@ class BuzzFeedIE(InfoExtractor):
             'info_dict': {
                 'id': 'mVmBL8B-In0',
                 'ext': 'mp4',
-                'title': 're:Munchkin the Teddy Bear gets her exercise',
-                'description': 'md5:28faab95cda6e361bcff06ec12fc21d8',
                 'upload_date': '20141124',
                 'uploader_id': 'CindysMunchkin',
+                'description': 're:© 2014 Munchkin the',
                 'uploader': 're:^Munchkin the',
+                'title': 're:Munchkin the Teddy Bear gets her exercise',
             },
         }]
-    }, {
-        'url': 'http://www.buzzfeed.com/craigsilverman/the-most-adorable-crash-landing-ever#.eq7pX0BAmK',
-        'info_dict': {
-            'id': 'the-most-adorable-crash-landing-ever',
-            'title': 'Watch This Baby Goose Make The Most Adorable Crash Landing',
-            'description': 'This gosling knows how to stick a landing.',
-        },
-        'playlist': [{
-            'md5': '763ca415512f91ca62e4621086900a23',
-            'info_dict': {
-                'id': '971793786185728',
-                'ext': 'mp4',
-                'title': 'We set up crash pads so that the goslings on our roof would have a safe landi...',
-                'uploader': 'Calgary Outdoor Centre-University of Calgary',
-            },
-        }],
-        'add_ie': ['Facebook'],
     }]
 
     def _real_extract(self, url):
@@ -84,10 +66,6 @@ class BuzzFeedIE(InfoExtractor):
                 continue
             entries.append(self.url_result(video['url']))
 
-        facebook_url = FacebookIE._extract_url(webpage)
-        if facebook_url:
-            entries.append(self.url_result(facebook_url))
-
         return {
             '_type': 'playlist',
             'id': playlist_id,

youtube_dl/extractor/camdemy.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import datetime
 import re
 
 from .common import InfoExtractor
@@ -9,10 +10,8 @@ from ..compat import (
     compat_urlparse,
 )
 from ..utils import (
-    clean_html,
-    parse_duration,
+    parse_iso8601,
     str_to_int,
-    unified_strdate,
 )
 
 
@@ -27,14 +26,14 @@ class CamdemyIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Ch1-1 Introduction, Signals (02-23-2012)',
             'thumbnail': 're:^https?://.*\.jpg$',
+            'description': '',
             'creator': 'ss11spring',
-            'duration': 1591,
             'upload_date': '20130114',
+            'timestamp': 1358154556,
             'view_count': int,
         }
     }, {
         # With non-empty description
-        # webpage returns "No permission or not login"
         'url': 'http://www.camdemy.com/media/13885',
         'md5': '4576a3bb2581f86c61044822adbd1249',
         'info_dict': {
@@ -42,71 +41,64 @@ class CamdemyIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'EverCam + Camdemy QuickStart',
             'thumbnail': 're:^https?://.*\.jpg$',
-            'description': 'md5:2a9f989c2b153a2342acee579c6e7db6',
+            'description': 'md5:050b62f71ed62928f8a35f1a41e186c9',
             'creator': 'evercam',
-            'duration': 318,
+            'upload_date': '20140620',
+            'timestamp': 1403271569,
         }
     }, {
-        # External source (YouTube)
+        # External source
         'url': 'http://www.camdemy.com/media/14842',
+        'md5': '50e1c3c3aa233d3d7b7daa2fa10b1cf7',
         'info_dict': {
             'id': '2vsYQzNIsJo',
             'ext': 'mp4',
-            'title': 'Excel 2013 Tutorial - How to add Password Protection',
-            'description': 'Excel 2013 Tutorial for Beginners - How to add Password Protection',
             'upload_date': '20130211',
             'uploader': 'Hun Kim',
+            'description': 'Excel 2013 Tutorial for Beginners - How to add Password Protection',
             'uploader_id': 'hunkimtutorials',
-        },
-        'params': {
-            'skip_download': True,
-        },
+            'title': 'Excel 2013 Tutorial - How to add Password Protection',
+        }
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, video_id)
+        page = self._download_webpage(url, video_id)
 
         src_from = self._html_search_regex(
-            r"class=['\"]srcFrom['\"][^>]*>Sources?(?:\s+from)?\s*:\s*<a[^>]+(?:href|title)=(['\"])(?P<url>(?:(?!\1).)+)\1",
-            webpage, 'external source', default=None, group='url')
+            r"<div class='srcFrom'>Source: <a title='([^']+)'", page,
+            'external source', default=None)
         if src_from:
             return self.url_result(src_from)
 
         oembed_obj = self._download_json(
             'http://www.camdemy.com/oembed/?format=json&url=' + url, video_id)
 
-        title = oembed_obj['title']
         thumb_url = oembed_obj['thumbnail_url']
         video_folder = compat_urlparse.urljoin(thumb_url, 'video/')
         file_list_doc = self._download_xml(
             compat_urlparse.urljoin(video_folder, 'fileList.xml'),
-            video_id, 'Downloading filelist XML')
+            video_id, 'Filelist XML')
         file_name = file_list_doc.find('./video/item/fileName').text
         video_url = compat_urlparse.urljoin(video_folder, file_name)
 
-        # Some URLs return "No permission or not login" in a webpage despite being
-        # freely available via oembed JSON URL (e.g. http://www.camdemy.com/media/13885)
-        upload_date = unified_strdate(self._search_regex(
-            r'>published on ([^<]+)<', webpage,
-            'upload date', default=None))
-        view_count = str_to_int(self._search_regex(
-            r'role=["\']viewCnt["\'][^>]*>([\d,.]+) views',
-            webpage, 'view count', default=None))
-        description = self._html_search_meta(
-            'description', webpage, default=None) or clean_html(
-            oembed_obj.get('description'))
+        timestamp = parse_iso8601(self._html_search_regex(
+            r"<div class='title'>Posted\s*:</div>\s*<div class='value'>([^<>]+)<",
+            page, 'creation time', fatal=False),
+            delimiter=' ', timezone=datetime.timedelta(hours=8))
+        view_count = str_to_int(self._html_search_regex(
+            r"<div class='title'>Views\s*:</div>\s*<div class='value'>([^<>]+)<",
+            page, 'view count', fatal=False))
 
         return {
             'id': video_id,
             'url': video_url,
-            'title': title,
+            'title': oembed_obj['title'],
             'thumbnail': thumb_url,
-            'description': description,
-            'creator': oembed_obj.get('author_name'),
-            'duration': parse_duration(oembed_obj.get('duration')),
-            'upload_date': upload_date,
+            'description': self._html_search_meta('description', page),
+            'creator': oembed_obj['author_name'],
+            'duration': oembed_obj['duration'],
+            'timestamp': timestamp,
             'view_count': view_count,
         }
 

youtube_dl/extractor/canalplus.py

@@ -4,11 +4,11 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_urllib_parse_urlparse
 from ..utils import (
     ExtractorError,
     HEADRequest,
     unified_strdate,
+    url_basename,
     qualities,
     int_or_none,
 )
@@ -16,38 +16,24 @@ from ..utils import (
 
 class CanalplusIE(InfoExtractor):
     IE_DESC = 'canalplus.fr, piwiplus.fr and d8.tv'
-    _VALID_URL = r'''(?x)
-                        https?://
-                            (?:
-                                (?:
-                                    (?:(?:www|m)\.)?canalplus\.fr|
-                                    (?:www\.)?piwiplus\.fr|
-                                    (?:www\.)?d8\.tv|
-                                    (?:www\.)?d17\.tv|
-                                    (?:www\.)?itele\.fr
-                                )/(?:(?:[^/]+/)*(?P<display_id>[^/?#&]+))?(?:\?.*\bvid=(?P<vid>\d+))?|
-                                player\.canalplus\.fr/#/(?P<id>\d+)
-                            )
-
-                    '''
+    _VALID_URL = r'https?://(?:www\.(?P<site>canalplus\.fr|piwiplus\.fr|d8\.tv|itele\.fr)/.*?/(?P<path>.*)|player\.canalplus\.fr/#/(?P<id>[0-9]+))'
     _VIDEO_INFO_TEMPLATE = 'http://service.canal-plus.com/video/rest/getVideosLiees/%s/%s?format=json'
     _SITE_ID_MAP = {
-        'canalplus': 'cplus',
-        'piwiplus': 'teletoon',
-        'd8': 'd8',
-        'd17': 'd17',
-        'itele': 'itele',
+        'canalplus.fr': 'cplus',
+        'piwiplus.fr': 'teletoon',
+        'd8.tv': 'd8',
+        'itele.fr': 'itele',
     }
 
     _TESTS = [{
-        'url': 'http://www.canalplus.fr/c-emissions/pid1830-c-zapping.html?vid=1192814',
-        'md5': '41f438a4904f7664b91b4ed0dec969dc',
+        'url': 'http://www.canalplus.fr/c-emissions/pid1830-c-zapping.html?vid=1263092',
+        'md5': '12164a6f14ff6df8bd628e8ba9b10b78',
         'info_dict': {
-            'id': '1192814',
+            'id': '1263092',
             'ext': 'mp4',
-            'title': "L'Année du Zapping 2014 - L'Année du Zapping 2014",
-            'description': "Toute l'année 2014 dans un Zapping exceptionnel !",
-            'upload_date': '20150105',
+            'title': 'Le Zapping - 13/05/15',
+            'description': 'md5:09738c0d06be4b5d06a0940edb0da73f',
+            'upload_date': '20150513',
         },
     }, {
         'url': 'http://www.piwiplus.fr/videos-piwi/pid1405-le-labyrinthe-boing-super-ranger.html?vid=1108190',
@@ -60,45 +46,35 @@ class CanalplusIE(InfoExtractor):
         },
         'skip': 'Only works from France',
     }, {
-        'url': 'http://www.d8.tv/d8-docs-mags/pid5198-d8-en-quete-d-actualite.html?vid=1390231',
+        'url': 'http://www.d8.tv/d8-docs-mags/pid6589-d8-campagne-intime.html',
         'info_dict': {
-            'id': '1390231',
-            'ext': 'mp4',
-            'title': "Vacances pas chères : prix discount ou grosses dépenses ? - En quête d'actualité",
-            'description': 'md5:edb6cf1cb4a1e807b5dd089e1ac8bfc6',
-            'upload_date': '20160512',
-        },
-        'params': {
-            'skip_download': True,
+            'id': '966289',
+            'ext': 'flv',
+            'title': 'Campagne intime - Documentaire exceptionnel',
+            'description': 'md5:d2643b799fb190846ae09c61e59a859f',
+            'upload_date': '20131108',
         },
+        'skip': 'videos get deleted after a while',
     }, {
-        'url': 'http://www.itele.fr/chroniques/invite-bruce-toussaint/thierry-solere-nicolas-sarkozy-officialisera-sa-candidature-a-la-primaire-quand-il-le-voudra-167224',
+        'url': 'http://www.itele.fr/france/video/aubervilliers-un-lycee-en-colere-111559',
+        'md5': '38b8f7934def74f0d6f3ba6c036a5f82',
         'info_dict': {
-            'id': '1398334',
+            'id': '1213714',
             'ext': 'mp4',
-            'title': "L'invité de Bruce Toussaint du 07/06/2016 - ",
-            'description': 'md5:40ac7c9ad0feaeb6f605bad986f61324',
-            'upload_date': '20160607',
+            'title': 'Aubervilliers : un lycée en colère - Le 11/02/2015 à 06h45',
+            'description': 'md5:8216206ec53426ea6321321f3b3c16db',
+            'upload_date': '20150211',
         },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://m.canalplus.fr/?vid=1398231',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.d17.tv/emissions/pid8303-lolywood.html?vid=1397061',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.groupdict().get('id') or mobj.groupdict().get('vid')
+        video_id = mobj.groupdict().get('id')
 
-        site_id = self._SITE_ID_MAP[compat_urllib_parse_urlparse(url).netloc.rsplit('.', 2)[-2]]
+        site_id = self._SITE_ID_MAP[mobj.group('site') or 'canal']
 
         # Beware, some subclasses do not define an id group
-        display_id = mobj.group('display_id') or video_id
+        display_id = url_basename(mobj.group('path'))
 
         if video_id is None:
             webpage = self._download_webpage(url, display_id)

youtube_dl/extractor/carambatv.py

@@ -1,88 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import compat_str
-from ..utils import (
-    float_or_none,
-    int_or_none,
-    try_get,
-)
-
-
-class CarambaTVIE(InfoExtractor):
-    _VALID_URL = r'(?:carambatv:|https?://video1\.carambatv\.ru/v/)(?P<id>\d+)'
-    _TESTS = [{
-        'url': 'http://video1.carambatv.ru/v/191910501',
-        'md5': '2f4a81b7cfd5ab866ee2d7270cb34a2a',
-        'info_dict': {
-            'id': '191910501',
-            'ext': 'mp4',
-            'title': '[BadComedian] - Разборка в Маниле (Абсолютный обзор)',
-            'thumbnail': 're:^https?://.*\.jpg',
-            'duration': 2678.31,
-        },
-    }, {
-        'url': 'carambatv:191910501',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        video = self._download_json(
-            'http://video1.carambatv.ru/v/%s/videoinfo.js' % video_id,
-            video_id)
-
-        title = video['title']
-
-        base_url = video.get('video') or 'http://video1.carambatv.ru/v/%s/' % video_id
-
-        formats = [{
-            'url': base_url + f['fn'],
-            'height': int_or_none(f.get('height')),
-            'format_id': '%sp' % f['height'] if f.get('height') else None,
-        } for f in video['qualities'] if f.get('fn')]
-        self._sort_formats(formats)
-
-        thumbnail = video.get('splash')
-        duration = float_or_none(try_get(
-            video, lambda x: x['annotations'][0]['end_time'], compat_str))
-
-        return {
-            'id': video_id,
-            'title': title,
-            'thumbnail': thumbnail,
-            'duration': duration,
-            'formats': formats,
-        }
-
-
-class CarambaTVPageIE(InfoExtractor):
-    _VALID_URL = r'https?://carambatv\.ru/(?:[^/]+/)+(?P<id>[^/?#&]+)'
-    _TEST = {
-        'url': 'http://carambatv.ru/movie/bad-comedian/razborka-v-manile/',
-        'md5': '',
-        'info_dict': {
-            'id': '191910501',
-            'ext': 'mp4',
-            'title': '[BadComedian] - Разборка в Маниле (Абсолютный обзор)',
-            'thumbnail': 're:^https?://.*\.jpg$',
-            'duration': 2678.31,
-        },
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, video_id)
-
-        video_url = self._og_search_property('video:iframe', webpage, default=None)
-
-        if not video_url:
-            video_id = self._search_regex(
-                r'(?:video_id|crmb_vuid)\s*[:=]\s*["\']?(\d+)',
-                webpage, 'video id')
-            video_url = 'carambatv:%s' % video_id
-
-        return self.url_result(video_url, CarambaTVIE.ie_key())

youtube_dl/extractor/cartoonnetwork.py

@@ -1,36 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .turner import TurnerBaseIE
-
-
-class CartoonNetworkIE(TurnerBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cartoonnetwork\.com/video/(?:[^/]+/)+(?P<id>[^/?#]+)-(?:clip|episode)\.html'
-    _TEST = {
-        'url': 'http://www.cartoonnetwork.com/video/teen-titans-go/starfire-the-cat-lady-clip.html',
-        'info_dict': {
-            'id': '8a250ab04ed07e6c014ef3f1e2f9016c',
-            'ext': 'mp4',
-            'title': 'Starfire the Cat Lady',
-            'description': 'Robin decides to become a cat so that Starfire will finally love him.',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-    }
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        id_type, video_id = re.search(r"_cnglobal\.cvp(Video|Title)Id\s*=\s*'([^']+)';", webpage).groups()
-        query = ('id' if id_type == 'Video' else 'titleId') + '=' + video_id
-        return self._extract_cvp_info(
-            'http://www.cartoonnetwork.com/video-seo-svc/episodeservices/getCvpPlaylist?networkName=CN2&' + query, video_id, {
-                'secure': {
-                    'media_src': 'http://apple-secure.cdn.turner.com/toon/big',
-                    'tokenizer_src': 'http://www.cartoonnetwork.com/cntv/mvpd/processors/services/token_ipadAdobe.do',
-                },
-            })

youtube_dl/extractor/cbc.py

@@ -4,24 +4,13 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_str
 from ..utils import (
     js_to_json,
     smuggle_url,
-    try_get,
-    xpath_text,
-    xpath_element,
-    xpath_with_ns,
-    find_xpath_attr,
-    parse_iso8601,
-    parse_age_limit,
-    int_or_none,
-    ExtractorError,
 )
 
 
 class CBCIE(InfoExtractor):
-    IE_NAME = 'cbc.ca'
     _VALID_URL = r'https?://(?:www\.)?cbc\.ca/(?!player/)(?:[^/]+/)+(?P<id>[^/?#]+)'
     _TESTS = [{
         # with mediaId
@@ -36,22 +25,8 @@ class CBCIE(InfoExtractor):
             'upload_date': '20160203',
             'uploader': 'CBCC-NEW',
         },
-        'skip': 'Geo-restricted to Canada',
     }, {
-        # with clipId, feed available via tpfeed.cbc.ca and feed.theplatform.com
-        'url': 'http://www.cbc.ca/22minutes/videos/22-minutes-update/22-minutes-update-episode-4',
-        'md5': '162adfa070274b144f4fdc3c3b8207db',
-        'info_dict': {
-            'id': '2414435309',
-            'ext': 'mp4',
-            'title': '22 Minutes Update: What Not To Wear Quebec',
-            'description': "This week's latest Canadian top political story is What Not To Wear Quebec.",
-            'upload_date': '20131025',
-            'uploader': 'CBCC-NEW',
-            'timestamp': 1382717907,
-        },
-    }, {
-        # with clipId, feed only available via tpfeed.cbc.ca
+        # with clipId
         'url': 'http://www.cbc.ca/archives/entry/1978-robin-williams-freestyles-on-90-minutes-live',
         'md5': '0274a90b51a9b4971fe005c63f592f12',
         'info_dict': {
@@ -89,7 +64,6 @@ class CBCIE(InfoExtractor):
                 'uploader': 'CBCC-NEW',
             },
         }],
-        'skip': 'Geo-restricted to Canada',
     }]
 
     @classmethod
@@ -107,15 +81,9 @@ class CBCIE(InfoExtractor):
             media_id = player_info.get('mediaId')
             if not media_id:
                 clip_id = player_info['clipId']
-                feed = self._download_json(
-                    'http://tpfeed.cbc.ca/f/ExhSPC/vms_5akSXx4Ng_Zn?byCustomValue={:mpsReleases}{%s}' % clip_id,
-                    clip_id, fatal=False)
-                if feed:
-                    media_id = try_get(feed, lambda x: x['entries'][0]['guid'], compat_str)
-                if not media_id:
-                    media_id = self._download_json(
-                        'http://feed.theplatform.com/f/h9dtGB/punlNGjMlc1F?fields=id&byContent=byReleases%3DbyId%253D' + clip_id,
-                        clip_id)['entries'][0]['id'].split('/')[-1]
+                media_id = self._download_json(
+                    'http://feed.theplatform.com/f/h9dtGB/punlNGjMlc1F?fields=id&byContent=byReleases%3DbyId%253D' + clip_id,
+                    clip_id)['entries'][0]['id'].split('/')[-1]
             return self.url_result('cbcplayer:%s' % media_id, 'CBCPlayer', media_id)
         else:
             entries = [self.url_result('cbcplayer:%s' % media_id, 'CBCPlayer', media_id) for media_id in re.findall(r'<iframe[^>]+src="[^"]+?mediaId=(\d+)"', webpage)]
@@ -123,7 +91,6 @@ class CBCIE(InfoExtractor):
 
 
 class CBCPlayerIE(InfoExtractor):
-    IE_NAME = 'cbc.ca:player'
     _VALID_URL = r'(?:cbcplayer:|https?://(?:www\.)?cbc\.ca/(?:player/play/|i/caffeine/syndicate/\?mediaId=))(?P<id>\d+)'
     _TESTS = [{
         'url': 'http://www.cbc.ca/player/play/2683190193',
@@ -137,7 +104,6 @@ class CBCPlayerIE(InfoExtractor):
             'upload_date': '20160210',
             'uploader': 'CBCC-NEW',
         },
-        'skip': 'Geo-restricted to Canada',
     }, {
         # Redirected from http://www.cbc.ca/player/AudioMobile/All%20in%20a%20Weekend%20Montreal/ID/2657632011/
         'url': 'http://www.cbc.ca/player/play/2657631896',
@@ -177,165 +143,3 @@ class CBCPlayerIE(InfoExtractor):
                 }),
             'id': video_id,
         }
-
-
-class CBCWatchBaseIE(InfoExtractor):
-    _device_id = None
-    _device_token = None
-    _API_BASE_URL = 'https://api-cbc.cloud.clearleap.com/cloffice/client/'
-    _NS_MAP = {
-        'media': 'http://search.yahoo.com/mrss/',
-        'clearleap': 'http://www.clearleap.com/namespace/clearleap/1.0/',
-    }
-
-    def _call_api(self, path, video_id):
-        url = path if path.startswith('http') else self._API_BASE_URL + path
-        result = self._download_xml(url, video_id, headers={
-            'X-Clearleap-DeviceId': self._device_id,
-            'X-Clearleap-DeviceToken': self._device_token,
-        })
-        error_message = xpath_text(result, 'userMessage') or xpath_text(result, 'systemMessage')
-        if error_message:
-            raise ExtractorError('%s said: %s' % (self.IE_NAME, error_message))
-        return result
-
-    def _real_initialize(self):
-        if not self._device_id or not self._device_token:
-            device = self._downloader.cache.load('cbcwatch', 'device') or {}
-            self._device_id, self._device_token = device.get('id'), device.get('token')
-            if not self._device_id or not self._device_token:
-                result = self._download_xml(
-                    self._API_BASE_URL + 'device/register',
-                    None, data=b'<device><type>web</type></device>')
-                self._device_id = xpath_text(result, 'deviceId', fatal=True)
-                self._device_token = xpath_text(result, 'deviceToken', fatal=True)
-                self._downloader.cache.store(
-                    'cbcwatch', 'device', {
-                        'id': self._device_id,
-                        'token': self._device_token,
-                    })
-
-    def _parse_rss_feed(self, rss):
-        channel = xpath_element(rss, 'channel', fatal=True)
-
-        def _add_ns(path):
-            return xpath_with_ns(path, self._NS_MAP)
-
-        entries = []
-        for item in channel.findall('item'):
-            guid = xpath_text(item, 'guid', fatal=True)
-            title = xpath_text(item, 'title', fatal=True)
-
-            media_group = xpath_element(item, _add_ns('media:group'), fatal=True)
-            content = xpath_element(media_group, _add_ns('media:content'), fatal=True)
-            content_url = content.attrib['url']
-
-            thumbnails = []
-            for thumbnail in media_group.findall(_add_ns('media:thumbnail')):
-                thumbnail_url = thumbnail.get('url')
-                if not thumbnail_url:
-                    continue
-                thumbnails.append({
-                    'id': thumbnail.get('profile'),
-                    'url': thumbnail_url,
-                    'width': int_or_none(thumbnail.get('width')),
-                    'height': int_or_none(thumbnail.get('height')),
-                })
-
-            timestamp = None
-            release_date = find_xpath_attr(
-                item, _add_ns('media:credit'), 'role', 'releaseDate')
-            if release_date is not None:
-                timestamp = parse_iso8601(release_date.text)
-
-            entries.append({
-                '_type': 'url_transparent',
-                'url': content_url,
-                'id': guid,
-                'title': title,
-                'description': xpath_text(item, 'description'),
-                'timestamp': timestamp,
-                'duration': int_or_none(content.get('duration')),
-                'age_limit': parse_age_limit(xpath_text(item, _add_ns('media:rating'))),
-                'episode': xpath_text(item, _add_ns('clearleap:episode')),
-                'episode_number': int_or_none(xpath_text(item, _add_ns('clearleap:episodeInSeason'))),
-                'series': xpath_text(item, _add_ns('clearleap:series')),
-                'season_number': int_or_none(xpath_text(item, _add_ns('clearleap:season'))),
-                'thumbnails': thumbnails,
-                'ie_key': 'CBCWatchVideo',
-            })
-
-        return self.playlist_result(
-            entries, xpath_text(channel, 'guid'),
-            xpath_text(channel, 'title'),
-            xpath_text(channel, 'description'))
-
-
-class CBCWatchVideoIE(CBCWatchBaseIE):
-    IE_NAME = 'cbc.ca:watch:video'
-    _VALID_URL = r'https?://api-cbc\.cloud\.clearleap\.com/cloffice/client/web/play/?\?.*?\bcontentId=(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        result = self._call_api(url, video_id)
-
-        m3u8_url = xpath_text(result, 'url', fatal=True)
-        formats = self._extract_m3u8_formats(re.sub(r'/([^/]+)/[^/?]+\.m3u8', r'/\1/\1.m3u8', m3u8_url), video_id, 'mp4', fatal=False)
-        if len(formats) < 2:
-            formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4')
-        # Despite metadata in m3u8 all video+audio formats are
-        # actually video-only (no audio)
-        for f in formats:
-            if f.get('acodec') != 'none' and f.get('vcodec') != 'none':
-                f['acodec'] = 'none'
-        self._sort_formats(formats)
-
-        info = {
-            'id': video_id,
-            'title': video_id,
-            'formats': formats,
-        }
-
-        rss = xpath_element(result, 'rss')
-        if rss:
-            info.update(self._parse_rss_feed(rss)['entries'][0])
-            del info['url']
-            del info['_type']
-            del info['ie_key']
-        return info
-
-
-class CBCWatchIE(CBCWatchBaseIE):
-    IE_NAME = 'cbc.ca:watch'
-    _VALID_URL = r'https?://watch\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
-    _TESTS = [{
-        'url': 'http://watch.cbc.ca/doc-zone/season-6/customer-disservice/38e815a-009e3ab12e4',
-        'info_dict': {
-            'id': '38e815a-009e3ab12e4',
-            'ext': 'mp4',
-            'title': 'Customer (Dis)Service',
-            'description': 'md5:8bdd6913a0fe03d4b2a17ebe169c7c87',
-            'upload_date': '20160219',
-            'timestamp': 1455840000,
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-            'format': 'bestvideo',
-        },
-        'skip': 'Geo-restricted to Canada',
-    }, {
-        'url': 'http://watch.cbc.ca/arthur/all/1ed4b385-cd84-49cf-95f0-80f004680057',
-        'info_dict': {
-            'id': '1ed4b385-cd84-49cf-95f0-80f004680057',
-            'title': 'Arthur',
-            'description': 'Arthur, the sweetest 8-year-old aardvark, and his pals solve all kinds of problems with humour, kindness and teamwork.',
-        },
-        'playlist_mincount': 30,
-        'skip': 'Geo-restricted to Canada',
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        rss = self._call_api('web/browse/' + video_id, video_id)
-        return self._parse_rss_feed(rss)

youtube_dl/extractor/cbs.py

@@ -1,14 +1,17 @@
 from __future__ import unicode_literals
 
-from .theplatform import ThePlatformFeedIE
+import re
+
+from .theplatform import ThePlatformIE
 from ..utils import (
+    xpath_text,
+    xpath_element,
     int_or_none,
     find_xpath_attr,
-    ExtractorError,
 )
 
 
-class CBSBaseIE(ThePlatformFeedIE):
+class CBSBaseIE(ThePlatformIE):
     def _parse_smil_subtitles(self, smil, namespace=None, subtitles_lang='en'):
         closed_caption_e = find_xpath_attr(smil, self._xpath_ns('.//param', namespace), 'name', 'ClosedCaptionURL')
         return {
@@ -20,12 +23,13 @@ class CBSBaseIE(ThePlatformFeedIE):
 
 
 class CBSIE(CBSBaseIE):
-    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
+    _VALID_URL = r'(?:cbs:(?P<content_id>\w+)|https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/(?:video|artist)|colbertlateshow\.com/(?:video|podcasts))/[^/]+/(?P<display_id>[^/]+))'
 
     _TESTS = [{
         'url': 'http://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
         'info_dict': {
             'id': '_u7W953k6la293J7EPTd9oHkSPs6Xn6_',
+            'display_id': 'connect-chat-feat-garth-brooks',
             'ext': 'mp4',
             'title': 'Connect Chat feat. Garth Brooks',
             'description': 'Connect with country music singer Garth Brooks, as he chats with fans on Wednesday November 27, 2013. Be sure to tune in to Garth Brooks: Live from Las Vegas, Friday November 29, at 9/8c on CBS!',
@@ -35,7 +39,22 @@ class CBSIE(CBSBaseIE):
             'uploader': 'CBSI-NEW',
         },
         'params': {
-            # m3u8 download
+            # rtmp download
+            'skip_download': True,
+        },
+        '_skip': 'Blocked outside the US',
+    }, {
+        'url': 'http://www.cbs.com/shows/liveonletterman/artist/221752/st-vincent/',
+        'info_dict': {
+            'id': 'WWF_5KqY3PK1',
+            'display_id': 'st-vincent',
+            'ext': 'flv',
+            'title': 'Live on Letterman - St. Vincent',
+            'description': 'Live On Letterman: St. Vincent in concert from New York\'s Ed Sullivan Theater on Tuesday, July 16, 2014.',
+            'duration': 3221,
+        },
+        'params': {
+            # rtmp download
             'skip_download': True,
         },
         '_skip': 'Blocked outside the US',
@@ -46,31 +65,47 @@ class CBSIE(CBSBaseIE):
         'url': 'http://www.colbertlateshow.com/podcasts/dYSwjqPs_X1tvbV_P2FcPWRa_qT6akTC/in-the-bad-room-with-stephen/',
         'only_matching': True,
     }]
-
-    def _extract_video_info(self, guid):
-        path = 'dJ5BDC/media/guid/2198311517/' + guid
-        smil_url = 'http://link.theplatform.com/s/%s?mbr=true' % path
-        formats, subtitles = self._extract_theplatform_smil(smil_url + '&manifest=m3u', guid)
-        for r in ('OnceURL&formats=M3U', 'HLS&formats=M3U', 'RTMP', 'WIFI', '3G'):
-            try:
-                tp_formats, _ = self._extract_theplatform_smil(smil_url + '&assetTypes=' + r, guid, 'Downloading %s SMIL data' % r.split('&')[0])
-                formats.extend(tp_formats)
-            except ExtractorError:
-                continue
-        self._sort_formats(formats)
-        metadata = self._download_theplatform_metadata(path, guid)
-        info = self._parse_theplatform_metadata(metadata)
-        info.update({
-            'id': guid,
-            'formats': formats,
-            'subtitles': subtitles,
-            'series': metadata.get('cbs$SeriesTitle'),
-            'season_number': int_or_none(metadata.get('cbs$SeasonNumber')),
-            'episode': metadata.get('cbs$EpisodeTitle'),
-            'episode_number': int_or_none(metadata.get('cbs$EpisodeNumber')),
-        })
-        return info
+    TP_RELEASE_URL_TEMPLATE = 'http://link.theplatform.com/s/dJ5BDC/%s?mbr=true'
 
     def _real_extract(self, url):
-        content_id = self._match_id(url)
-        return self._extract_video_info(content_id)
+        content_id, display_id = re.match(self._VALID_URL, url).groups()
+        if not content_id:
+            webpage = self._download_webpage(url, display_id)
+            content_id = self._search_regex(
+                [r"video\.settings\.content_id\s*=\s*'([^']+)';", r"cbsplayer\.contentId\s*=\s*'([^']+)';"],
+                webpage, 'content id')
+        items_data = self._download_xml(
+            'http://can.cbs.com/thunder/player/videoPlayerService.php',
+            content_id, query={'partner': 'cbs', 'contentId': content_id})
+        video_data = xpath_element(items_data, './/item')
+        title = xpath_text(video_data, 'videoTitle', 'title', True)
+
+        subtitles = {}
+        formats = []
+        for item in items_data.findall('.//item'):
+            pid = xpath_text(item, 'pid')
+            if not pid:
+                continue
+            tp_release_url = self.TP_RELEASE_URL_TEMPLATE % pid
+            if '.m3u8' in xpath_text(item, 'contentUrl', default=''):
+                tp_release_url += '&manifest=m3u'
+            tp_formats, tp_subtitles = self._extract_theplatform_smil(
+                tp_release_url, content_id, 'Downloading %s SMIL data' % pid)
+            formats.extend(tp_formats)
+            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
+        self._sort_formats(formats)
+
+        info = self.get_metadata('dJ5BDC/media/guid/2198311517/%s' % content_id, content_id)
+        info.update({
+            'id': content_id,
+            'display_id': display_id,
+            'title': title,
+            'series': xpath_text(video_data, 'seriesTitle'),
+            'season_number': int_or_none(xpath_text(video_data, 'seasonNumber')),
+            'episode_number': int_or_none(xpath_text(video_data, 'episodeNumber')),
+            'duration': int_or_none(xpath_text(video_data, 'videoLength'), 1000),
+            'thumbnail': xpath_text(video_data, 'previewImageURL'),
+            'formats': formats,
+            'subtitles': subtitles,
+        })
+        return info

youtube_dl/extractor/cbsinteractive.py

@@ -80,6 +80,9 @@ class CBSInteractiveIE(ThePlatformIE):
 
         media_guid_path = 'media/guid/%d/%s' % (self.MPX_ACCOUNTS[site], vdata['mpxRefId'])
         formats, subtitles = [], {}
+        if site == 'cnet':
+            formats, subtitles = self._extract_theplatform_smil(
+                self.TP_RELEASE_URL_TEMPLATE % media_guid_path, video_id)
         for (fkey, vid) in vdata['files'].items():
             if fkey == 'hls_phone' and 'hls_tablet' in vdata['files']:
                 continue
@@ -91,7 +94,7 @@ class CBSInteractiveIE(ThePlatformIE):
             subtitles = self._merge_subtitles(subtitles, tp_subtitles)
         self._sort_formats(formats)
 
-        info = self._extract_theplatform_metadata('kYEXFC/%s' % media_guid_path, video_id)
+        info = self.get_metadata('kYEXFC/%s' % media_guid_path, video_id)
         info.update({
             'id': video_id,
             'display_id': display_id,

youtube_dl/extractor/cbslocal.py

@@ -1,10 +1,12 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import calendar
+import datetime
+
 from .anvato import AnvatoIE
 from .sendtonews import SendtoNewsIE
 from ..compat import compat_urlparse
-from ..utils import unified_timestamp
 
 
 class CBSLocalIE(AnvatoIE):
@@ -41,8 +43,13 @@ class CBSLocalIE(AnvatoIE):
         'url': 'http://cleveland.cbslocal.com/2016/05/16/indians-score-season-high-15-runs-in-blowout-win-over-reds-rapid-reaction/',
         'info_dict': {
             'id': 'GxfCe0Zo7D-175909-5588',
+            'ext': 'mp4',
+            'title': 'Recap: CLE 15, CIN 6',
+            'description': '5/16/16: Indians\' bats explode for 15 runs in a win',
+            'upload_date': '20160516',
+            'timestamp': 1463433840,
+            'duration': 49,
         },
-        'playlist_count': 9,
         'params': {
             # m3u8 download
             'skip_download': True,
@@ -55,15 +62,19 @@ class CBSLocalIE(AnvatoIE):
 
         sendtonews_url = SendtoNewsIE._extract_url(webpage)
         if sendtonews_url:
-            return self.url_result(
-                compat_urlparse.urljoin(url, sendtonews_url),
-                ie=SendtoNewsIE.ie_key())
-
-        info_dict = self._extract_anvato_videos(webpage, display_id)
+            info_dict = {
+                '_type': 'url_transparent',
+                'url': compat_urlparse.urljoin(url, sendtonews_url),
+            }
+        else:
+            info_dict = self._extract_anvato_videos(webpage, display_id)
 
         time_str = self._html_search_regex(
             r'class="entry-date">([^<]+)<', webpage, 'released date', fatal=False)
-        timestamp = unified_timestamp(time_str)
+        timestamp = None
+        if time_str:
+            timestamp = calendar.timegm(datetime.datetime.strptime(
+                time_str, '%b %d, %Y %I:%M %p').timetuple())
 
         info_dict.update({
             'display_id': display_id,

youtube_dl/extractor/cbsnews.py

@@ -2,13 +2,13 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from .cbs import CBSIE
+from .cbs import CBSBaseIE
 from ..utils import (
     parse_duration,
 )
 
 
-class CBSNewsIE(CBSIE):
+class CBSNewsIE(CBSBaseIE):
     IE_DESC = 'CBS News'
     _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/(?:news|videos)/(?P<id>[\da-z_-]+)'
 
@@ -26,18 +26,13 @@ class CBSNewsIE(CBSIE):
                 # rtmp download
                 'skip_download': True,
             },
-            'skip': 'Subscribers only',
         },
         {
             'url': 'http://www.cbsnews.com/videos/fort-hood-shooting-army-downplays-mental-illness-as-cause-of-attack/',
             'info_dict': {
-                'id': 'SNJBOYzXiWBOvaLsdzwH8fmtP1SCd91Y',
+                'id': 'fort-hood-shooting-army-downplays-mental-illness-as-cause-of-attack',
                 'ext': 'mp4',
                 'title': 'Fort Hood shooting: Army downplays mental illness as cause of attack',
-                'description': 'md5:4a6983e480542d8b333a947bfc64ddc7',
-                'upload_date': '20140404',
-                'timestamp': 1396650660,
-                'uploader': 'CBSI-NEW',
                 'thumbnail': 're:^https?://.*\.jpg$',
                 'duration': 205,
                 'subtitles': {
@@ -63,15 +58,36 @@ class CBSNewsIE(CBSIE):
             webpage, 'video JSON info'), video_id)
 
         item = video_info['item'] if 'item' in video_info else video_info
-        guid = item['mpxRefId']
-        return self._extract_video_info(guid)
+        title = item.get('articleTitle') or item.get('hed')
+        duration = item.get('duration')
+        thumbnail = item.get('mediaImage') or item.get('thumbnail')
+
+        subtitles = {}
+        formats = []
+        for format_id in ['RtmpMobileLow', 'RtmpMobileHigh', 'Hls', 'RtmpDesktop']:
+            pid = item.get('media' + format_id)
+            if not pid:
+                continue
+            release_url = 'http://link.theplatform.com/s/dJ5BDC/%s?mbr=true' % pid
+            tp_formats, tp_subtitles = self._extract_theplatform_smil(release_url, video_id, 'Downloading %s SMIL data' % pid)
+            formats.extend(tp_formats)
+            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'thumbnail': thumbnail,
+            'duration': duration,
+            'formats': formats,
+            'subtitles': subtitles,
+        }
 
 
 class CBSNewsLiveVideoIE(InfoExtractor):
     IE_DESC = 'CBS News Live Videos'
     _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/live/video/(?P<id>[\da-z_-]+)'
 
-    # Live videos get deleted soon. See http://www.cbsnews.com/live/ for the latest examples
     _TEST = {
         'url': 'http://www.cbsnews.com/live/video/clinton-sanders-prepare-to-face-off-in-nh/',
         'info_dict': {
@@ -80,7 +96,6 @@ class CBSNewsLiveVideoIE(InfoExtractor):
             'title': 'Clinton, Sanders Prepare To Face Off In NH',
             'duration': 334,
         },
-        'skip': 'Video gone',
     }
 
     def _real_extract(self, url):

youtube_dl/extractor/cbssports.py

@@ -1,31 +1,30 @@
 from __future__ import unicode_literals
 
-from .cbs import CBSBaseIE
+import re
+
+from .common import InfoExtractor
 
 
-class CBSSportsIE(CBSBaseIE):
-    _VALID_URL = r'https?://www\.cbssports\.com/video/player/[^/]+/(?P<id>\d+)'
+class CBSSportsIE(InfoExtractor):
+    _VALID_URL = r'https?://www\.cbssports\.com/video/player/(?P<section>[^/]+)/(?P<id>[^/]+)'
 
-    _TESTS = [{
-        'url': 'http://www.cbssports.com/video/player/videos/708337219968/0/ben-simmons-the-next-lebron?-not-so-fast',
+    _TEST = {
+        'url': 'http://www.cbssports.com/video/player/tennis/318462531970/0/us-open-flashbacks-1990s',
         'info_dict': {
-            'id': '708337219968',
-            'ext': 'mp4',
-            'title': 'Ben Simmons the next LeBron? Not so fast',
-            'description': 'md5:854294f627921baba1f4b9a990d87197',
-            'timestamp': 1466293740,
-            'upload_date': '20160618',
-            'uploader': 'CBSI-NEW',
+            'id': '_d5_GbO8p1sT',
+            'ext': 'flv',
+            'title': 'US Open flashbacks: 1990s',
+            'description': 'Bill Macatee relives the best moments in US Open history from the 1990s.',
         },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        }
-    }]
-
-    def _extract_video_info(self, filter_query, video_id):
-        return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
+    }
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
-        return self._extract_video_info('byId=%s' % video_id, video_id)
+        mobj = re.match(self._VALID_URL, url)
+        section = mobj.group('section')
+        video_id = mobj.group('id')
+        all_videos = self._download_json(
+            'http://www.cbssports.com/data/video/player/getVideos/%s?as=json' % section,
+            video_id)
+        # The json file contains the info of all the videos in the section
+        video_info = next(v for v in all_videos if v['pcid'] == video_id)
+        return self.url_result('theplatform:%s' % video_info['pid'], 'ThePlatform')

youtube_dl/extractor/cda.py

@@ -58,8 +58,7 @@ class CDAIE(InfoExtractor):
         def extract_format(page, version):
             unpacked = decode_packed_codes(page)
             format_url = self._search_regex(
-                r"(?:file|url)\s*:\s*(\\?[\"'])(?P<url>http.+?)\1", unpacked,
-                '%s url' % version, fatal=False, group='url')
+                r"url:\\'(.+?)\\'", unpacked, '%s url' % version, fatal=False)
             if not format_url:
                 return
             f = {
@@ -76,8 +75,7 @@ class CDAIE(InfoExtractor):
             info_dict['formats'].append(f)
             if not info_dict['duration']:
                 info_dict['duration'] = parse_duration(self._search_regex(
-                    r"duration\s*:\s*(\\?[\"'])(?P<duration>.+?)\1",
-                    unpacked, 'duration', fatal=False, group='duration'))
+                    r"duration:\\'(.+?)\\'", unpacked, 'duration', fatal=False))
 
         extract_format(webpage, 'default')
 

youtube_dl/extractor/channel9.py

@@ -20,64 +20,54 @@ class Channel9IE(InfoExtractor):
     '''
     IE_DESC = 'Channel 9'
     IE_NAME = 'channel9'
-    _VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+?)(?P<rss>/RSS)?/?(?:[?#&]|$)'
+    _VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+)/?'
 
-    _TESTS = [{
-        'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
-        'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
-        'info_dict': {
-            'id': 'Events/TechEd/Australia/2013/KOS002',
-            'ext': 'mp4',
-            'title': 'Developer Kick-Off Session: Stuff We Love',
-            'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
-            'duration': 4576,
-            'thumbnail': 're:http://.*\.jpg',
-            'session_code': 'KOS002',
-            'session_day': 'Day 1',
-            'session_room': 'Arena 1A',
-            'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug',
-                                 'Mads Kristensen'],
+    _TESTS = [
+        {
+            'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
+            'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
+            'info_dict': {
+                'id': 'Events/TechEd/Australia/2013/KOS002',
+                'ext': 'mp4',
+                'title': 'Developer Kick-Off Session: Stuff We Love',
+                'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
+                'duration': 4576,
+                'thumbnail': 're:http://.*\.jpg',
+                'session_code': 'KOS002',
+                'session_day': 'Day 1',
+                'session_room': 'Arena 1A',
+                'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug', 'Mads Kristensen'],
+            },
         },
-    }, {
-        'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
-        'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
-        'info_dict': {
-            'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
-            'ext': 'mp4',
-            'title': 'Self-service BI with Power BI - nuclear testing',
-            'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
-            'duration': 1540,
-            'thumbnail': 're:http://.*\.jpg',
-            'authors': ['Mike Wilmot'],
+        {
+            'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
+            'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
+            'info_dict': {
+                'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
+                'ext': 'mp4',
+                'title': 'Self-service BI with Power BI - nuclear testing',
+                'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
+                'duration': 1540,
+                'thumbnail': 're:http://.*\.jpg',
+                'authors': ['Mike Wilmot'],
+            },
         },
-    }, {
-        # low quality mp4 is best
-        'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
-        'info_dict': {
-            'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
-            'ext': 'mp4',
-            'title': 'Ranges for the Standard Library',
-            'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
-            'duration': 5646,
-            'thumbnail': 're:http://.*\.jpg',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'https://channel9.msdn.com/Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b/RSS',
-        'info_dict': {
-            'id': 'Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b',
-            'title': 'Channel 9',
-        },
-        'playlist_count': 2,
-    }, {
-        'url': 'https://channel9.msdn.com/Events/DEVintersection/DEVintersection-2016/RSS',
-        'only_matching': True,
-    }, {
-        'url': 'https://channel9.msdn.com/Events/Speakers/scott-hanselman/RSS?UrlSafeName=scott-hanselman',
-        'only_matching': True,
-    }]
+        {
+            # low quality mp4 is best
+            'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
+            'info_dict': {
+                'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
+                'ext': 'mp4',
+                'title': 'Ranges for the Standard Library',
+                'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
+                'duration': 5646,
+                'thumbnail': 're:http://.*\.jpg',
+            },
+            'params': {
+                'skip_download': True,
+            },
+        }
+    ]
 
     _RSS_URL = 'http://channel9.msdn.com/%s/RSS'
 
@@ -264,30 +254,22 @@ class Channel9IE(InfoExtractor):
 
         return self.playlist_result(contents)
 
-    def _extract_list(self, video_id, rss_url=None):
-        if not rss_url:
-            rss_url = self._RSS_URL % video_id
-        rss = self._download_xml(rss_url, video_id, 'Downloading RSS')
+    def _extract_list(self, content_path):
+        rss = self._download_xml(self._RSS_URL % content_path, content_path, 'Downloading RSS')
         entries = [self.url_result(session_url.text, 'Channel9')
                    for session_url in rss.findall('./channel/item/link')]
         title_text = rss.find('./channel/title').text
-        return self.playlist_result(entries, video_id, title_text)
+        return self.playlist_result(entries, content_path, title_text)
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
         content_path = mobj.group('contentpath')
-        rss = mobj.group('rss')
 
-        if rss:
-            return self._extract_list(content_path, url)
+        webpage = self._download_webpage(url, content_path, 'Downloading web page')
 
-        webpage = self._download_webpage(
-            url, content_path, 'Downloading web page')
-
-        page_type = self._search_regex(
-            r'<meta[^>]+name=(["\'])WT\.entryid\1[^>]+content=(["\'])(?P<pagetype>[^:]+).+?\2',
-            webpage, 'page type', default=None, group='pagetype')
-        if page_type:
+        page_type_m = re.search(r'<meta name="WT.entryid" content="(?P<pagetype>[^:]+)[^"]+"/>', webpage)
+        if page_type_m is not None:
+            page_type = page_type_m.group('pagetype')
             if page_type == 'Entry':      # Any 'item'-like page, may contain downloadable content
                 return self._extract_entry_item(webpage, content_path)
             elif page_type == 'Session':  # Event session page, may contain downloadable content
@@ -296,5 +278,6 @@ class Channel9IE(InfoExtractor):
                 return self._extract_list(content_path)
             else:
                 raise ExtractorError('Unexpected WT.entryid %s' % page_type, expected=True)
+
         else:  # Assuming list
             return self._extract_list(content_path)

youtube_dl/extractor/charlierose.py

@@ -1,51 +0,0 @@
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import remove_end
-
-
-class CharlieRoseIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?charlierose\.com/video(?:s|/player)/(?P<id>\d+)'
-    _TESTS = [{
-        'url': 'https://charlierose.com/videos/27996',
-        'md5': 'fda41d49e67d4ce7c2411fd2c4702e09',
-        'info_dict': {
-            'id': '27996',
-            'ext': 'mp4',
-            'title': 'Remembering Zaha Hadid',
-            'thumbnail': 're:^https?://.*\.jpg\?\d+',
-            'description': 'We revisit past conversations with Zaha Hadid, in memory of the world renowned Iraqi architect.',
-            'subtitles': {
-                'en': [{
-                    'ext': 'vtt',
-                }],
-            },
-        },
-    }, {
-        'url': 'https://charlierose.com/videos/27996',
-        'only_matching': True,
-    }]
-
-    _PLAYER_BASE = 'https://charlierose.com/video/player/%s'
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(self._PLAYER_BASE % video_id, video_id)
-
-        title = remove_end(self._og_search_title(webpage), ' - Charlie Rose')
-
-        info_dict = self._parse_html5_media_entries(
-            self._PLAYER_BASE % video_id, webpage, video_id,
-            m3u8_entry_protocol='m3u8_native')[0]
-
-        self._sort_formats(info_dict['formats'])
-        self._remove_duplicate_formats(info_dict['formats'])
-
-        info_dict.update({
-            'id': video_id,
-            'title': title,
-            'thumbnail': self._og_search_thumbnail(webpage),
-            'description': self._og_search_description(webpage),
-        })
-
-        return info_dict

youtube_dl/extractor/chaturbate.py

@@ -17,8 +17,7 @@ class ChaturbateIE(InfoExtractor):
         },
         'params': {
             'skip_download': True,
-        },
-        'skip': 'Room is offline',
+        }
     }, {
         'url': 'https://en.chaturbate.com/siswet19/',
         'only_matching': True,

youtube_dl/extractor/chirbit.py

@@ -1,33 +1,30 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import base64
-
 from .common import InfoExtractor
-from ..utils import parse_duration
+from ..utils import (
+    parse_duration,
+    int_or_none,
+)
 
 
 class ChirbitIE(InfoExtractor):
     IE_NAME = 'chirbit'
     _VALID_URL = r'https?://(?:www\.)?chirb\.it/(?:(?:wp|pl)/|fb_chirbit_player\.swf\?key=)?(?P<id>[\da-zA-Z]+)'
     _TESTS = [{
-        'url': 'http://chirb.it/be2abG',
+        'url': 'http://chirb.it/PrIPv5',
+        'md5': '9847b0dad6ac3e074568bf2cfb197de8',
         'info_dict': {
-            'id': 'be2abG',
+            'id': 'PrIPv5',
             'ext': 'mp3',
-            'title': 'md5:f542ea253f5255240be4da375c6a5d7e',
-            'description': 'md5:f24a4e22a71763e32da5fed59e47c770',
-            'duration': 306,
-        },
-        'params': {
-            'skip_download': True,
+            'title': 'Фасадстрой',
+            'duration': 52,
+            'view_count': int,
+            'comment_count': int,
         }
     }, {
         'url': 'https://chirb.it/fb_chirbit_player.swf?key=PrIPv5',
         'only_matching': True,
-    }, {
-        'url': 'https://chirb.it/wp/MN58c2',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -36,30 +33,27 @@ class ChirbitIE(InfoExtractor):
         webpage = self._download_webpage(
             'http://chirb.it/%s' % audio_id, audio_id)
 
-        data_fd = self._search_regex(
-            r'data-fd=(["\'])(?P<url>(?:(?!\1).)+)\1',
-            webpage, 'data fd', group='url')
-
-        # Reverse engineered from https://chirb.it/js/chirbit.player.js (look
-        # for soundURL)
-        audio_url = base64.b64decode(
-            data_fd[::-1].encode('ascii')).decode('utf-8')
+        audio_url = self._search_regex(
+            r'"setFile"\s*,\s*"([^"]+)"', webpage, 'audio url')
 
         title = self._search_regex(
-            r'class=["\']chirbit-title["\'][^>]*>([^<]+)', webpage, 'title')
-        description = self._search_regex(
-            r'<h3>Description</h3>\s*<pre[^>]*>([^<]+)</pre>',
-            webpage, 'description', default=None)
-        duration = parse_duration(self._search_regex(
-            r'class=["\']c-length["\'][^>]*>([^<]+)',
-            webpage, 'duration', fatal=False))
+            r'itemprop="name">([^<]+)', webpage, 'title')
+        duration = parse_duration(self._html_search_meta(
+            'duration', webpage, 'duration', fatal=False))
+        view_count = int_or_none(self._search_regex(
+            r'itemprop="playCount"\s*>(\d+)', webpage,
+            'listen count', fatal=False))
+        comment_count = int_or_none(self._search_regex(
+            r'>(\d+) Comments?:', webpage,
+            'comment count', fatal=False))
 
         return {
             'id': audio_id,
             'url': audio_url,
             'title': title,
-            'description': description,
             'duration': duration,
+            'view_count': view_count,
+            'comment_count': comment_count,
         }
 
 

youtube_dl/extractor/cliphunter.py

@@ -23,7 +23,7 @@ class CliphunterIE(InfoExtractor):
         (?P<id>[0-9]+)/
         (?P<seo>.+?)(?:$|[#\?])
     '''
-    _TESTS = [{
+    _TEST = {
         'url': 'http://www.cliphunter.com/w/1012420/Fun_Jynx_Maze_solo',
         'md5': 'b7c9bbd4eb3a226ab91093714dcaa480',
         'info_dict': {
@@ -32,19 +32,8 @@ class CliphunterIE(InfoExtractor):
             'title': 'Fun Jynx Maze solo',
             'thumbnail': 're:^https?://.*\.jpg$',
             'age_limit': 18,
-        },
-        'skip': 'Video gone',
-    }, {
-        'url': 'http://www.cliphunter.com/w/2019449/ShesNew__My_booty_girlfriend_Victoria_Paradices_pussy_filled_with_jizz',
-        'md5': '55a723c67bfc6da6b0cfa00d55da8a27',
-        'info_dict': {
-            'id': '2019449',
-            'ext': 'mp4',
-            'title': 'ShesNew - My booty girlfriend, Victoria Paradice\'s pussy filled with jizz',
-            'thumbnail': 're:^https?://.*\.jpg$',
-            'age_limit': 18,
-        },
-    }]
+        }
+    }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)

youtube_dl/extractor/cliprs.py

@@ -1,10 +1,16 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-from .onet import OnetBaseIE
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    float_or_none,
+    int_or_none,
+    parse_iso8601,
+)
 
 
-class ClipRsIE(OnetBaseIE):
+class ClipRsIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?clip\.rs/(?P<id>[^/]+)/\d+'
     _TEST = {
         'url': 'http://www.clip.rs/premijera-frajle-predstavljaju-novi-spot-za-pesmu-moli-me-moli/3732',
@@ -21,13 +27,64 @@ class ClipRsIE(OnetBaseIE):
     }
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
+        video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, display_id)
+        webpage = self._download_webpage(url, video_id)
 
-        mvp_id = self._search_mvp_id(webpage)
+        video_id = self._search_regex(
+            r'id=(["\'])mvp:(?P<id>.+?)\1', webpage, 'mvp id', group='id')
 
-        info_dict = self._extract_from_id(mvp_id, webpage)
-        info_dict['display_id'] = display_id
+        response = self._download_json(
+            'http://qi.ckm.onetapi.pl/', video_id,
+            query={
+                'body[id]': video_id,
+                'body[jsonrpc]': '2.0',
+                'body[method]': 'get_asset_detail',
+                'body[params][ID_Publikacji]': video_id,
+                'body[params][Service]': 'www.onet.pl',
+                'content-type': 'application/jsonp',
+                'x-onet-app': 'player.front.onetapi.pl',
+            })
 
-        return info_dict
+        error = response.get('error')
+        if error:
+            raise ExtractorError(
+                '%s said: %s' % (self.IE_NAME, error['message']), expected=True)
+
+        video = response['result'].get('0')
+
+        formats = []
+        for _, formats_dict in video['formats'].items():
+            if not isinstance(formats_dict, dict):
+                continue
+            for format_id, format_list in formats_dict.items():
+                if not isinstance(format_list, list):
+                    continue
+                for f in format_list:
+                    if not f.get('url'):
+                        continue
+                    formats.append({
+                        'url': f['url'],
+                        'format_id': format_id,
+                        'height': int_or_none(f.get('vertical_resolution')),
+                        'width': int_or_none(f.get('horizontal_resolution')),
+                        'abr': float_or_none(f.get('audio_bitrate')),
+                        'vbr': float_or_none(f.get('video_bitrate')),
+                    })
+        self._sort_formats(formats)
+
+        meta = video.get('meta', {})
+
+        title = self._og_search_title(webpage, default=None) or meta['title']
+        description = self._og_search_description(webpage, default=None) or meta.get('description')
+        duration = meta.get('length') or meta.get('lenght')
+        timestamp = parse_iso8601(meta.get('addDate'), ' ')
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'duration': duration,
+            'timestamp': timestamp,
+            'formats': formats,
+        }

youtube_dl/extractor/closertotruth.py

@@ -1,92 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-
-
-class CloserToTruthIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?closertotruth\.com/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'http://closertotruth.com/series/solutions-the-mind-body-problem#video-3688',
-        'info_dict': {
-            'id': '0_zof1ktre',
-            'display_id': 'solutions-the-mind-body-problem',
-            'ext': 'mov',
-            'title': 'Solutions to the Mind-Body Problem?',
-            'upload_date': '20140221',
-            'timestamp': 1392956007,
-            'uploader_id': 'CTTXML'
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://closertotruth.com/episodes/how-do-brains-work',
-        'info_dict': {
-            'id': '0_iuxai6g6',
-            'display_id': 'how-do-brains-work',
-            'ext': 'mov',
-            'title': 'How do Brains Work?',
-            'upload_date': '20140221',
-            'timestamp': 1392956024,
-            'uploader_id': 'CTTXML'
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://closertotruth.com/interviews/1725',
-        'info_dict': {
-            'id': '1725',
-            'title': 'AyaFr-002',
-        },
-        'playlist_mincount': 2,
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, display_id)
-
-        partner_id = self._search_regex(
-            r'<script[^>]+src=["\'].*?\b(?:partner_id|p)/(\d+)',
-            webpage, 'kaltura partner_id')
-
-        title = self._search_regex(
-            r'<title>(.+?)\s*\|\s*.+?</title>', webpage, 'video title')
-
-        select = self._search_regex(
-            r'(?s)<select[^>]+id="select-version"[^>]*>(.+?)</select>',
-            webpage, 'select version', default=None)
-        if select:
-            entry_ids = set()
-            entries = []
-            for mobj in re.finditer(
-                    r'<option[^>]+value=(["\'])(?P<id>[0-9a-z_]+)(?:#.+?)?\1[^>]*>(?P<title>[^<]+)',
-                    webpage):
-                entry_id = mobj.group('id')
-                if entry_id in entry_ids:
-                    continue
-                entry_ids.add(entry_id)
-                entries.append({
-                    '_type': 'url_transparent',
-                    'url': 'kaltura:%s:%s' % (partner_id, entry_id),
-                    'ie_key': 'Kaltura',
-                    'title': mobj.group('title'),
-                })
-            if entries:
-                return self.playlist_result(entries, display_id, title)
-
-        entry_id = self._search_regex(
-            r'<a[^>]+id=(["\'])embed-kaltura\1[^>]+data-kaltura=(["\'])(?P<id>[0-9a-z_]+)\2',
-            webpage, 'kaltura entry_id', group='id')
-
-        return {
-            '_type': 'url_transparent',
-            'display_id': display_id,
-            'url': 'kaltura:%s:%s' % (partner_id, entry_id),
-            'ie_key': 'Kaltura',
-            'title': title
-        }

youtube_dl/extractor/cloudy.py

@@ -6,6 +6,7 @@ import re
 from .common import InfoExtractor
 from ..compat import (
     compat_parse_qs,
+    compat_urllib_parse_urlencode,
     compat_HTTPError,
 )
 from ..utils import (
@@ -16,26 +17,37 @@ from ..utils import (
 
 
 class CloudyIE(InfoExtractor):
-    _IE_DESC = 'cloudy.ec'
+    _IE_DESC = 'cloudy.ec and videoraj.ch'
     _VALID_URL = r'''(?x)
-        https?://(?:www\.)?cloudy\.ec/
+        https?://(?:www\.)?(?P<host>cloudy\.ec|videoraj\.(?:ch|to))/
         (?:v/|embed\.php\?id=)
         (?P<id>[A-Za-z0-9]+)
         '''
-    _EMBED_URL = 'http://www.cloudy.ec/embed.php?id=%s'
-    _API_URL = 'http://www.cloudy.ec/api/player.api.php'
+    _EMBED_URL = 'http://www.%s/embed.php?id=%s'
+    _API_URL = 'http://www.%s/api/player.api.php?%s'
     _MAX_TRIES = 2
-    _TEST = {
-        'url': 'https://www.cloudy.ec/v/af511e2527aac',
-        'md5': '5cb253ace826a42f35b4740539bedf07',
-        'info_dict': {
-            'id': 'af511e2527aac',
-            'ext': 'flv',
-            'title': 'Funny Cats and Animals Compilation june 2013',
+    _TESTS = [
+        {
+            'url': 'https://www.cloudy.ec/v/af511e2527aac',
+            'md5': '5cb253ace826a42f35b4740539bedf07',
+            'info_dict': {
+                'id': 'af511e2527aac',
+                'ext': 'flv',
+                'title': 'Funny Cats and Animals Compilation june 2013',
+            }
+        },
+        {
+            'url': 'http://www.videoraj.to/v/47f399fd8bb60',
+            'md5': '7d0f8799d91efd4eda26587421c3c3b0',
+            'info_dict': {
+                'id': '47f399fd8bb60',
+                'ext': 'flv',
+                'title': 'Burning a New iPhone 5 with Gasoline - Will it Survive?',
+            }
         }
-    }
+    ]
 
-    def _extract_video(self, video_id, file_key, error_url=None, try_num=0):
+    def _extract_video(self, video_host, video_id, file_key, error_url=None, try_num=0):
 
         if try_num > self._MAX_TRIES - 1:
             raise ExtractorError('Unable to extract video URL', expected=True)
@@ -52,8 +64,9 @@ class CloudyIE(InfoExtractor):
                 'errorUrl': error_url,
             })
 
+        data_url = self._API_URL % (video_host, compat_urllib_parse_urlencode(form))
         player_data = self._download_webpage(
-            self._API_URL, video_id, 'Downloading player data', query=form)
+            data_url, video_id, 'Downloading player data')
         data = compat_parse_qs(player_data)
 
         try_num += 1
@@ -75,7 +88,7 @@ class CloudyIE(InfoExtractor):
             except ExtractorError as e:
                 if isinstance(e.cause, compat_HTTPError) and e.cause.code in [404, 410]:
                     self.report_warning('Invalid video URL, requesting another', video_id)
-                    return self._extract_video(video_id, file_key, video_url, try_num)
+                    return self._extract_video(video_host, video_id, file_key, video_url, try_num)
 
         return {
             'id': video_id,
@@ -85,13 +98,14 @@ class CloudyIE(InfoExtractor):
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
+        video_host = mobj.group('host')
         video_id = mobj.group('id')
 
-        url = self._EMBED_URL % video_id
+        url = self._EMBED_URL % (video_host, video_id)
         webpage = self._download_webpage(url, video_id)
 
         file_key = self._search_regex(
             [r'key\s*:\s*"([^"]+)"', r'filekey\s*=\s*"([^"]+)"'],
             webpage, 'file_key')
 
-        return self._extract_video(video_id, file_key)
+        return self._extract_video(video_host, video_id, file_key)

youtube_dl/extractor/cmt.py

@@ -1,7 +1,5 @@
 from __future__ import unicode_literals
-
 from .mtv import MTVIE
-from ..utils import ExtractorError
 
 
 class CMTIE(MTVIE):
@@ -18,27 +16,7 @@ class CMTIE(MTVIE):
             'title': 'Garth Brooks - "The Call (featuring Trisha Yearwood)"',
             'description': 'Blame It All On My Roots',
         },
-        'skip': 'Video not available',
-    }, {
-        'url': 'http://www.cmt.com/videos/misc/1504699/still-the-king-ep-109-in-3-minutes.jhtml#id=1739908',
-        'md5': 'e61a801ca4a183a466c08bd98dccbb1c',
-        'info_dict': {
-            'id': '1504699',
-            'ext': 'mp4',
-            'title': 'Still The King Ep. 109 in 3 Minutes',
-            'description': 'Relive or catch up with Still The King by watching this recap of season 1, episode 9. New episodes Sundays 9/8c.',
-            'timestamp': 1469421000.0,
-            'upload_date': '20160725',
-        },
     }, {
         'url': 'http://www.cmt.com/shows/party-down-south/party-down-south-ep-407-gone-girl/1738172/playlist/#id=1738172',
         'only_matching': True,
     }]
-
-    @classmethod
-    def _transform_rtmp_url(cls, rtmp_video_url):
-        if 'error_not_available.swf' in rtmp_video_url:
-            raise ExtractorError(
-                '%s said: video is not available' % cls.IE_NAME, expected=True)
-
-        return super(CMTIE, cls)._transform_rtmp_url(rtmp_video_url)

youtube_dl/extractor/cnn.py

@@ -3,12 +3,15 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from .turner import TurnerBaseIE
-from ..utils import url_basename
+from ..utils import (
+    int_or_none,
+    parse_duration,
+    url_basename,
+)
 
 
-class CNNIE(TurnerBaseIE):
-    _VALID_URL = r'''(?x)https?://(?:(?P<sub_domain>edition|www|money)\.)?cnn\.com/(?:video/(?:data/.+?|\?)/)?videos?/
+class CNNIE(InfoExtractor):
+    _VALID_URL = r'''(?x)https?://(?:(?:edition|www)\.)?cnn\.com/video/(?:data/.+?|\?)/
         (?P<path>.+?/(?P<title>[^/]+?)(?:\.(?:[a-z\-]+)|(?=&)))'''
 
     _TESTS = [{
@@ -22,7 +25,6 @@ class CNNIE(TurnerBaseIE):
             'duration': 135,
             'upload_date': '20130609',
         },
-        'expected_warnings': ['Failed to download m3u8 information'],
     }, {
         'url': 'http://edition.cnn.com/video/?/video/us/2013/08/21/sot-student-gives-epic-speech.georgia-institute-of-technology&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+rss%2Fcnn_topstories+%28RSS%3A+Top+Stories%29',
         'md5': 'b5cc60c60a3477d185af8f19a2a26f4e',
@@ -32,8 +34,7 @@ class CNNIE(TurnerBaseIE):
             'title': "Student's epic speech stuns new freshmen",
             'description': "A Georgia Tech student welcomes the incoming freshmen with an epic speech backed by music from \"2001: A Space Odyssey.\"",
             'upload_date': '20130821',
-        },
-        'expected_warnings': ['Failed to download m3u8 information'],
+        }
     }, {
         'url': 'http://www.cnn.com/video/data/2.0/video/living/2014/12/22/growing-america-nashville-salemtown-board-episode-1.hln.html',
         'md5': 'f14d02ebd264df951feb2400e2c25a1b',
@@ -43,61 +44,80 @@ class CNNIE(TurnerBaseIE):
             'title': 'Nashville Ep. 1: Hand crafted skateboards',
             'description': 'md5:e7223a503315c9f150acac52e76de086',
             'upload_date': '20141222',
-        },
-        'expected_warnings': ['Failed to download m3u8 information'],
-    }, {
-        'url': 'http://money.cnn.com/video/news/2016/08/19/netflix-stunning-stats.cnnmoney/index.html',
-        'md5': '52a515dc1b0f001cd82e4ceda32be9d1',
-        'info_dict': {
-            'id': '/video/news/2016/08/19/netflix-stunning-stats.cnnmoney',
-            'ext': 'mp4',
-            'title': '5 stunning stats about Netflix',
-            'description': 'Did you know that Netflix has more than 80 million members? Here are five facts about the online video distributor that you probably didn\'t know.',
-            'upload_date': '20160819',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
+        }
     }, {
         'url': 'http://cnn.com/video/?/video/politics/2015/03/27/pkg-arizona-senator-church-attendance-mandatory.ktvk',
         'only_matching': True,
     }, {
         'url': 'http://cnn.com/video/?/video/us/2015/04/06/dnt-baker-refuses-anti-gay-order.wkmg',
         'only_matching': True,
-    }, {
-        'url': 'http://edition.cnn.com/videos/arts/2016/04/21/olympic-games-cultural-a-z-brazil.cnn',
-        'only_matching': True,
     }]
 
-    _CONFIG = {
-        # http://edition.cnn.com/.element/apps/cvp/3.0/cfg/spider/cnn/expansion/config.xml
-        'edition': {
-            'data_src': 'http://edition.cnn.com/video/data/3.0/video/%s/index.xml',
-            'media_src': 'http://pmd.cdn.turner.com/cnn/big',
-        },
-        # http://money.cnn.com/.element/apps/cvp2/cfg/config.xml
-        'money': {
-            'data_src': 'http://money.cnn.com/video/data/4.0/video/%s.xml',
-            'media_src': 'http://ht3.cdn.turner.com/money/big',
-        },
-    }
-
-    def _extract_timestamp(self, video_data):
-        # TODO: fix timestamp extraction
-        return None
-
     def _real_extract(self, url):
-        sub_domain, path, page_title = re.match(self._VALID_URL, url).groups()
-        if sub_domain not in ('money', 'edition'):
-            sub_domain = 'edition'
-        config = self._CONFIG[sub_domain]
-        return self._extract_cvp_info(
-            config['data_src'] % path, page_title, {
-                'default': {
-                    'media_src': config['media_src'],
-                }
-            })
+        mobj = re.match(self._VALID_URL, url)
+        path = mobj.group('path')
+        page_title = mobj.group('title')
+        info_url = 'http://edition.cnn.com/video/data/3.0/%s/index.xml' % path
+        info = self._download_xml(info_url, page_title)
+
+        formats = []
+        rex = re.compile(r'''(?x)
+            (?P<width>[0-9]+)x(?P<height>[0-9]+)
+            (?:_(?P<bitrate>[0-9]+)k)?
+        ''')
+        for f in info.findall('files/file'):
+            video_url = 'http://ht.cdn.turner.com/cnn/big%s' % (f.text.strip())
+            fdct = {
+                'format_id': f.attrib['bitrate'],
+                'url': video_url,
+            }
+
+            mf = rex.match(f.attrib['bitrate'])
+            if mf:
+                fdct['width'] = int(mf.group('width'))
+                fdct['height'] = int(mf.group('height'))
+                fdct['tbr'] = int_or_none(mf.group('bitrate'))
+            else:
+                mf = rex.search(f.text)
+                if mf:
+                    fdct['width'] = int(mf.group('width'))
+                    fdct['height'] = int(mf.group('height'))
+                    fdct['tbr'] = int_or_none(mf.group('bitrate'))
+                else:
+                    mi = re.match(r'ios_(audio|[0-9]+)$', f.attrib['bitrate'])
+                    if mi:
+                        if mi.group(1) == 'audio':
+                            fdct['vcodec'] = 'none'
+                            fdct['ext'] = 'm4a'
+                        else:
+                            fdct['tbr'] = int(mi.group(1))
+
+            formats.append(fdct)
+
+        self._sort_formats(formats)
+
+        thumbnails = [{
+            'height': int(t.attrib['height']),
+            'width': int(t.attrib['width']),
+            'url': t.text,
+        } for t in info.findall('images/image')]
+
+        metas_el = info.find('metas')
+        upload_date = (
+            metas_el.attrib.get('version') if metas_el is not None else None)
+
+        duration_el = info.find('length')
+        duration = parse_duration(duration_el.text)
+
+        return {
+            'id': info.attrib['id'],
+            'title': info.find('headline').text,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'description': info.find('description').text,
+            'duration': duration,
+            'upload_date': upload_date,
+        }
 
 
 class CNNBlogsIE(InfoExtractor):
@@ -112,7 +132,6 @@ class CNNBlogsIE(InfoExtractor):
             'description': 'Glenn Greenwald responds to comments made this week on Capitol Hill that journalists could be criminal accessories.',
             'upload_date': '20140209',
         },
-        'expected_warnings': ['Failed to download m3u8 information'],
         'add_ie': ['CNN'],
     }
 
@@ -127,7 +146,7 @@ class CNNBlogsIE(InfoExtractor):
 
 
 class CNNArticleIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:edition|www)\.)?cnn\.com/(?!videos?/)'
+    _VALID_URL = r'https?://(?:(?:edition|www)\.)?cnn\.com/(?!video/)'
     _TEST = {
         'url': 'http://www.cnn.com/2014/12/21/politics/obama-north-koreas-hack-not-war-but-cyber-vandalism/',
         'md5': '689034c2a3d9c6dc4aa72d65a81efd01',
@@ -135,10 +154,9 @@ class CNNArticleIE(InfoExtractor):
             'id': 'bestoftv/2014/12/21/ip-north-korea-obama.cnn',
             'ext': 'mp4',
             'title': 'Obama: Cyberattack not an act of war',
-            'description': 'md5:0a802a40d2376f60e6b04c8d5bcebc4b',
+            'description': 'md5:51ce6750450603795cad0cdfbd7d05c5',
             'upload_date': '20141221',
         },
-        'expected_warnings': ['Failed to download m3u8 information'],
         'add_ie': ['CNN'],
     }
 

youtube_dl/extractor/comedycentral.py

@@ -1,7 +1,17 @@
 from __future__ import unicode_literals
 
+import re
+
 from .mtv import MTVServicesInfoExtractor
-from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urllib_parse_urlencode,
+)
+from ..utils import (
+    ExtractorError,
+    float_or_none,
+    unified_strdate,
+)
 
 
 class ComedyCentralIE(MTVServicesInfoExtractor):
@@ -16,10 +26,8 @@ class ComedyCentralIE(MTVServicesInfoExtractor):
         'info_dict': {
             'id': 'cef0cbb3-e776-4bc9-b62e-8016deccb354',
             'ext': 'mp4',
-            'title': 'CC:Stand-Up|August 18, 2013|1|0101|Uncensored - Too Good of a Mother',
+            'title': 'CC:Stand-Up|Greg Fitzsimmons: Life on Stage|Uncensored - Too Good of a Mother',
             'description': 'After a certain point, breastfeeding becomes c**kblocking.',
-            'timestamp': 1376798400,
-            'upload_date': '20130818',
         },
     }, {
         'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/interviews/6yx39d/exclusive-rand-paul-extended-interview',
@@ -27,92 +35,241 @@ class ComedyCentralIE(MTVServicesInfoExtractor):
     }]
 
 
-class ToshIE(MTVServicesInfoExtractor):
-    IE_DESC = 'Tosh.0'
-    _VALID_URL = r'^https?://tosh\.cc\.com/video-(?:clips|collections)/[^/]+/(?P<videotitle>[^/?#]+)'
-    _FEED_URL = 'http://tosh.cc.com/feeds/mrss'
-
+class ComedyCentralShowsIE(MTVServicesInfoExtractor):
+    IE_DESC = 'The Daily Show / The Colbert Report'
+    # urls can be abbreviations like :thedailyshow
+    # urls for episodes like:
+    # or urls for clips like: http://www.thedailyshow.com/watch/mon-december-10-2012/any-given-gun-day
+    #                     or: http://www.colbertnation.com/the-colbert-report-videos/421667/november-29-2012/moon-shattering-news
+    #                     or: http://www.colbertnation.com/the-colbert-report-collections/422008/festival-of-lights/79524
+    _VALID_URL = r'''(?x)^(:(?P<shortname>tds|thedailyshow)
+                      |https?://(:www\.)?
+                          (?P<showname>thedailyshow|thecolbertreport|tosh)\.(?:cc\.)?com/
+                         ((?:full-)?episodes/(?:[0-9a-z]{6}/)?(?P<episode>.*)|
+                          (?P<clip>
+                              (?:(?:guests/[^/]+|videos|video-(?:clips|playlists)|special-editions|news-team/[^/]+)/[^/]+/(?P<videotitle>[^/?#]+))
+                              |(the-colbert-report-(videos|collections)/(?P<clipID>[0-9]+)/[^/]*/(?P<cntitle>.*?))
+                              |(watch/(?P<date>[^/]*)/(?P<tdstitle>.*))
+                          )|
+                          (?P<interview>
+                              extended-interviews/(?P<interID>[0-9a-z]+)/
+                              (?:playlist_tds_extended_)?(?P<interview_title>[^/?#]*?)
+                              (?:/[^/?#]?|[?#]|$))))
+                     '''
     _TESTS = [{
-        'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
+        'url': 'http://thedailyshow.cc.com/watch/thu-december-13-2012/kristen-stewart',
+        'md5': '4e2f5cb088a83cd8cdb7756132f9739d',
         'info_dict': {
-            'description': 'Tosh asked fans to share their summer plans.',
-            'title': 'Twitter Users Share Summer Plans',
+            'id': 'ab9ab3e7-5a98-4dbe-8b21-551dc0523d55',
+            'ext': 'mp4',
+            'upload_date': '20121213',
+            'description': 'Kristen Stewart learns to let loose in "On the Road."',
+            'uploader': 'thedailyshow',
+            'title': 'thedailyshow kristen-stewart part 1',
+        }
+    }, {
+        'url': 'http://thedailyshow.cc.com/extended-interviews/b6364d/sarah-chayes-extended-interview',
+        'info_dict': {
+            'id': 'sarah-chayes-extended-interview',
+            'description': 'Carnegie Endowment Senior Associate Sarah Chayes discusses how corrupt institutions function throughout the world in her book "Thieves of State: Why Corruption Threatens Global Security."',
+            'title': 'thedailyshow Sarah Chayes Extended Interview',
         },
-        'playlist': [{
-            'md5': 'f269e88114c1805bb6d7653fecea9e06',
-            'info_dict': {
-                'id': '90498ec2-ed00-11e0-aca6-0026b9414f30',
-                'ext': 'mp4',
-                'title': 'Tosh.0|June 9, 2077|2|211|Twitter Users Share Summer Plans',
-                'description': 'Tosh asked fans to share their summer plans.',
-                'thumbnail': 're:^https?://.*\.jpg',
-                # It's really reported to be published on year 2077
-                'upload_date': '20770610',
-                'timestamp': 3390510600,
-                'subtitles': {
-                    'en': 'mincount:3',
+        'playlist': [
+            {
+                'info_dict': {
+                    'id': '0baad492-cbec-4ec1-9e50-ad91c291127f',
+                    'ext': 'mp4',
+                    'upload_date': '20150129',
+                    'description': 'Carnegie Endowment Senior Associate Sarah Chayes discusses how corrupt institutions function throughout the world in her book "Thieves of State: Why Corruption Threatens Global Security."',
+                    'uploader': 'thedailyshow',
+                    'title': 'thedailyshow sarah-chayes-extended-interview part 1',
                 },
             },
-        }]
-    }, {
-        'url': 'http://tosh.cc.com/video-collections/x2iz7k/just-plain-foul/m5q4fp',
-        'only_matching': True,
-    }]
-
-    @classmethod
-    def _transform_rtmp_url(cls, rtmp_video_url):
-        new_urls = super(ToshIE, cls)._transform_rtmp_url(rtmp_video_url)
-        new_urls['rtmp'] = rtmp_video_url.replace('viacomccstrm', 'viacommtvstrm')
-        return new_urls
-
-
-class ComedyCentralTVIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/(?:staffeln|shows)/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'http://www.comedycentral.tv/staffeln/7436-the-mindy-project-staffel-4',
-        'info_dict': {
-            'id': 'local_playlist-f99b626bdfe13568579a',
-            'ext': 'flv',
-            'title': 'Episode_the-mindy-project_shows_season-4_episode-3_full-episode_part1',
-        },
+            {
+                'info_dict': {
+                    'id': '1e4fb91b-8ce7-4277-bd7c-98c9f1bbd283',
+                    'ext': 'mp4',
+                    'upload_date': '20150129',
+                    'description': 'Carnegie Endowment Senior Associate Sarah Chayes discusses how corrupt institutions function throughout the world in her book "Thieves of State: Why Corruption Threatens Global Security."',
+                    'uploader': 'thedailyshow',
+                    'title': 'thedailyshow sarah-chayes-extended-interview part 2',
+                },
+            },
+        ],
         'params': {
-            # rtmp download
             'skip_download': True,
         },
     }, {
-        'url': 'http://www.comedycentral.tv/shows/1074-workaholics',
+        'url': 'http://thedailyshow.cc.com/extended-interviews/xm3fnq/andrew-napolitano-extended-interview',
         'only_matching': True,
     }, {
-        'url': 'http://www.comedycentral.tv/shows/1727-the-mindy-project/bonus',
+        'url': 'http://thecolbertreport.cc.com/videos/29w6fx/-realhumanpraise-for-fox-news',
+        'only_matching': True,
+    }, {
+        'url': 'http://thecolbertreport.cc.com/videos/gh6urb/neil-degrasse-tyson-pt--1?xrs=eml_col_031114',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/guests/michael-lewis/3efna8/exclusive---michael-lewis-extended-interview-pt--3',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/episodes/sy7yv0/april-8--2014---denis-leary',
+        'only_matching': True,
+    }, {
+        'url': 'http://thecolbertreport.cc.com/episodes/8ase07/april-8--2014---jane-goodall',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/video-playlists/npde3s/the-daily-show-19088-highlights',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/video-playlists/t6d9sg/the-daily-show-20038-highlights/be3cwo',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/special-editions/2l8fdb/special-edition---a-look-back-at-food',
+        'only_matching': True,
+    }, {
+        'url': 'http://thedailyshow.cc.com/news-team/michael-che/7wnfel/we-need-to-talk-about-israel',
+        'only_matching': True,
+    }, {
+        'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
         'only_matching': True,
     }]
 
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
+    _available_formats = ['3500', '2200', '1700', '1200', '750', '400']
 
-        webpage = self._download_webpage(url, video_id)
-
-        mrss_url = self._search_regex(
-            r'data-mrss=(["\'])(?P<url>(?:(?!\1).)+)\1',
-            webpage, 'mrss url', group='url')
-
-        return self._get_videos_info_from_url(mrss_url, video_id)
-
-
-class ComedyCentralShortnameIE(InfoExtractor):
-    _VALID_URL = r'^:(?P<id>tds|thedailyshow)$'
-    _TESTS = [{
-        'url': ':tds',
-        'only_matching': True,
-    }, {
-        'url': ':thedailyshow',
-        'only_matching': True,
-    }]
+    _video_extensions = {
+        '3500': 'mp4',
+        '2200': 'mp4',
+        '1700': 'mp4',
+        '1200': 'mp4',
+        '750': 'mp4',
+        '400': 'mp4',
+    }
+    _video_dimensions = {
+        '3500': (1280, 720),
+        '2200': (960, 540),
+        '1700': (768, 432),
+        '1200': (640, 360),
+        '750': (512, 288),
+        '400': (384, 216),
+    }
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
-        shortcut_map = {
-            'tds': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'thedailyshow': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
+        mobj = re.match(self._VALID_URL, url)
+
+        if mobj.group('shortname'):
+            return self.url_result('http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes')
+
+        if mobj.group('clip'):
+            if mobj.group('videotitle'):
+                epTitle = mobj.group('videotitle')
+            elif mobj.group('showname') == 'thedailyshow':
+                epTitle = mobj.group('tdstitle')
+            else:
+                epTitle = mobj.group('cntitle')
+            dlNewest = False
+        elif mobj.group('interview'):
+            epTitle = mobj.group('interview_title')
+            dlNewest = False
+        else:
+            dlNewest = not mobj.group('episode')
+            if dlNewest:
+                epTitle = mobj.group('showname')
+            else:
+                epTitle = mobj.group('episode')
+        show_name = mobj.group('showname')
+
+        webpage, htmlHandle = self._download_webpage_handle(url, epTitle)
+        if dlNewest:
+            url = htmlHandle.geturl()
+            mobj = re.match(self._VALID_URL, url, re.VERBOSE)
+            if mobj is None:
+                raise ExtractorError('Invalid redirected URL: ' + url)
+            if mobj.group('episode') == '':
+                raise ExtractorError('Redirected URL is still not specific: ' + url)
+            epTitle = (mobj.group('episode') or mobj.group('videotitle')).rpartition('/')[-1]
+
+        mMovieParams = re.findall('(?:<param name="movie" value="|var url = ")(http://media.mtvnservices.com/([^"]*(?:episode|video).*?:.*?))"', webpage)
+        if len(mMovieParams) == 0:
+            # The Colbert Report embeds the information in a without
+            # a URL prefix; so extract the alternate reference
+            # and then add the URL prefix manually.
+
+            altMovieParams = re.findall('data-mgid="([^"]*(?:episode|video|playlist).*?:.*?)"', webpage)
+            if len(altMovieParams) == 0:
+                raise ExtractorError('unable to find Flash URL in webpage ' + url)
+            else:
+                mMovieParams = [('http://media.mtvnservices.com/' + altMovieParams[0], altMovieParams[0])]
+
+        uri = mMovieParams[0][1]
+        # Correct cc.com in uri
+        uri = re.sub(r'(episode:[^.]+)(\.cc)?\.com', r'\1.com', uri)
+
+        index_url = 'http://%s.cc.com/feeds/mrss?%s' % (show_name, compat_urllib_parse_urlencode({'uri': uri}))
+        idoc = self._download_xml(
+            index_url, epTitle,
+            'Downloading show index', 'Unable to download episode index')
+
+        title = idoc.find('./channel/title').text
+        description = idoc.find('./channel/description').text
+
+        entries = []
+        item_els = idoc.findall('.//item')
+        for part_num, itemEl in enumerate(item_els):
+            upload_date = unified_strdate(itemEl.findall('./pubDate')[0].text)
+            thumbnail = itemEl.find('.//{http://search.yahoo.com/mrss/}thumbnail').attrib.get('url')
+
+            content = itemEl.find('.//{http://search.yahoo.com/mrss/}content')
+            duration = float_or_none(content.attrib.get('duration'))
+            mediagen_url = content.attrib['url']
+            guid = itemEl.find('./guid').text.rpartition(':')[-1]
+
+            cdoc = self._download_xml(
+                mediagen_url, epTitle,
+                'Downloading configuration for segment %d / %d' % (part_num + 1, len(item_els)))
+
+            turls = []
+            for rendition in cdoc.findall('.//rendition'):
+                finfo = (rendition.attrib['bitrate'], rendition.findall('./src')[0].text)
+                turls.append(finfo)
+
+            formats = []
+            for format, rtmp_video_url in turls:
+                w, h = self._video_dimensions.get(format, (None, None))
+                formats.append({
+                    'format_id': 'vhttp-%s' % format,
+                    'url': self._transform_rtmp_url(rtmp_video_url),
+                    'ext': self._video_extensions.get(format, 'mp4'),
+                    'height': h,
+                    'width': w,
+                })
+                formats.append({
+                    'format_id': 'rtmp-%s' % format,
+                    'url': rtmp_video_url.replace('viacomccstrm', 'viacommtvstrm'),
+                    'ext': self._video_extensions.get(format, 'mp4'),
+                    'height': h,
+                    'width': w,
+                })
+                self._sort_formats(formats)
+
+            subtitles = self._extract_subtitles(cdoc, guid)
+
+            virtual_id = show_name + ' ' + epTitle + ' part ' + compat_str(part_num + 1)
+            entries.append({
+                'id': guid,
+                'title': virtual_id,
+                'formats': formats,
+                'uploader': show_name,
+                'upload_date': upload_date,
+                'duration': duration,
+                'thumbnail': thumbnail,
+                'description': description,
+                'subtitles': subtitles,
+            })
+
+        return {
+            '_type': 'playlist',
+            'id': epTitle,
+            'entries': entries,
+            'title': show_name + ' ' + title,
+            'description': description,
         }
-        return self.url_result(shortcut_map[video_id])

youtube_dl/extractor/common.py

@@ -44,9 +44,7 @@ from ..utils import (
     sanitized_Request,
     unescapeHTML,
     unified_strdate,
-    unified_timestamp,
     url_basename,
-    xpath_element,
     xpath_text,
     xpath_with_ns,
     determine_protocol,
@@ -54,9 +52,6 @@ from ..utils import (
     mimetype2ext,
     update_Request,
     update_url_query,
-    parse_m3u8_attributes,
-    extract_attributes,
-    parse_codecs,
 )
 
 
@@ -164,7 +159,6 @@ class InfoExtractor(object):
                         * "height" (optional, int)
                         * "resolution" (optional, string "{width}x{height"},
                                         deprecated)
-                        * "filesize" (optional, int)
     thumbnail:      Full URL to a video thumbnail image.
     description:    Full video description.
     uploader:       Full name of the video uploader.
@@ -662,24 +656,6 @@ class InfoExtractor(object):
         else:
             return res
 
-    def _get_netrc_login_info(self, netrc_machine=None):
-        username = None
-        password = None
-        netrc_machine = netrc_machine or self._NETRC_MACHINE
-
-        if self._downloader.params.get('usenetrc', False):
-            try:
-                info = netrc.netrc().authenticators(netrc_machine)
-                if info is not None:
-                    username = info[0]
-                    password = info[2]
-                else:
-                    raise netrc.NetrcParseError('No authenticators for %s' % netrc_machine)
-            except (IOError, netrc.NetrcParseError) as err:
-                self._downloader.report_warning('parsing .netrc: %s' % error_to_compat_str(err))
-
-        return (username, password)
-
     def _get_login_info(self):
         """
         Get the login info as (username, password)
@@ -697,8 +673,16 @@ class InfoExtractor(object):
         if downloader_params.get('username') is not None:
             username = downloader_params['username']
             password = downloader_params['password']
-        else:
-            username, password = self._get_netrc_login_info()
+        elif downloader_params.get('usenetrc', False):
+            try:
+                info = netrc.netrc().authenticators(self._NETRC_MACHINE)
+                if info is not None:
+                    username = info[0]
+                    password = info[2]
+                else:
+                    raise netrc.NetrcParseError('No authenticators for %s' % self._NETRC_MACHINE)
+            except (IOError, netrc.NetrcParseError) as err:
+                self._downloader.report_warning('parsing .netrc: %s' % error_to_compat_str(err))
 
         return (username, password)
 
@@ -737,14 +721,9 @@ class InfoExtractor(object):
                     [^>]+?content=(["\'])(?P<content>.*?)\2''' % re.escape(prop)
 
     def _og_search_property(self, prop, html, name=None, **kargs):
-        if not isinstance(prop, (list, tuple)):
-            prop = [prop]
         if name is None:
-            name = 'OpenGraph %s' % prop[0]
-        og_regexes = []
-        for p in prop:
-            og_regexes.extend(self._og_regexes(p))
-        escaped = self._search_regex(og_regexes, html, name, flags=re.DOTALL, **kargs)
+            name = 'OpenGraph %s' % prop
+        escaped = self._search_regex(self._og_regexes(prop), html, name, flags=re.DOTALL, **kargs)
         if escaped is None:
             return None
         return unescapeHTML(escaped)
@@ -768,12 +747,10 @@ class InfoExtractor(object):
         return self._og_search_property('url', html, **kargs)
 
     def _html_search_meta(self, name, html, display_name=None, fatal=False, **kwargs):
-        if not isinstance(name, (list, tuple)):
-            name = [name]
         if display_name is None:
-            display_name = name[0]
+            display_name = name
         return self._html_search_regex(
-            [self._meta_regex(n) for n in name],
+            self._meta_regex(name),
             html, display_name, fatal=fatal, group='content', **kwargs)
 
     def _dc_search_uploader(self, html):
@@ -822,66 +799,40 @@ class InfoExtractor(object):
         return self._html_search_meta('twitter:player', html,
                                       'twitter card player')
 
-    def _search_json_ld(self, html, video_id, expected_type=None, **kwargs):
+    def _search_json_ld(self, html, video_id, **kwargs):
         json_ld = self._search_regex(
             r'(?s)<script[^>]+type=(["\'])application/ld\+json\1[^>]*>(?P<json_ld>.+?)</script>',
             html, 'JSON-LD', group='json_ld', **kwargs)
-        default = kwargs.get('default', NO_DEFAULT)
         if not json_ld:
-            return default if default is not NO_DEFAULT else {}
-        # JSON-LD may be malformed and thus `fatal` should be respected.
-        # At the same time `default` may be passed that assumes `fatal=False`
-        # for _search_regex. Let's simulate the same behavior here as well.
-        fatal = kwargs.get('fatal', True) if default == NO_DEFAULT else False
-        return self._json_ld(json_ld, video_id, fatal=fatal, expected_type=expected_type)
+            return {}
+        return self._json_ld(json_ld, video_id, fatal=kwargs.get('fatal', True))
 
-    def _json_ld(self, json_ld, video_id, fatal=True, expected_type=None):
+    def _json_ld(self, json_ld, video_id, fatal=True):
         if isinstance(json_ld, compat_str):
             json_ld = self._parse_json(json_ld, video_id, fatal=fatal)
         if not json_ld:
             return {}
         info = {}
-        if not isinstance(json_ld, (list, tuple, dict)):
-            return info
-        if isinstance(json_ld, dict):
-            json_ld = [json_ld]
-        for e in json_ld:
-            if e.get('@context') == 'http://schema.org':
-                item_type = e.get('@type')
-                if expected_type is not None and expected_type != item_type:
-                    return info
-                if item_type == 'TVEpisode':
-                    info.update({
-                        'episode': unescapeHTML(e.get('name')),
-                        'episode_number': int_or_none(e.get('episodeNumber')),
-                        'description': unescapeHTML(e.get('description')),
-                    })
-                    part_of_season = e.get('partOfSeason')
-                    if isinstance(part_of_season, dict) and part_of_season.get('@type') == 'TVSeason':
-                        info['season_number'] = int_or_none(part_of_season.get('seasonNumber'))
-                    part_of_series = e.get('partOfSeries') or e.get('partOfTVSeries')
-                    if isinstance(part_of_series, dict) and part_of_series.get('@type') == 'TVSeries':
-                        info['series'] = unescapeHTML(part_of_series.get('name'))
-                elif item_type == 'Article':
-                    info.update({
-                        'timestamp': parse_iso8601(e.get('datePublished')),
-                        'title': unescapeHTML(e.get('headline')),
-                        'description': unescapeHTML(e.get('articleBody')),
-                    })
-                elif item_type == 'VideoObject':
-                    info.update({
-                        'url': e.get('contentUrl'),
-                        'title': unescapeHTML(e.get('name')),
-                        'description': unescapeHTML(e.get('description')),
-                        'thumbnail': e.get('thumbnailUrl'),
-                        'duration': parse_duration(e.get('duration')),
-                        'timestamp': unified_timestamp(e.get('uploadDate')),
-                        'filesize': float_or_none(e.get('contentSize')),
-                        'tbr': int_or_none(e.get('bitrate')),
-                        'width': int_or_none(e.get('width')),
-                        'height': int_or_none(e.get('height')),
-                    })
-                break
+        if json_ld.get('@context') == 'http://schema.org':
+            item_type = json_ld.get('@type')
+            if item_type == 'TVEpisode':
+                info.update({
+                    'episode': unescapeHTML(json_ld.get('name')),
+                    'episode_number': int_or_none(json_ld.get('episodeNumber')),
+                    'description': unescapeHTML(json_ld.get('description')),
+                })
+                part_of_season = json_ld.get('partOfSeason')
+                if isinstance(part_of_season, dict) and part_of_season.get('@type') == 'TVSeason':
+                    info['season_number'] = int_or_none(part_of_season.get('seasonNumber'))
+                part_of_series = json_ld.get('partOfSeries')
+                if isinstance(part_of_series, dict) and part_of_series.get('@type') == 'TVSeries':
+                    info['series'] = unescapeHTML(part_of_series.get('name'))
+            elif item_type == 'Article':
+                info.update({
+                    'timestamp': parse_iso8601(json_ld.get('datePublished')),
+                    'title': unescapeHTML(json_ld.get('headline')),
+                    'description': unescapeHTML(json_ld.get('articleBody')),
+                })
         return dict((k, v) for k, v in info.items() if v is not None)
 
     @staticmethod
@@ -923,11 +874,7 @@ class InfoExtractor(object):
                 f['ext'] = determine_ext(f['url'])
 
             if isinstance(field_preference, (list, tuple)):
-                return tuple(
-                    f.get(field)
-                    if f.get(field) is not None
-                    else ('' if field == 'format_id' else -1)
-                    for field in field_preference)
+                return tuple(f.get(field) if f.get(field) is not None else -1 for field in field_preference)
 
             preference = f.get('preference')
             if preference is None:
@@ -935,8 +882,7 @@ class InfoExtractor(object):
                 if f.get('ext') in ['f4f', 'f4m']:  # Not yet supported
                     preference -= 0.5
 
-            protocol = f.get('protocol') or determine_protocol(f)
-            proto_preference = 0 if protocol in ['http', 'https'] else (-0.5 if protocol == 'rtsp' else -0.1)
+            proto_preference = 0 if determine_protocol(f) in ['http', 'https'] else -0.1
 
             if f.get('vcodec') == 'none':  # audio only
                 preference -= 50
@@ -1084,7 +1030,7 @@ class InfoExtractor(object):
         if base_url:
             base_url = base_url.strip()
 
-        bootstrap_info = xpath_element(
+        bootstrap_info = xpath_text(
             manifest, ['{http://ns.adobe.com/f4m/1.0}bootstrapInfo', '{http://ns.adobe.com/f4m/2.0}bootstrapInfo'],
             'bootstrap info', default=None)
 
@@ -1139,7 +1085,7 @@ class InfoExtractor(object):
             formats.append({
                 'format_id': format_id,
                 'url': manifest_url,
-                'ext': 'flv' if bootstrap_info is not None else None,
+                'ext': 'flv' if bootstrap_info else None,
                 'tbr': tbr,
                 'width': width,
                 'height': height,
@@ -1153,7 +1099,7 @@ class InfoExtractor(object):
             'url': m3u8_url,
             'ext': ext,
             'protocol': 'm3u8',
-            'preference': preference - 100 if preference else -100,
+            'preference': preference - 1 if preference else -1,
             'resolution': 'multiple',
             'format_note': 'Quality selection URL',
         }
@@ -1163,6 +1109,13 @@ class InfoExtractor(object):
                               m3u8_id=None, note=None, errnote=None,
                               fatal=True, live=False):
 
+        formats = [self._m3u8_meta_format(m3u8_url, ext, preference, m3u8_id)]
+
+        format_url = lambda u: (
+            u
+            if re.match(r'^https?://', u)
+            else compat_urlparse.urljoin(m3u8_url, u))
+
         res = self._download_webpage_handle(
             m3u8_url, video_id,
             note=note or 'Downloading m3u8 information',
@@ -1173,13 +1126,6 @@ class InfoExtractor(object):
         m3u8_doc, urlh = res
         m3u8_url = urlh.geturl()
 
-        formats = [self._m3u8_meta_format(m3u8_url, ext, preference, m3u8_id)]
-
-        format_url = lambda u: (
-            u
-            if re.match(r'^https?://', u)
-            else compat_urlparse.urljoin(m3u8_url, u))
-
         # We should try extracting formats only from master playlists [1], i.e.
         # playlists that describe available qualities. On the other hand media
         # playlists [2] should be returned as is since they contain just the media
@@ -1201,44 +1147,39 @@ class InfoExtractor(object):
                 'protocol': entry_protocol,
                 'preference': preference,
             }]
-        last_info = {}
-        last_media = {}
+        last_info = None
+        last_media = None
+        kv_rex = re.compile(
+            r'(?P<key>[a-zA-Z_-]+)=(?P<val>"[^"]+"|[^",]+)(?:,|$)')
         for line in m3u8_doc.splitlines():
             if line.startswith('#EXT-X-STREAM-INF:'):
-                last_info = parse_m3u8_attributes(line)
+                last_info = {}
+                for m in kv_rex.finditer(line):
+                    v = m.group('val')
+                    if v.startswith('"'):
+                        v = v[1:-1]
+                    last_info[m.group('key')] = v
             elif line.startswith('#EXT-X-MEDIA:'):
-                media = parse_m3u8_attributes(line)
-                media_type = media.get('TYPE')
-                if media_type in ('VIDEO', 'AUDIO'):
-                    media_url = media.get('URI')
-                    if media_url:
-                        format_id = []
-                        for v in (media.get('GROUP-ID'), media.get('NAME')):
-                            if v:
-                                format_id.append(v)
-                        formats.append({
-                            'format_id': '-'.join(format_id),
-                            'url': format_url(media_url),
-                            'language': media.get('LANGUAGE'),
-                            'vcodec': 'none' if media_type == 'AUDIO' else None,
-                            'ext': ext,
-                            'protocol': entry_protocol,
-                            'preference': preference,
-                        })
-                    else:
-                        # When there is no URI in EXT-X-MEDIA let this tag's
-                        # data be used by regular URI lines below
-                        last_media = media
+                last_media = {}
+                for m in kv_rex.finditer(line):
+                    v = m.group('val')
+                    if v.startswith('"'):
+                        v = v[1:-1]
+                    last_media[m.group('key')] = v
             elif line.startswith('#') or not line.strip():
                 continue
             else:
-                tbr = int_or_none(last_info.get('AVERAGE-BANDWIDTH') or last_info.get('BANDWIDTH'), scale=1000)
+                if last_info is None:
+                    formats.append({'url': format_url(line)})
+                    continue
+                tbr = int_or_none(last_info.get('BANDWIDTH'), scale=1000)
                 format_id = []
                 if m3u8_id:
                     format_id.append(m3u8_id)
+                last_media_name = last_media.get('NAME') if last_media and last_media.get('TYPE') not in ('SUBTITLES', 'CLOSED-CAPTIONS') else None
                 # Despite specification does not mention NAME attribute for
                 # EXT-X-STREAM-INF it still sometimes may be present
-                stream_name = last_info.get('NAME') or last_media.get('NAME')
+                stream_name = last_info.get('NAME') or last_media_name
                 # Bandwidth of live streams may differ over time thus making
                 # format_id unpredictable. So it's better to keep provided
                 # format_id intact.
@@ -1249,7 +1190,6 @@ class InfoExtractor(object):
                     'url': format_url(line.strip()),
                     'tbr': tbr,
                     'ext': ext,
-                    'fps': float_or_none(last_info.get('FRAME-RATE')),
                     'protocol': entry_protocol,
                     'preference': preference,
                 }
@@ -1258,20 +1198,29 @@ class InfoExtractor(object):
                     width_str, height_str = resolution.split('x')
                     f['width'] = int(width_str)
                     f['height'] = int(height_str)
-                # Unified Streaming Platform
-                mobj = re.search(
-                    r'audio.*?(?:%3D|=)(\d+)(?:-video.*?(?:%3D|=)(\d+))?', f['url'])
-                if mobj:
-                    abr, vbr = mobj.groups()
-                    abr, vbr = float_or_none(abr, 1000), float_or_none(vbr, 1000)
+                codecs = last_info.get('CODECS')
+                if codecs:
+                    vcodec, acodec = [None] * 2
+                    va_codecs = codecs.split(',')
+                    if len(va_codecs) == 1:
+                        # Audio only entries usually come with single codec and
+                        # no resolution. For more robustness we also check it to
+                        # be mp4 audio.
+                        if not resolution and va_codecs[0].startswith('mp4a'):
+                            vcodec, acodec = 'none', va_codecs[0]
+                        else:
+                            vcodec = va_codecs[0]
+                    else:
+                        vcodec, acodec = va_codecs[:2]
                     f.update({
-                        'vbr': vbr,
-                        'abr': abr,
+                        'acodec': acodec,
+                        'vcodec': vcodec,
                     })
-                f.update(parse_codecs(last_info.get('CODECS')))
+                if last_media is not None:
+                    f['m3u8_media'] = last_media
+                    last_media = None
                 formats.append(f)
                 last_info = {}
-                last_media = {}
         return formats
 
     @staticmethod
@@ -1521,13 +1470,6 @@ class InfoExtractor(object):
             compat_etree_fromstring(mpd.encode('utf-8')), mpd_id, mpd_base_url, formats_dict=formats_dict)
 
     def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', formats_dict={}):
-        """
-        Parse formats from MPD manifest.
-        References:
-         1. MPEG-DASH Standard, ISO/IEC 23009-1:2014(E),
-            http://standards.iso.org/ittf/PubliclyAvailableStandards/c065274_ISO_IEC_23009-1_2014.zip
-         2. https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP
-        """
         if mpd_doc.get('type') == 'dynamic':
             return []
 
@@ -1560,16 +1502,8 @@ class InfoExtractor(object):
                         s_e = segment_timeline.findall(_add_ns('S'))
                         if s_e:
                             ms_info['total_number'] = 0
-                            ms_info['s'] = []
                             for s in s_e:
-                                r = int(s.get('r', 0))
-                                ms_info['total_number'] += 1 + r
-                                ms_info['s'].append({
-                                    't': int(s.get('t', 0)),
-                                    # @d is mandatory (see [1, 5.3.9.6.2, Table 17, page 60])
-                                    'd': int(s.attrib['d']),
-                                    'r': r,
-                                })
+                                ms_info['total_number'] += 1 + int(s.get('r', '0'))
                     else:
                         timescale = segment_template.get('timescale')
                         if timescale:
@@ -1606,7 +1540,7 @@ class InfoExtractor(object):
                         continue
                     representation_attrib = adaptation_set.attrib.copy()
                     representation_attrib.update(representation.attrib)
-                    # According to [1, 5.3.7.2, Table 9, page 41], @mimeType is mandatory
+                    # According to page 41 of ISO/IEC 29001-1:2014, @mimeType is mandatory
                     mime_type = representation_attrib['mimeType']
                     content_type = mime_type.split('/')[0]
                     if content_type == 'text':
@@ -1650,40 +1584,16 @@ class InfoExtractor(object):
                                 representation_ms_info['total_number'] = int(math.ceil(float(period_duration) / segment_duration))
                             media_template = representation_ms_info['media_template']
                             media_template = media_template.replace('$RepresentationID$', representation_id)
-                            media_template = re.sub(r'\$(Number|Bandwidth|Time)\$', r'%(\1)d', media_template)
-                            media_template = re.sub(r'\$(Number|Bandwidth|Time)%([^$]+)\$', r'%(\1)\2', media_template)
+                            media_template = re.sub(r'\$(Number|Bandwidth)\$', r'%(\1)d', media_template)
+                            media_template = re.sub(r'\$(Number|Bandwidth)%([^$]+)\$', r'%(\1)\2', media_template)
                             media_template.replace('$$', '$')
-
-                            # As per [1, 5.3.9.4.4, Table 16, page 55] $Number$ and $Time$
-                            # can't be used at the same time
-                            if '%(Number' in media_template:
-                                representation_ms_info['segment_urls'] = [
-                                    media_template % {
-                                        'Number': segment_number,
-                                        'Bandwidth': representation_attrib.get('bandwidth'),
-                                    }
-                                    for segment_number in range(
-                                        representation_ms_info['start_number'],
-                                        representation_ms_info['total_number'] + representation_ms_info['start_number'])]
-                            else:
-                                representation_ms_info['segment_urls'] = []
-                                segment_time = 0
-
-                                def add_segment_url():
-                                    representation_ms_info['segment_urls'].append(
-                                        media_template % {
-                                            'Time': segment_time,
-                                            'Bandwidth': representation_attrib.get('bandwidth'),
-                                        }
-                                    )
-
-                                for num, s in enumerate(representation_ms_info['s']):
-                                    segment_time = s.get('t') or segment_time
-                                    add_segment_url()
-                                    for r in range(s.get('r', 0)):
-                                        segment_time += s['d']
-                                        add_segment_url()
-                                    segment_time += s['d']
+                            representation_ms_info['segment_urls'] = [
+                                media_template % {
+                                    'Number': segment_number,
+                                    'Bandwidth': representation_attrib.get('bandwidth')}
+                                for segment_number in range(
+                                    representation_ms_info['start_number'],
+                                    representation_ms_info['total_number'] + representation_ms_info['start_number'])]
                         if 'segment_urls' in representation_ms_info:
                             f.update({
                                 'segment_urls': representation_ms_info['segment_urls'],
@@ -1710,88 +1620,6 @@ class InfoExtractor(object):
                         self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
         return formats
 
-    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8'):
-        def absolute_url(video_url):
-            return compat_urlparse.urljoin(base_url, video_url)
-
-        def parse_content_type(content_type):
-            if not content_type:
-                return {}
-            ctr = re.search(r'(?P<mimetype>[^/]+/[^;]+)(?:;\s*codecs="?(?P<codecs>[^"]+))?', content_type)
-            if ctr:
-                mimetype, codecs = ctr.groups()
-                f = parse_codecs(codecs)
-                f['ext'] = mimetype2ext(mimetype)
-                return f
-            return {}
-
-        def _media_formats(src, cur_media_type):
-            full_url = absolute_url(src)
-            if determine_ext(full_url) == 'm3u8':
-                is_plain_url = False
-                formats = self._extract_m3u8_formats(
-                    full_url, video_id, ext='mp4',
-                    entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id)
-            else:
-                is_plain_url = True
-                formats = [{
-                    'url': full_url,
-                    'vcodec': 'none' if cur_media_type == 'audio' else None,
-                }]
-            return is_plain_url, formats
-
-        entries = []
-        for media_tag, media_type, media_content in re.findall(r'(?s)(<(?P<tag>video|audio)[^>]*>)(.*?)</(?P=tag)>', webpage):
-            media_info = {
-                'formats': [],
-                'subtitles': {},
-            }
-            media_attributes = extract_attributes(media_tag)
-            src = media_attributes.get('src')
-            if src:
-                _, formats = _media_formats(src, media_type)
-                media_info['formats'].extend(formats)
-            media_info['thumbnail'] = media_attributes.get('poster')
-            if media_content:
-                for source_tag in re.findall(r'<source[^>]+>', media_content):
-                    source_attributes = extract_attributes(source_tag)
-                    src = source_attributes.get('src')
-                    if not src:
-                        continue
-                    is_plain_url, formats = _media_formats(src, media_type)
-                    if is_plain_url:
-                        f = parse_content_type(source_attributes.get('type'))
-                        f.update(formats[0])
-                        media_info['formats'].append(f)
-                    else:
-                        media_info['formats'].extend(formats)
-                for track_tag in re.findall(r'<track[^>]+>', media_content):
-                    track_attributes = extract_attributes(track_tag)
-                    kind = track_attributes.get('kind')
-                    if not kind or kind == 'subtitles':
-                        src = track_attributes.get('src')
-                        if not src:
-                            continue
-                        lang = track_attributes.get('srclang') or track_attributes.get('lang') or track_attributes.get('label')
-                        media_info['subtitles'].setdefault(lang, []).append({
-                            'url': absolute_url(src),
-                        })
-            if media_info['formats']:
-                entries.append(media_info)
-        return entries
-
-    def _extract_akamai_formats(self, manifest_url, video_id):
-        formats = []
-        f4m_url = re.sub(r'(https?://.+?)/i/', r'\1/z/', manifest_url).replace('/master.m3u8', '/manifest.f4m')
-        formats.extend(self._extract_f4m_formats(
-            update_url_query(f4m_url, {'hdcore': '3.7.0'}),
-            video_id, f4m_id='hds', fatal=False))
-        m3u8_url = re.sub(r'(https?://.+?)/z/', r'\1/i/', manifest_url).replace('/manifest.f4m', '/master.m3u8')
-        formats.extend(self._extract_m3u8_formats(
-            m3u8_url, video_id, 'mp4', 'm3u8_native',
-            m3u8_id='hls', fatal=False))
-        return formats
-
     def _live_title(self, name):
         """ Generate the title for a live video """
         now = datetime.datetime.now()
@@ -1852,7 +1680,7 @@ class InfoExtractor(object):
 
         any_restricted = False
         for tc in self.get_testcases(include_onlymatching=False):
-            if tc.get('playlist', []):
+            if 'playlist' in tc:
                 tc = tc['playlist'][0]
             is_restricted = age_restricted(
                 tc.get('info_dict', {}).get('age_limit'), age_limit)
@@ -1905,13 +1733,6 @@ class InfoExtractor(object):
     def _mark_watched(self, *args, **kwargs):
         raise NotImplementedError('This method must be implemented by subclasses')
 
-    def geo_verification_headers(self):
-        headers = {}
-        geo_verification_proxy = self._downloader.params.get('geo_verification_proxy')
-        if geo_verification_proxy:
-            headers['Ytdl-request-proxy'] = geo_verification_proxy
-        return headers
-
 
 class SearchInfoExtractor(InfoExtractor):
     """

youtube_dl/extractor/condenast.py

@@ -5,17 +5,13 @@ import re
 
 from .common import InfoExtractor
 from ..compat import (
+    compat_urllib_parse_urlencode,
     compat_urllib_parse_urlparse,
     compat_urlparse,
 )
 from ..utils import (
     orderedSet,
     remove_end,
-    extract_attributes,
-    mimetype2ext,
-    determine_ext,
-    int_or_none,
-    parse_iso8601,
 )
 
 
@@ -62,9 +58,6 @@ class CondeNastIE(InfoExtractor):
             'ext': 'mp4',
             'title': '3D Printed Speakers Lit With LED',
             'description': 'Check out these beautiful 3D printed LED speakers.  You can\'t actually buy them, but LumiGeek is working on a board that will let you make you\'re own.',
-            'uploader': 'wired',
-            'upload_date': '20130314',
-            'timestamp': 1363219200,
         }
     }, {
         # JS embed
@@ -74,93 +67,70 @@ class CondeNastIE(InfoExtractor):
             'id': '55f9cf8b61646d1acf00000c',
             'ext': 'mp4',
             'title': '3D printed TSA Travel Sentry keys really do open TSA locks',
-            'uploader': 'arstechnica',
-            'upload_date': '20150916',
-            'timestamp': 1442434955,
         }
     }]
 
     def _extract_series(self, url, webpage):
-        title = self._html_search_regex(
-            r'(?s)<div class="cne-series-info">.*?<h1>(.+?)</h1>',
-            webpage, 'series title')
+        title = self._html_search_regex(r'<div class="cne-series-info">.*?<h1>(.+?)</h1>',
+                                        webpage, 'series title', flags=re.DOTALL)
         url_object = compat_urllib_parse_urlparse(url)
         base_url = '%s://%s' % (url_object.scheme, url_object.netloc)
-        m_paths = re.finditer(
-            r'(?s)<p class="cne-thumb-title">.*?<a href="(/watch/.+?)["\?]', webpage)
+        m_paths = re.finditer(r'<p class="cne-thumb-title">.*?<a href="(/watch/.+?)["\?]',
+                              webpage, flags=re.DOTALL)
         paths = orderedSet(m.group(1) for m in m_paths)
         build_url = lambda path: compat_urlparse.urljoin(base_url, path)
         entries = [self.url_result(build_url(path), 'CondeNast') for path in paths]
         return self.playlist_result(entries, playlist_title=title)
 
     def _extract_video(self, webpage, url_type):
-        query = {}
-        params = self._search_regex(
-            r'(?s)var params = {(.+?)}[;,]', webpage, 'player params', default=None)
-        if params:
-            query.update({
-                'videoId': self._search_regex(r'videoId: [\'"](.+?)[\'"]', params, 'video id'),
-                'playerId': self._search_regex(r'playerId: [\'"](.+?)[\'"]', params, 'player id'),
-                'target': self._search_regex(r'target: [\'"](.+?)[\'"]', params, 'target'),
-            })
+        if url_type != 'embed':
+            description = self._html_search_regex(
+                [
+                    r'<div class="cne-video-description">(.+?)</div>',
+                    r'<div class="video-post-content">(.+?)</div>',
+                ],
+                webpage, 'description', fatal=False, flags=re.DOTALL)
         else:
-            params = extract_attributes(self._search_regex(
-                r'(<[^>]+data-js="video-player"[^>]+>)',
-                webpage, 'player params element'))
-            query.update({
-                'videoId': params['data-video'],
-                'playerId': params['data-player'],
-                'target': params['id'],
-            })
-        video_id = query['videoId']
-        video_info = None
-        info_page = self._download_webpage(
-            'http://player.cnevids.com/player/video.js',
-            video_id, 'Downloading video info', query=query, fatal=False)
-        if info_page:
-            video_info = self._parse_json(self._search_regex(
-                r'loadCallback\(({.+})\)', info_page, 'video info'), video_id)['video']
-        else:
-            info_page = self._download_webpage(
-                'http://player.cnevids.com/player/loader.js',
-                video_id, 'Downloading loader info', query=query)
-            video_info = self._parse_json(self._search_regex(
-                r'var\s+video\s*=\s*({.+?});', info_page, 'video info'), video_id)
-        title = video_info['title']
+            description = None
+        params = self._search_regex(r'var params = {(.+?)}[;,]', webpage,
+                                    'player params', flags=re.DOTALL)
+        video_id = self._search_regex(r'videoId: [\'"](.+?)[\'"]', params, 'video id')
+        player_id = self._search_regex(r'playerId: [\'"](.+?)[\'"]', params, 'player id')
+        target = self._search_regex(r'target: [\'"](.+?)[\'"]', params, 'target')
+        data = compat_urllib_parse_urlencode({'videoId': video_id,
+                                              'playerId': player_id,
+                                              'target': target,
+                                              })
+        base_info_url = self._search_regex(r'url = [\'"](.+?)[\'"][,;]',
+                                           webpage, 'base info url',
+                                           default='http://player.cnevids.com/player/loader.js?')
+        info_url = base_info_url + data
+        info_page = self._download_webpage(info_url, video_id,
+                                           'Downloading video info')
+        video_info = self._search_regex(r'var\s+video\s*=\s*({.+?});', info_page, 'video info')
+        video_info = self._parse_json(video_info, video_id)
 
-        formats = []
-        for fdata in video_info.get('sources', [{}])[0]:
-            src = fdata.get('src')
-            if not src:
-                continue
-            ext = mimetype2ext(fdata.get('type')) or determine_ext(src)
-            quality = fdata.get('quality')
-            formats.append({
-                'format_id': ext + ('-%s' % quality if quality else ''),
-                'url': src,
-                'ext': ext,
-                'quality': 1 if quality == 'high' else 0,
-            })
+        formats = [{
+            'format_id': '%s-%s' % (fdata['type'].split('/')[-1], fdata['quality']),
+            'url': fdata['src'],
+            'ext': fdata['type'].split('/')[-1],
+            'quality': 1 if fdata['quality'] == 'high' else 0,
+        } for fdata in video_info['sources'][0]]
         self._sort_formats(formats)
 
-        info = self._search_json_ld(
-            webpage, video_id, fatal=False) if url_type != 'embed' else {}
-        info.update({
+        return {
             'id': video_id,
             'formats': formats,
-            'title': title,
-            'thumbnail': video_info.get('poster_frame'),
-            'uploader': video_info.get('brand'),
-            'duration': int_or_none(video_info.get('duration')),
-            'tags': video_info.get('tags'),
-            'series': video_info.get('series_title'),
-            'season': video_info.get('season_title'),
-            'timestamp': parse_iso8601(video_info.get('premiere_date')),
-        })
-        return info
+            'title': video_info['title'],
+            'thumbnail': video_info['poster_frame'],
+            'description': description,
+        }
 
     def _real_extract(self, url):
-        site, url_type, item_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
+        site = mobj.group('site')
+        url_type = mobj.group('type')
+        item_id = mobj.group('id')
 
         # Convert JS embed to regular embed
         if url_type == 'embedjs':

youtube_dl/extractor/crackle.py

@@ -1,5 +1,5 @@
 # coding: utf-8
-from __future__ import unicode_literals, division
+from __future__ import unicode_literals
 
 from .common import InfoExtractor
 from ..utils import int_or_none
@@ -8,22 +8,12 @@ from ..utils import int_or_none
 class CrackleIE(InfoExtractor):
     _VALID_URL = r'(?:crackle:|https?://(?:www\.)?crackle\.com/(?:playlist/\d+/|(?:[^/]+/)+))(?P<id>\d+)'
     _TEST = {
-        'url': 'http://www.crackle.com/comedians-in-cars-getting-coffee/2498934',
+        'url': 'http://www.crackle.com/the-art-of-more/2496419',
         'info_dict': {
-            'id': '2498934',
+            'id': '2496419',
             'ext': 'mp4',
-            'title': 'Everybody Respects A Bloody Nose',
-            'description': 'Jerry is kaffeeklatsching in L.A. with funnyman J.B. Smoove (Saturday Night Live, Real Husbands of Hollywood). They’re headed for brew at 10 Speed Coffee in a 1964 Studebaker Avanti.',
-            'thumbnail': 're:^https?://.*\.jpg',
-            'duration': 906,
-            'series': 'Comedians In Cars Getting Coffee',
-            'season_number': 8,
-            'episode_number': 4,
-            'subtitles': {
-                'en-US': [{
-                    'ext': 'ttml',
-                }]
-            },
+            'title': 'Heavy Lies the Head',
+            'description': 'md5:bb56aa0708fe7b9a4861535f15c3abca',
         },
         'params': {
             # m3u8 download
@@ -31,8 +21,12 @@ class CrackleIE(InfoExtractor):
         }
     }
 
-    # extracted from http://legacyweb-us.crackle.com/flash/ReferrerRedirect.ashx
+    # extracted from http://legacyweb-us.crackle.com/flash/QueryReferrer.ashx
+    _SUBTITLE_SERVER = 'http://web-us-az.crackle.com'
+    _UPLYNK_OWNER_ID = 'e8773f7770a44dbd886eee4fca16a66b'
     _THUMBNAIL_TEMPLATE = 'http://images-us-am.crackle.com/%stnl_1920x1080.jpg?ts=20140107233116?c=635333335057637614'
+
+    # extracted from http://legacyweb-us.crackle.com/flash/ReferrerRedirect.ashx
     _MEDIA_FILE_SLOTS = {
         'c544.flv': {
             'width': 544,
@@ -54,21 +48,16 @@ class CrackleIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-
-        config_doc = self._download_xml(
-            'http://legacyweb-us.crackle.com/flash/QueryReferrer.ashx?site=16',
-            video_id, 'Downloading config')
-
         item = self._download_xml(
             'http://legacyweb-us.crackle.com/app/revamp/vidwallcache.aspx?flags=-1&fm=%s' % video_id,
             video_id).find('i')
         title = item.attrib['t']
 
+        thumbnail = None
         subtitles = {}
         formats = self._extract_m3u8_formats(
-            'http://content.uplynk.com/ext/%s/%s.m3u8' % (config_doc.attrib['strUplynkOwnerId'], video_id),
+            'http://content.uplynk.com/ext/%s/%s.m3u8' % (self._UPLYNK_OWNER_ID, video_id),
             video_id, 'mp4', m3u8_id='hls', fatal=None)
-        thumbnail = None
         path = item.attrib.get('p')
         if path:
             thumbnail = self._THUMBNAIL_TEMPLATE % path
@@ -87,7 +76,7 @@ class CrackleIE(InfoExtractor):
                     if locale not in subtitles:
                         subtitles[locale] = []
                     subtitles[locale] = [{
-                        'url': '%s/%s%s_%s.xml' % (config_doc.attrib['strSubtitleServer'], path, locale, v),
+                        'url': '%s/%s%s_%s.xml' % (self._SUBTITLE_SERVER, path, locale, v),
                         'ext': 'ttml',
                     }]
         self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
@@ -96,7 +85,7 @@ class CrackleIE(InfoExtractor):
             'id': video_id,
             'title': title,
             'description': item.attrib.get('d'),
-            'duration': int(item.attrib.get('r'), 16) / 1000 if item.attrib.get('r') else None,
+            'duration': int(item.attrib.get('r'), 16) if item.attrib.get('r') else None,
             'series': item.attrib.get('sn'),
             'season_number': int_or_none(item.attrib.get('se')),
             'episode_number': int_or_none(item.attrib.get('ep')),

youtube_dl/extractor/crunchyroll.py

@@ -114,21 +114,6 @@ class CrunchyrollIE(CrunchyrollBaseIE):
             # rtmp
             'skip_download': True,
         },
-    }, {
-        'url': 'http://www.crunchyroll.com/rezero-starting-life-in-another-world-/episode-5-the-morning-of-our-promise-is-still-distant-702409',
-        'info_dict': {
-            'id': '702409',
-            'ext': 'mp4',
-            'title': 'Re:ZERO -Starting Life in Another World- Episode 5 – The Morning of Our Promise Is Still Distant',
-            'description': 'md5:97664de1ab24bbf77a9c01918cb7dca9',
-            'thumbnail': 're:^https?://.*\.jpg$',
-            'uploader': 'TV TOKYO',
-            'upload_date': '20160508',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
     }, {
         'url': 'http://www.crunchyroll.fr/girl-friend-beta/episode-11-goodbye-la-mode-661697',
         'only_matching': True,
@@ -351,18 +336,9 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
             if video_encode_id in video_encode_ids:
                 continue
             video_encode_ids.append(video_encode_id)
-
-            video_file = xpath_text(stream_info, './file')
-            if not video_file:
-                continue
-            if video_file.startswith('http'):
-                formats.extend(self._extract_m3u8_formats(
-                    video_file, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
-                continue
-
             video_url = xpath_text(stream_info, './host')
-            if not video_url:
+            video_play_path = xpath_text(stream_info, './file')
+            if not video_url or not video_play_path:
                 continue
             metadata = stream_info.find('./metadata')
             format_info = {
@@ -377,7 +353,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                 parsed_video_url = compat_urlparse.urlparse(video_url)
                 direct_video_url = compat_urlparse.urlunparse(parsed_video_url._replace(
                     netloc='v.lvlt.crcdn.net',
-                    path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_file.split(':')[-1])))
+                    path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_play_path.split(':')[-1])))
                 if self._is_valid_url(direct_video_url, video_id, video_format):
                     format_info.update({
                         'url': direct_video_url,
@@ -387,7 +363,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
 
             format_info.update({
                 'url': video_url,
-                'play_path': video_file,
+                'play_path': video_play_path,
                 'ext': 'flv',
             })
             formats.append(format_info)

youtube_dl/extractor/cspan.py

@@ -51,11 +51,8 @@ class CSpanIE(InfoExtractor):
         'url': 'http://www.c-span.org/video/?104517-1/immigration-reforms-needed-protect-skilled-american-workers',
         'info_dict': {
             'id': 'judiciary031715',
-            'ext': 'mp4',
+            'ext': 'flv',
             'title': 'Immigration Reforms Needed to Protect Skilled American Workers',
-        },
-        'params': {
-            'skip_download': True,  # m3u8 downloads
         }
     }]
 

youtube_dl/extractor/ctsnews.py

@@ -1,12 +1,13 @@
-# coding: utf-8
+# -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..utils import unified_timestamp
+from ..utils import parse_iso8601, ExtractorError
 
 
 class CtsNewsIE(InfoExtractor):
     IE_DESC = '華視新聞'
+    # https connection failed (Connection reset)
     _VALID_URL = r'https?://news\.cts\.com\.tw/[a-z]+/[a-z]+/\d+/(?P<id>\d+)\.html'
     _TESTS = [{
         'url': 'http://news.cts.com.tw/cts/international/201501/201501291578109.html',
@@ -15,7 +16,7 @@ class CtsNewsIE(InfoExtractor):
             'id': '201501291578109',
             'ext': 'mp4',
             'title': '以色列.真主黨交火 3人死亡',
-            'description': '以色列和黎巴嫩真主黨，爆發五年最嚴重衝突，雙方砲轟交火，兩名以軍死亡，還有一名西班牙籍的聯合國維和人...',
+            'description': 'md5:95e9b295c898b7ff294f09d450178d7d',
             'timestamp': 1422528540,
             'upload_date': '20150129',
         }
@@ -27,7 +28,7 @@ class CtsNewsIE(InfoExtractor):
             'id': '201309031304098',
             'ext': 'mp4',
             'title': '韓國31歲童顏男 貌如十多歲小孩',
-            'description': '越有年紀的人，越希望看起來年輕一點，而南韓卻有一位31歲的男子，看起來像是11、12歲的小孩，身...',
+            'description': 'md5:f183feeba3752b683827aab71adad584',
             'thumbnail': 're:^https?://.*\.jpg$',
             'timestamp': 1378205880,
             'upload_date': '20130903',
@@ -35,7 +36,8 @@ class CtsNewsIE(InfoExtractor):
     }, {
         # With Youtube embedded video
         'url': 'http://news.cts.com.tw/cts/money/201501/201501291578003.html',
-        'md5': 'e4726b2ccd70ba2c319865e28f0a91d1',
+        'md5': '1d842c771dc94c8c3bca5af2cc1db9c5',
+        'add_ie': ['Youtube'],
         'info_dict': {
             'id': 'OVbfO7d0_hQ',
             'ext': 'mp4',
@@ -45,37 +47,42 @@ class CtsNewsIE(InfoExtractor):
             'upload_date': '20150128',
             'uploader_id': 'TBSCTS',
             'uploader': '中華電視公司',
-        },
-        'add_ie': ['Youtube'],
+        }
     }]
 
     def _real_extract(self, url):
         news_id = self._match_id(url)
         page = self._download_webpage(url, news_id)
 
-        news_id = self._hidden_inputs(page).get('get_id')
-
-        if news_id:
-            mp4_feed = self._download_json(
-                'http://news.cts.com.tw/action/test_mp4feed.php',
-                news_id, note='Fetching feed', query={'news_id': news_id})
-            video_url = mp4_feed['source_url']
+        if self._search_regex(r'(CTSPlayer2)', page, 'CTSPlayer2 identifier', default=None):
+            feed_url = self._html_search_regex(
+                r'(http://news\.cts\.com\.tw/action/mp4feed\.php\?news_id=\d+)',
+                page, 'feed url')
+            video_url = self._download_webpage(
+                feed_url, news_id, note='Fetching feed')
         else:
             self.to_screen('Not CTSPlayer video, trying Youtube...')
             youtube_url = self._search_regex(
-                r'src="(//www\.youtube\.com/embed/[^"]+)"', page, 'youtube url')
+                r'src="(//www\.youtube\.com/embed/[^"]+)"', page, 'youtube url',
+                default=None)
+            if not youtube_url:
+                raise ExtractorError('The news includes no videos!', expected=True)
 
-            return self.url_result(youtube_url, ie='Youtube')
+            return {
+                '_type': 'url',
+                'url': youtube_url,
+                'ie_key': 'Youtube',
+            }
 
         description = self._html_search_meta('description', page)
-        title = self._html_search_meta('title', page, fatal=True)
+        title = self._html_search_meta('title', page)
         thumbnail = self._html_search_meta('image', page)
 
         datetime_str = self._html_search_regex(
-            r'(\d{4}/\d{2}/\d{2} \d{2}:\d{2})', page, 'date and time', fatal=False)
-        timestamp = None
-        if datetime_str:
-            timestamp = unified_timestamp(datetime_str) - 8 * 3600
+            r'(\d{4}/\d{2}/\d{2} \d{2}:\d{2})', page, 'date and time')
+        # Transform into ISO 8601 format with timezone info
+        datetime_str = datetime_str.replace('/', '-') + ':00+0800'
+        timestamp = parse_iso8601(datetime_str, delimiter=' ')
 
         return {
             'id': news_id,

youtube_dl/extractor/ctv.py

@@ -1,46 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-
-
-class CTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?(?P<domain>ctv|tsn|bnn|thecomedynetwork)\.ca/.*?(?:\bvid=|-vid|~|%7E)(?P<id>[0-9.]+)'
-    _TESTS = [{
-        'url': 'http://www.ctv.ca/video/player?vid=706966',
-        'md5': 'ff2ebbeae0aa2dcc32a830c3fd69b7b0',
-        'info_dict': {
-            'id': '706966',
-            'ext': 'mp4',
-            'title': 'Larry Day and Richard Jutras on the TIFF red carpet of \'Stonewall\'',
-            'description': 'etalk catches up with Larry Day and Richard Jutras on the TIFF red carpet of "Stonewall”.',
-            'upload_date': '20150919',
-            'timestamp': 1442624700,
-        },
-        'expected_warnings': ['HTTP Error 404'],
-    }, {
-        'url': 'http://www.thecomedynetwork.ca/video/player?vid=923582',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.tsn.ca/video/expectations-high-for-milos-raonic-at-us-open~939549',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.bnn.ca/video/berman-s-call-part-two-viewer-questions~939654',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.ctv.ca/YourMorning/Video/S1E6-Monday-August-29-2016-vid938009',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        domain, video_id = re.match(self._VALID_URL, url).groups()
-        if domain == 'thecomedynetwork':
-            domain = 'comedy'
-        return {
-            '_type': 'url_transparent',
-            'id': video_id,
-            'url': '9c9media:%s_web:%s' % (domain, video_id),
-            'ie_key': 'NineCNineMedia',
-        }

youtube_dl/extractor/ctvnews.py

@@ -1,65 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import orderedSet
-
-
-class CTVNewsIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?ctvnews\.ca/(?:video\?(?:clip|playlist|bin)Id=|.*?)(?P<id>[0-9.]+)'
-    _TESTS = [{
-        'url': 'http://www.ctvnews.ca/video?clipId=901995',
-        'md5': '10deb320dc0ccb8d01d34d12fc2ea672',
-        'info_dict': {
-            'id': '901995',
-            'ext': 'mp4',
-            'title': 'Extended: \'That person cannot be me\' Johnson says',
-            'description': 'md5:958dd3b4f5bbbf0ed4d045c790d89285',
-            'timestamp': 1467286284,
-            'upload_date': '20160630',
-        }
-    }, {
-        'url': 'http://www.ctvnews.ca/video?playlistId=1.2966224',
-        'info_dict':
-        {
-            'id': '1.2966224',
-        },
-        'playlist_mincount': 19,
-    }, {
-        'url': 'http://www.ctvnews.ca/video?binId=1.2876780',
-        'info_dict':
-        {
-            'id': '1.2876780',
-        },
-        'playlist_mincount': 100,
-    }, {
-        'url': 'http://www.ctvnews.ca/1.810401',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.ctvnews.ca/canadiens-send-p-k-subban-to-nashville-in-blockbuster-trade-1.2967231',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        page_id = self._match_id(url)
-
-        def ninecninemedia_url_result(clip_id):
-            return {
-                '_type': 'url_transparent',
-                'id': clip_id,
-                'url': '9c9media:ctvnews_web:%s' % clip_id,
-                'ie_key': 'NineCNineMedia',
-            }
-
-        if page_id.isdigit():
-            return ninecninemedia_url_result(page_id)
-        else:
-            webpage = self._download_webpage('http://www.ctvnews.ca/%s' % page_id, page_id, query={
-                'ot': 'example.AjaxPageLayout.ot',
-                'maxItemsPerPage': 1000000,
-            })
-            entries = [ninecninemedia_url_result(clip_id) for clip_id in orderedSet(
-                re.findall(r'clip\.id\s*=\s*(\d+);', webpage))]
-            return self.playlist_result(entries, page_id)

youtube_dl/extractor/cultureunplugged.py

@@ -1,13 +1,9 @@
 from __future__ import unicode_literals
 
 import re
-import time
 
 from .common import InfoExtractor
-from ..utils import (
-    int_or_none,
-    HEADRequest,
-)
+from ..utils import int_or_none
 
 
 class CultureUnpluggedIE(InfoExtractor):
@@ -36,9 +32,6 @@ class CultureUnpluggedIE(InfoExtractor):
         video_id = mobj.group('id')
         display_id = mobj.group('display_id') or video_id
 
-        # request setClientTimezone.php to get PHPSESSID cookie which is need to get valid json data in the next request
-        self._request_webpage(HEADRequest(
-            'http://www.cultureunplugged.com/setClientTimezone.php?timeOffset=%d' % -(time.timezone / 3600)), display_id)
         movie_data = self._download_json(
             'http://www.cultureunplugged.com/movie-data/cu-%s.json' % video_id, display_id)
 

youtube_dl/extractor/curiositystream.py

@@ -1,120 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    int_or_none,
-    urlencode_postdata,
-    compat_str,
-    ExtractorError,
-)
-
-
-class CuriosityStreamBaseIE(InfoExtractor):
-    _NETRC_MACHINE = 'curiositystream'
-    _auth_token = None
-    _API_BASE_URL = 'https://api.curiositystream.com/v1/'
-
-    def _handle_errors(self, result):
-        error = result.get('error', {}).get('message')
-        if error:
-            if isinstance(error, dict):
-                error = ', '.join(error.values())
-            raise ExtractorError(
-                '%s said: %s' % (self.IE_NAME, error), expected=True)
-
-    def _call_api(self, path, video_id):
-        headers = {}
-        if self._auth_token:
-            headers['X-Auth-Token'] = self._auth_token
-        result = self._download_json(
-            self._API_BASE_URL + path, video_id, headers=headers)
-        self._handle_errors(result)
-        return result['data']
-
-    def _real_initialize(self):
-        (email, password) = self._get_login_info()
-        if email is None:
-            return
-        result = self._download_json(
-            self._API_BASE_URL + 'login', None, data=urlencode_postdata({
-                'email': email,
-                'password': password,
-            }))
-        self._handle_errors(result)
-        self._auth_token = result['message']['auth_token']
-
-    def _extract_media_info(self, media):
-        video_id = compat_str(media['id'])
-        limelight_media_id = media['limelight_media_id']
-        title = media['title']
-
-        subtitles = {}
-        for closed_caption in media.get('closed_captions', []):
-            sub_url = closed_caption.get('file')
-            if not sub_url:
-                continue
-            lang = closed_caption.get('code') or closed_caption.get('language') or 'en'
-            subtitles.setdefault(lang, []).append({
-                'url': sub_url,
-            })
-
-        return {
-            '_type': 'url_transparent',
-            'id': video_id,
-            'url': 'limelight:media:' + limelight_media_id,
-            'title': title,
-            'description': media.get('description'),
-            'thumbnail': media.get('image_large') or media.get('image_medium') or media.get('image_small'),
-            'duration': int_or_none(media.get('duration')),
-            'tags': media.get('tags'),
-            'subtitles': subtitles,
-            'ie_key': 'LimelightMedia',
-        }
-
-
-class CuriosityStreamIE(CuriosityStreamBaseIE):
-    IE_NAME = 'curiositystream'
-    _VALID_URL = r'https?://app\.curiositystream\.com/video/(?P<id>\d+)'
-    _TEST = {
-        'url': 'https://app.curiositystream.com/video/2',
-        'md5': 'a0074c190e6cddaf86900b28d3e9ee7a',
-        'info_dict': {
-            'id': '2',
-            'ext': 'mp4',
-            'title': 'How Did You Develop The Internet?',
-            'description': 'Vint Cerf, Google\'s Chief Internet Evangelist, describes how he and Bob Kahn created the internet.',
-            'timestamp': 1448388615,
-            'upload_date': '20151124',
-        }
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        media = self._call_api('media/' + video_id, video_id)
-        return self._extract_media_info(media)
-
-
-class CuriosityStreamCollectionIE(CuriosityStreamBaseIE):
-    IE_NAME = 'curiositystream:collection'
-    _VALID_URL = r'https?://app\.curiositystream\.com/collection/(?P<id>\d+)'
-    _TEST = {
-        'url': 'https://app.curiositystream.com/collection/2',
-        'info_dict': {
-            'id': '2',
-            'title': 'Curious Minds: The Internet',
-            'description': 'How is the internet shaping our lives in the 21st Century?',
-        },
-        'playlist_mincount': 17,
-    }
-
-    def _real_extract(self, url):
-        collection_id = self._match_id(url)
-        collection = self._call_api(
-            'collections/' + collection_id, collection_id)
-        entries = []
-        for media in collection.get('media', []):
-            entries.append(self._extract_media_info(media))
-        return self.playlist_result(
-            entries, collection_id,
-            collection.get('title'), collection.get('description'))

youtube_dl/extractor/cwtv.py

@@ -9,7 +9,7 @@ from ..utils import (
 
 
 class CWTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?cw(?:tv(?:pr)?|seed)\.com/(?:shows/)?(?:[^/]+/)+[^?]*\?.*\b(?:play|watch)=(?P<id>[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})'
+    _VALID_URL = r'https?://(?:www\.)?cw(?:tv|seed)\.com/(?:shows/)?(?:[^/]+/){2}\?.*\bplay=(?P<id>[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})'
     _TESTS = [{
         'url': 'http://cwtv.com/shows/arrow/legends-of-yesterday/?play=6b15e985-9345-4f60-baf8-56e96be57c63',
         'info_dict': {
@@ -28,8 +28,7 @@ class CWTVIE(InfoExtractor):
         'params': {
             # m3u8 download
             'skip_download': True,
-        },
-        'skip': 'redirect to http://cwtv.com/shows/arrow/',
+        }
     }, {
         'url': 'http://www.cwseed.com/shows/whose-line-is-it-anyway/jeff-davis-4/?play=24282b12-ead2-42f2-95ad-26770c2c6088',
         'info_dict': {
@@ -45,43 +44,22 @@ class CWTVIE(InfoExtractor):
             'upload_date': '20151006',
             'timestamp': 1444107300,
         },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
     }, {
         'url': 'http://cwtv.com/thecw/chroniclesofcisco/?play=8adebe35-f447-465f-ab52-e863506ff6d6',
         'only_matching': True,
-    }, {
-        'url': 'http://cwtvpr.com/the-cw/video?watch=9eee3f60-ef4e-440b-b3b2-49428ac9c54e',
-        'only_matching': True,
-    }, {
-        'url': 'http://cwtv.com/shows/arrow/legends-of-yesterday/?watch=6b15e985-9345-4f60-baf8-56e96be57c63',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_data = None
-        formats = []
-        for partner in (154, 213):
-            vdata = self._download_json(
-                'http://metaframe.digitalsmiths.tv/v2/CWtv/assets/%s/partner/%d?format=json' % (video_id, partner), video_id, fatal=False)
-            if not vdata:
-                continue
-            video_data = vdata
-            for quality, quality_data in vdata.get('videos', {}).items():
-                quality_url = quality_data.get('uri')
-                if not quality_url:
-                    continue
-                if quality == 'variantplaylist':
-                    formats.extend(self._extract_m3u8_formats(
-                        quality_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
-                else:
-                    tbr = int_or_none(quality_data.get('bitrate'))
-                    format_id = 'http' + ('-%d' % tbr if tbr else '')
-                    if self._is_valid_url(quality_url, video_id, format_id):
-                        formats.append({
-                            'format_id': format_id,
-                            'url': quality_url,
-                            'tbr': tbr,
-                        })
+        video_data = self._download_json(
+            'http://metaframe.digitalsmiths.tv/v2/CWtv/assets/%s/partner/132?format=json' % video_id, video_id)
+
+        formats = self._extract_m3u8_formats(
+            video_data['videos']['variantplaylist']['uri'], video_id, 'mp4')
         self._sort_formats(formats)
 
         thumbnails = [{

youtube_dl/extractor/dailymail.py

@@ -5,20 +5,19 @@ from .common import InfoExtractor
 from ..utils import (
     int_or_none,
     determine_protocol,
-    unescapeHTML,
 )
 
 
 class DailyMailIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?dailymail\.co\.uk/video/[^/]+/video-(?P<id>[0-9]+)'
     _TEST = {
-        'url': 'http://www.dailymail.co.uk/video/tvshowbiz/video-1295863/The-Mountain-appears-sparkling-water-ad-Heavy-Bubbles.html',
-        'md5': 'f6129624562251f628296c3a9ffde124',
+        'url': 'http://www.dailymail.co.uk/video/sciencetech/video-1288527/Turn-video-impressionist-masterpiece.html',
+        'md5': '2f639d446394f53f3a33658b518b6615',
         'info_dict': {
-            'id': '1295863',
+            'id': '1288527',
             'ext': 'mp4',
-            'title': 'The Mountain appears in sparkling water ad for \'Heavy Bubbles\'',
-            'description': 'md5:a93d74b6da172dd5dc4d973e0b766a84',
+            'title': 'Turn any video into an impressionist masterpiece',
+            'description': 'md5:88ddbcb504367987b2708bb38677c9d2',
         }
     }
 
@@ -27,7 +26,7 @@ class DailyMailIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
         video_data = self._parse_json(self._search_regex(
             r"data-opts='({.+?})'", webpage, 'video data'), video_id)
-        title = unescapeHTML(video_data['title'])
+        title = video_data['title']
         video_sources = self._download_json(video_data.get(
             'sources', {}).get('url') or 'http://www.dailymail.co.uk/api/player/%s/video-sources.json' % video_id, video_id)
 
@@ -56,7 +55,7 @@ class DailyMailIE(InfoExtractor):
         return {
             'id': video_id,
             'title': title,
-            'description': unescapeHTML(video_data.get('descr')),
+            'description': video_data.get('descr'),
             'thumbnail': video_data.get('poster') or video_data.get('thumbnail'),
             'formats': formats,
         }

youtube_dl/extractor/dailymotion.py

@@ -16,7 +16,6 @@ from ..utils import (
     sanitized_Request,
     str_to_int,
     unescapeHTML,
-    mimetype2ext,
 )
 
 
@@ -112,13 +111,6 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
         }
     ]
 
-    @staticmethod
-    def _extract_urls(webpage):
-        # Look for embedded Dailymotion player
-        matches = re.findall(
-            r'<(?:(?:embed|iframe)[^>]+?src=|input[^>]+id=[\'"]dmcloudUrlEmissionSelect[\'"][^>]+value=)(["\'])(?P<url>(?:https?:)?//(?:www\.)?dailymotion\.com/(?:embed|swf)/video/.+?)\1', webpage)
-        return list(map(lambda m: unescapeHTML(m[1]), matches))
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
@@ -161,19 +153,18 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
                     type_ = media.get('type')
                     if type_ == 'application/vnd.lumberjack.manifest':
                         continue
-                    ext = mimetype2ext(type_) or determine_ext(media_url)
-                    if ext == 'm3u8':
+                    ext = determine_ext(media_url)
+                    if type_ == 'application/x-mpegURL' or ext == 'm3u8':
                         formats.extend(self._extract_m3u8_formats(
                             media_url, video_id, 'mp4', preference=-1,
                             m3u8_id='hls', fatal=False))
-                    elif ext == 'f4m':
+                    elif type_ == 'application/f4m' or ext == 'f4m':
                         formats.extend(self._extract_f4m_formats(
                             media_url, video_id, preference=-1, f4m_id='hds', fatal=False))
                     else:
                         f = {
                             'url': media_url,
                             'format_id': 'http-%s' % quality,
-                            'ext': ext,
                         }
                         m = re.search(r'H264-(?P<width>\d+)x(?P<height>\d+)', media_url)
                         if m:
@@ -331,9 +322,7 @@ class DailymotionPlaylistIE(DailymotionBaseInfoExtractor):
 
             for video_id in re.findall(r'data-xid="(.+?)"', webpage):
                 if video_id not in video_ids:
-                    yield self.url_result(
-                        'http://www.dailymotion.com/video/%s' % video_id,
-                        DailymotionIE.ie_key(), video_id)
+                    yield self.url_result('http://www.dailymotion.com/video/%s' % video_id, 'Dailymotion')
                     video_ids.add(video_id)
 
             if re.search(self._MORE_PAGES_INDICATOR, webpage) is None: